date:20171226

Re: Remove retries from DML

2017-12-26 Thread Dmitriy Setrakyan

On Tue, Dec 26, 2017 at 11:21 PM, Vladimir Ozerov 
wrote:

> Agree. AFAIK ATOMIC cache doesn't perform retries, does it?
>

I think it does, in some cases. Alexey Goncharuk, can you share some wisdom
here?

Re: Remove retries from DML

2017-12-26 Thread Vladimir Ozerov

Agree. AFAIK ATOMIC cache doesn't perform retries, does it?

On Wed, Dec 27, 2017 at 12:12 AM, Dmitriy Setrakyan 
wrote:

> Vladimir,
>
> I think non-transactional DML should have the same guarantees as we have in
> Atomic caches. Do you agree? If yes, we should discuss DML behavior in
> conjunction with Atomic cache behavior.
>
> D.
>
> On Tue, Dec 26, 2017 at 10:44 AM, Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > Current implementation of DML is not transactional. We have not
> guarantees
> > on what is updated and what is not. When certain update fails due to
> > concurrent entry change, we perform a retry.
> >
> > The thing is that re-try doesn't guarantee anything still and it might
> > introduce subtle performance degradation in case of complex queries.
> >
> > As it has no value for users, I propose to drop it altogether.
> Semantically
> > nothing will change from user perspective as we have no guarantees.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>

Re: ATOMIC caches consistency - IEP-12

2017-12-26 Thread Dmitriy Setrakyan

Yakov,

As one of the prevention mechanisms for the scenario you describe, would it
be possible to send the update message to backup nodes in the order in
which they would become primary nodes? For example, if backup1 node is the
next in chain to become the primary node, then the current primary node
should send the update message to backup1 before it sends it to backup2.

In this case, if the primary and client nodes fail, then backup1 node will
become primary and will have the latest version of the data, no? Will this
solve the situation you describe, at least partially?

D.

On Mon, Dec 25, 2017 at 7:23 AM, Yakov Zhdanov  wrote:

> Hello!
>
> I tried to summarize some ideas on how to make ATOMIC caches stay
> consistent in case of topology changes.
>
> Imagine partitioned ATOMIC cache with 2 backups configured (primary + 2
> backup copies of the partition). One of possible scenarios to get to
> primary and backups divergence is the following - update initiating node
> sends update operation to primary node, primary node propagates update to 1
> of 2 backups and then dies. If initiating node crashes as well (and
> therefore cannot retry the operation) then in current implementation system
> comes to a situation when 2 copies of the partition present in cluster may
> be different. Note that both situations possible - new primary contains the
> latest update and backup does not and vice versa. New backup will be
> elected according to configured affinity and will rebalance the partition
> from random owner, but copies may not be consistent due to described above.
>
> This problem does not affect TRANSACTIONAL caches as 2PC protocol deals
> with scenarios of the kind very well.
>
> Here is the link to IEP -
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 12+Make+ATOMIC+Caches+Consistent+Again
>
> Sam, Alex G, Vladimir, please share your thoughts.
>
> --Yakov
>

Re: IGNITE-4585 Support CLEAR command with REST

2017-12-26 Thread Saikat Maitra

Thank you Denis !!!

Regards,
Saikat

On Tue, Dec 26, 2017 at 12:18 PM, Denis Magda  wrote:

> Hi Saikat, review and merged! Thanks!
>
> —
> Denis
>
> > On Dec 25, 2017, at 10:22 PM, Saikat Maitra 
> wrote:
> >
> > Hi,
> >
> >
> > I have raised PR[1] for the following jira ticket[2].
> >
> >
> > Please review[3] and let me know if any changes required.
> >
> >
> > Regards,
> >
> > Saikat
> >
> >
> > [1] https://github.com/apache/ignite/pull/3288
> >
> > [2] https://issues.apache.org/jira/browse/IGNITE-4585
> >
> > [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-442
>
>

Request for contributors permissions

2017-12-26 Thread Jason Man

Hello,

Please grant me access to assign myself Ignite issues.  I want to fix one
of Ignite's issue IGNITE-6853, which I have a PR ready for review.

Thanks,
Jason

Re: Do we need ML profile in AI 2.4?

2017-12-26 Thread Denis Magda

Yuri,

Thanks for confirming that it’s on track! Just wanted to be sure we’ve not 
missed this.

—
Denis

> On Dec 26, 2017, at 2:26 PM, Yury Babak  wrote:
> 
> +1
> 
> Moreover, as far as I know in IGNITE-7203 (Java 8 by default) ml profile
> will be removed and sources will be moved into general folder in examples
> and yardstick modules.
> 
> Regards,
> Yury
> 
> 
> 
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Do we need ML profile in AI 2.4?

2017-12-26 Thread Yury Babak

+1

Moreover, as far as I know in IGNITE-7203 (Java 8 by default) ml profile
will be removed and sources will be moved into general folder in examples
and yardstick modules.

Regards,
Yury



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Remove retries from DML

2017-12-26 Thread Dmitriy Setrakyan

Vladimir,

I think non-transactional DML should have the same guarantees as we have in
Atomic caches. Do you agree? If yes, we should discuss DML behavior in
conjunction with Atomic cache behavior.

D.

On Tue, Dec 26, 2017 at 10:44 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> Current implementation of DML is not transactional. We have not guarantees
> on what is updated and what is not. When certain update fails due to
> concurrent entry change, we perform a retry.
>
> The thing is that re-try doesn't guarantee anything still and it might
> introduce subtle performance degradation in case of complex queries.
>
> As it has no value for users, I propose to drop it altogether. Semantically
> nothing will change from user perspective as we have no guarantees.
>
> Thoughts?
>
> Vladimir.
>

[jira] [Created] (IGNITE-7319) Memory leak during creating/destroying local cache

2017-12-26 Thread Mikhail Cherkasov (JIRA)

Mikhail Cherkasov created IGNITE-7319:
-

 Summary: Memory leak during creating/destroying local cache
 Key: IGNITE-7319
 URL: https://issues.apache.org/jira/browse/IGNITE-7319
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.3
Reporter: Mikhail Cherkasov
 Fix For: 2.4


The following code creates local caches:

private IgniteCache createLocalCache(String name) { 
CacheConfiguration cCfg = new 
CacheConfiguration<>(); 
cCfg.setName(name); 
cCfg.setGroupName("localCaches"); // without group leak is much 
bigger! 
cCfg.setStoreKeepBinary(true); 
cCfg.setCacheMode(CacheMode.LOCAL); 
cCfg.setOnheapCacheEnabled(false); 
cCfg.setCopyOnRead(false); 
cCfg.setBackups(0); 
cCfg.setWriteBehindEnabled(false); 
cCfg.setReadThrough(false); 
cCfg.setReadFromBackup(false); 
cCfg.setQueryEntities(); 
return ignite.createCache(cCfg).withKeepBinary(); 
} 

The caches are placed in the queue and are picked up by the worker thread which 
just destroys them after removing from the queue. 
This setup seems to generate a memory leak of about 1GB per day. 
When looking at heap dump, I see all space is occupied by instances of 
java.util.concurrent.ConcurrentSkipListMap$Node.

User list: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-leak-in-GridCachePartitionExchangeManager-tt18995.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Remove retries from DML

2017-12-26 Thread Vladimir Ozerov

Igniters,

Current implementation of DML is not transactional. We have not guarantees
on what is updated and what is not. When certain update fails due to
concurrent entry change, we perform a retry.

The thing is that re-try doesn't guarantee anything still and it might
introduce subtle performance degradation in case of complex queries.

As it has no value for users, I propose to drop it altogether. Semantically
nothing will change from user perspective as we have no guarantees.

Thoughts?

Vladimir.

Re: Do we need ML profile in AI 2.4?

2017-12-26 Thread Dmitriy Setrakyan

+1

On Tue, Dec 26, 2017 at 9:45 AM, Denis Magda  wrote:

> Ignite ML contributors,
>
> Considering that Java 8 becomes default JDK in AI 2.4, do we need to keep
> a separate “ML” profile?
>
> I vote to discontinue it and have all the ML examples under the general
> examples directory:
> https://github.com/apache/ignite/tree/master/examples/
> src/main/java/org/apache/ignite/examples
>
> —
> Denis
>
>

Re: Ignite Committership Bar Guidance

2017-12-26 Thread Denis Magda

Good Ilya, we’ll review and discuss your candidacy in the nearest time.

—
Denis

> On Dec 26, 2017, at 9:54 AM, Ilya Lantukh  wrote:
> 
> Denis,
> 
> I presume that number of committers in Ignite doesn't grow as fast as
> amount of contributions, so it requires additional effort to keep track on
> each individual's progress.
> 
> So yes, I'd like to propose my candidacy since I'm both willing and able to
> participate in development of Apache Ignite and it's community.
> 
> On Sun, Dec 24, 2017 at 6:04 AM, Denis Magda  wrote:
> 
>> Hi Ilya,
>> 
>> Basically one of committers who usually reviews and merges code should
>> spot that it’s time to discuss a candidacy.
>> 
>> Anyway, if a contributor feels that committers don’t keep an eye on
>> his/her progress the contributor can remind about personal existence by
>> sending a message to @dev. ;)
>> 
>> —
>> Denis
>> 
>>> On Dec 23, 2017, at 3:58 AM, Ilya Lantukh  wrote:
>>> 
>>> Hi Denis,
>>> Who should take the initiative in promotion? As a contributor who meets
>>> those criteria, should I send my application to the PMC or just wait?
>>> 
>>> On Fri, Dec 22, 2017 at 8:24 PM, Denis Magda  wrote:
>>> 
 Hi Vyacheslav,
 
 Some common sense, I presume. A committer who will be proposing a
 candidate can use JIRA to see a total number of resolved tickets and Git
 history to get insights on a complexity of every contribution.
 
 Do you have any suggestions on this?
 
 —
 Denis
 
> On Dec 22, 2017, at 12:56 AM, Vyacheslav Daradur 
 wrote:
> 
> Hi, Denis, how will the complexity of tasks be estimated?
> 
> 
> On Fri, Dec 22, 2017 at 3:16 AM, Denis Magda 
>> wrote:
>> Igniters,
>> 
>> We at Ignite PMC put together a page with rules-of-thumb on when it’s
>> a
 right time to promote a contributor to a committer:
>> https://cwiki.apache.org/confluence/display/IGNITE/
 Committership+Bar+Guidance 
>> 
>> It’s intended to make “contributor to committer” process more evident
 and simplify Ignite PMC voting process on this matter.
>> 
>> Please use it as a reference, comments and suggestions are welcomed!
>> 
>> —
>> Denis
> 
> 
> 
> --
> Best Regards, Vyacheslav D.
 
 
>>> 
>>> 
>>> --
>>> Best regards,
>>> Ilya
>> 
>> 
> 
> 
> -- 
> Best regards,
> Ilya

Re: IGNITE-4585 Support CLEAR command with REST

2017-12-26 Thread Denis Magda

Hi Saikat, review and merged! Thanks!

—
Denis

> On Dec 25, 2017, at 10:22 PM, Saikat Maitra  wrote:
> 
> Hi,
> 
> 
> I have raised PR[1] for the following jira ticket[2].
> 
> 
> Please review[3] and let me know if any changes required.
> 
> 
> Regards,
> 
> Saikat
> 
> 
> [1] https://github.com/apache/ignite/pull/3288
> 
> [2] https://issues.apache.org/jira/browse/IGNITE-4585
> 
> [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-442

Re: Jira access

2017-12-26 Thread Denis Magda

Hi Pavel,

Welcome to the Ignite community!

Please subscribe to both dev and user lists:
https://ignite.apache.org/community/resources.html#mail-lists

Get familiar with Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

Once you got familiar and were able to run a few examples, you should pick
a Jira ticket you would like to start on. Send an email to the dev list sharing 
your JIRA id, so we can add you as a contributor in Jira.

These are the easy tickets to start with:
https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN

While those are more advanced but appealing:
https://ignite.apache.org/community/contribute.html#pick-ticket

Looking forward to your contributions!

Denis

> On Dec 25, 2017, at 5:26 AM, Pavel Kouznetsov  wrote:
> 
> Hello dear developers,
> 
> I'm Pavel and I'm going to contribute to Apache Ignite (mostly tests at this 
> time).
> 
> Could someone please give me access to jira 
> (https://issues.apache.org/jira/projects/IGNITE)
> 
> my jira id is pkouznet
> 
> Thanks,
> 
> -Pavel
> 
>

Re: Partition eviction failed, this can cause grid hang. (Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted))

2017-12-26 Thread Denis Magda

Cross-posting to the dev list.

Ignite persistence maintainers please chime in.

—
Denis

> On Dec 26, 2017, at 2:17 AM, Arseny Kovalchuk  
> wrote:
> 
> Hi guys.
> 
> Another issue when using Ignite 2.3 with native persistence enabled. See 
> details below.
> 
> We deploy Ignite along with our services in Kubernetes (v 1.8) on premises. 
> Ignite cluster is a StatefulSet of 5 Pods (5 instances) of Ignite version 
> 2.3. Each Pod mounts PersistentVolume backed by CEPH RBD. 
> 
> We put about 230 events/second into Ignite, 70% of events are ~200KB in size 
> and 30% are 5000KB. Smaller events have indexed fields and we query them via 
> SQL.
> 
> The cluster is activated from a client node which also streams events into 
> Ignite from Kafka. We use custom implementation of streamer which uses 
> cache.putAll() API.
> 
> We started cluster from scratch without any persistent data. After a while we 
> got corrupted data with the error message.
> 
> [2017-12-26 07:44:14,251] ERROR [sys-#127%ignite-instance-2%] 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader:
>  - Partition eviction failed, this can cause grid hang.
> class org.apache.ignite.IgniteException: Runtime failure on search row: 
> Row@5b1479d6[ key: 171:1513946618964:3008806055072854, val: 
> ru.synesis.kipod.event.KipodEvent [idHash=510912646, hash=-387621419, 
> face_last_name=null, face_list_id=null, channel=171, source=, 
> face_similarity=null, license_plate_number=null, descriptors=null, 
> cacheName=kipod_events, cacheKey=171:1513946618964:3008806055072854, 
> stream=171, alarm=false, processed_at=0, face_id=null, id=3008806055072854, 
> persistent=false, face_first_name=null, license_plate_first_name=null, 
> face_full_name=null, level=0, module=Kpx.Synesis.Outdoor, 
> end_time=1513946624379, params=null, commented_at=0, tags=[vehicle, 0, human, 
> 0, truck, 0, start_time=1513946618964, processed=false, kafka_offset=111259, 
> license_plate_last_name=null, armed=false, license_plate_country=null, 
> topic=MovingObject, comment=, expiration=1514033024000, original_id=null, 
> license_plate_lists=null], ver: GridCacheVersion [topVer=125430590, 
> order=1513955001926, nodeOrder=3] ][ 3008806055072854, MovingObject, 
> Kpx.Synesis.Outdoor, 0, , 1513946618964, 1513946624379, 171, 171, FALSE, 
> FALSE, , FALSE, FALSE, 0, 0, 111259, 1514033024000, (vehicle, 0, human, 0, 
> truck, 0), null, null, null, null, null, null, null, null, null, null, null, 
> null ]
>   at 
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1787)
>   at 
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.remove(BPlusTree.java:1578)
>   at 
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.remove(H2TreeIndex.java:216)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:496)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:580)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2334)
>   at 
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:461)
>   at 
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1453)
>   at 
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1416)
>   at 
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1271)
>   at 
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:374)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3233)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:588)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:951)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryEvict(GridDhtLocalPartition.java:809)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:593)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:580)
>   at 
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6631)
>   at 
>

Re: Ignite Committership Bar Guidance

2017-12-26 Thread Ilya Lantukh

Denis,

I presume that number of committers in Ignite doesn't grow as fast as
amount of contributions, so it requires additional effort to keep track on
each individual's progress.

So yes, I'd like to propose my candidacy since I'm both willing and able to
participate in development of Apache Ignite and it's community.

On Sun, Dec 24, 2017 at 6:04 AM, Denis Magda  wrote:

> Hi Ilya,
>
> Basically one of committers who usually reviews and merges code should
> spot that it’s time to discuss a candidacy.
>
> Anyway, if a contributor feels that committers don’t keep an eye on
> his/her progress the contributor can remind about personal existence by
> sending a message to @dev. ;)
>
> —
> Denis
>
> > On Dec 23, 2017, at 3:58 AM, Ilya Lantukh  wrote:
> >
> > Hi Denis,
> > Who should take the initiative in promotion? As a contributor who meets
> > those criteria, should I send my application to the PMC or just wait?
> >
> > On Fri, Dec 22, 2017 at 8:24 PM, Denis Magda  wrote:
> >
> >> Hi Vyacheslav,
> >>
> >> Some common sense, I presume. A committer who will be proposing a
> >> candidate can use JIRA to see a total number of resolved tickets and Git
> >> history to get insights on a complexity of every contribution.
> >>
> >> Do you have any suggestions on this?
> >>
> >> —
> >> Denis
> >>
> >>> On Dec 22, 2017, at 12:56 AM, Vyacheslav Daradur 
> >> wrote:
> >>>
> >>> Hi, Denis, how will the complexity of tasks be estimated?
> >>>
> >>>
> >>> On Fri, Dec 22, 2017 at 3:16 AM, Denis Magda 
> wrote:
>  Igniters,
> 
>  We at Ignite PMC put together a page with rules-of-thumb on when it’s
> a
> >> right time to promote a contributor to a committer:
>  https://cwiki.apache.org/confluence/display/IGNITE/
> >> Committership+Bar+Guidance  >> confluence/display/IGNITE/Committership+Bar+Guidance>
> 
>  It’s intended to make “contributor to committer” process more evident
> >> and simplify Ignite PMC voting process on this matter.
> 
>  Please use it as a reference, comments and suggestions are welcomed!
> 
>  —
>  Denis
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Vyacheslav D.
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Ilya
>
>


-- 
Best regards,
Ilya

Do we need ML profile in AI 2.4?

2017-12-26 Thread Denis Magda

Ignite ML contributors, 

Considering that Java 8 becomes default JDK in AI 2.4, do we need to keep a 
separate “ML” profile?

I vote to discontinue it and have all the ML examples under the general 
examples directory:
https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples

—
Denis

[jira] [Created] (IGNITE-7318) Move all SQL examples out of "data grid" package

2017-12-26 Thread Denis Magda (JIRA)

Denis Magda created IGNITE-7318:
---

 Summary: Move all SQL examples out of "data grid" package
 Key: IGNITE-7318
 URL: https://issues.apache.org/jira/browse/IGNITE-7318
 Project: Ignite
  Issue Type: Task
Reporter: Denis Magda
Assignee: Denis Magda
Priority: Blocker
 Fix For: 2.4






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3288: IGNITE-4585 Support CLEAR command with REST

2017-12-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3288


---

[jira] [Created] (IGNITE-7317) Make SVM Linear example for binary classification

2017-12-26 Thread Aleksey Zinoviev (JIRA)

Aleksey Zinoviev created IGNITE-7317:


 Summary: Make SVM Linear example for binary classification
 Key: IGNITE-7317
 URL: https://issues.apache.org/jira/browse/IGNITE-7317
 Project: Ignite
  Issue Type: Sub-task
  Components: ml
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev


Play with params and different datasets in example
Optional: it could be compared with kNN classification method



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7316) Make Linear SVM for binary classification

2017-12-26 Thread Aleksey Zinoviev (JIRA)

Aleksey Zinoviev created IGNITE-7316:


 Summary: Make Linear SVM for binary classification
 Key: IGNITE-7316
 URL: https://issues.apache.org/jira/browse/IGNITE-7316
 Project: Ignite
  Issue Type: Sub-task
  Components: ml
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev


It should contain
# dataset for tests
# loss function
# binary classification metric (ROC AUC, for example)
# Common SVM model
# SVM Linear BInary Trainer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

RE: Runtime.availableProcessors() returns hardware's CPU count whichis the issue with Ignite in Kubernetes

2017-12-26 Thread Stanislav Lukyanov

Hi Arseny,

Both OpenJDK and Oracle JDK 8u151 should do.
I’ve checked and it seems that it is indeed about the specific way of invoking 
Docker. By default it doesn’t restrict container to particular CPUs, but you 
can make it do that by using `--cpuset-cpus` flag.

Example:
Without cpuset (Docker host has 4 CPUs)
>docker run -it java-docker bash
root@8a2cd9d06695:/usr/test# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
root@8a2cd9d06695:/usr/test# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
4

With cpuset specifying the first two CPUs
>docker run -it --cpuset-cpus=0,1 java-docker bash
root@7c2723a9819e:/usr/test# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
2

Note also that by using cpuset you change the settings for the whole container, 
not just for the Ignite, so that other Java pools in the same JVM, e.g. 
parallel stream executor, should also work better.

Unfortunately, I’m not familiar with Kubernetes, but the manual [1] says that 
you can enable the use of cpusets via a static CPU management policy.

Hope that helps,
Stan

[1] https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/

From: Arseny Kovalchuk
Sent: 26 декабря 2017 г. 17:37
To: u...@ignite.apache.org
Cc: dev@ignite.apache.org
Subject: Re: Runtime.availableProcessors() returns hardware's CPU count whichis 
the issue with Ignite in Kubernetes

Hi Stanislav.

We use OpenJDK and use Alpine Linux based images. See java version below. In 
our environment availableProcessors returns CPU's for the host. 

Did you mean to try Oracle's JDK 8u151?
arseny@kovalchuka-ubuntu:~/kipod-x$ ku exec ignite-instance-0 -ti bash
bash-4.4# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
bash-4.4# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
40
jjs> 

Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: +375 (29) 666-16-16
LinkedIn Profile

On 26 December 2017 at 16:56, Yakov Zhdanov  wrote:
Ilya, agree. I like IGNITE_AVAILABLE_CPU more.

Yakov Zhdanov,
www.gridgain.com

2017-12-26 16:36 GMT+03:00 Ilya Lantukh :
Hi Yakov,

I think that property IGNITE_NODES_PER_HOST, as you suggested, would be 
confusing, because users might want to reduce amount of available resources for 
ignite node not only because they run multiple nodes per host, but also because 
they run other software. Also, in my opinion all types of system resources 
(CPU, memory, network) shouldn't be scaled using the same value.

So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or IGNITE_AVAILABLE_PROCESSORS, 
as it was originally suggested.

On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov  wrote:
Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk :

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on 
> Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> 
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: +375 (29) 666-16-16
> LinkedIn Profile 
>

-- 
Best regards,
Ilya

[GitHub] ignite pull request #3183: IGNITE-6915: Fixed b+tree index names for non-bin...

2017-12-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3183


---

[jira] [Created] (IGNITE-7315) Use java.util.Base64 instead of current implementations of Base64Encoder interface

2017-12-26 Thread Andrey Gura (JIRA)

Andrey Gura created IGNITE-7315:
---

 Summary: Use java.util.Base64 instead of current implementations 
of Base64Encoder interface
 Key: IGNITE-7315
 URL: https://issues.apache.org/jira/browse/IGNITE-7315
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrey Gura
 Fix For: 2.4


In order to provide temp solution that is acceptable for Java 7 and Java 8+ a 
couple of {{Base64Encoder}} interface were introduced (see IGNITE-6734).

This implementations should be replaced by {{java.util.Base64}} after Java 7 
drop.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Runtime.availableProcessors() returns hardware's CPU count which is the issue with Ignite in Kubernetes

2017-12-26 Thread Arseny Kovalchuk

Hi Stanislav.

We use OpenJDK and use Alpine Linux based images. See java version below.
In our environment availableProcessors returns CPU's for the host.

Did you mean to try Oracle's JDK 8u151?

arseny@kovalchuka-ubuntu:~/kipod-x$ ku exec ignite-instance-0 -ti bash
bash-4.4# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
bash-4.4# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
40
jjs>



Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: +375 (29) 666-16-16
LinkedIn Profile 

On 26 December 2017 at 16:56, Yakov Zhdanov  wrote:

> Ilya, agree. I like IGNITE_AVAILABLE_CPU more.
>
> Yakov Zhdanov,
> www.gridgain.com
>
> 2017-12-26 16:36 GMT+03:00 Ilya Lantukh :
>
>> Hi Yakov,
>>
>> I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
>> confusing, because users might want to reduce amount of available resources
>> for ignite node not only because they run multiple nodes per host, but also
>> because they run other software. Also, in my opinion all types of system
>> resources (CPU, memory, network) shouldn't be scaled using the same value.
>>
>> So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
>> IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.
>>
>> On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov 
>> wrote:
>>
>>> Cross-posting to dev list.
>>>
>>> Guys,
>>>
>>> Suggestion below makes sense to me. Filed a ticket
>>> https://issues.apache.org/jira/browse/IGNITE-7310
>>>
>>> Perhaps, Arseny would like to provide a PR himself ;)
>>>
>>> --Yakov
>>>
>>> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk >> >:
>>>
>>> > Hi guys.
>>> >
>>> > Ignite configures all thread pools, selectors, etc. basing on
>>> Runtime.availableProcessors()
>>> > which seems not correct in containerized environment. In Kubernetes
>>> with
>>> > Docker that method returns CPU count of a Node/machine, which is 64 in
>>> our
>>> > particular case. But those 64 CPU and their timings are shared between
>>> > other stuff on the node like other Pods and services. Appropriate
>>> value of
>>> > available cores for Pod is usually configured as CPU Resource and
>>> estimated
>>> > basing on different things taking performance into account. General
>>> idea,
>>> > if you want to run several Pods on the same node, they all should
>>> request
>>> > less resources then the node provides. So, we give 4-8 cores for Ignite
>>> > instance in Kubernetes, but Ignite's thread pools are configured like
>>> they
>>> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with
>>> 4-8
>>> > cores available.
>>> >
>>> > Now we manually set appropriate values for all available properties
>>> which
>>> > relate to thread pools.
>>> >
>>> > Would it be correct to have one environment variable, say
>>> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
>>> those
>>> > configurations and by default equals to Runtime.availableProcessors()?
>>> >
>>> > Thanks.
>>> >
>>> > 
>>> > Arseny Kovalchuk
>>> >
>>> > Senior Software Engineer at Synesis
>>> > skype: arseny.kovalchuk
>>> > mobile: +375 (29) 666-16-16 <+375%2029%20666-16-16>
>>> > LinkedIn Profile 
>>> >
>>>
>>
>>
>>
>> --
>> Best regards,
>> Ilya
>>
>
>

Re: List of running Continuous queries or CacheEntryListener per cache or node

2017-12-26 Thread Dmitry Karachentsev


Hi Nikolay,

I think it may be useful too. Will try to describe possible API in a ticket.

Thanks!
-Dmitry

21.12.2017 13:18, Nikolay Izhikov пишет:

Hello, Dmitry.

I think it a great idea.

Do we a feature to list all running ComputeTasks?

I, personally, think we have to implement possibility to track all
user-provided tasks - CacheListener, ContinuousQuery, ComputeTasks,
etc.

В Чт, 21/12/2017 в 10:13 +0300, Dmitry Karachentsev пишет:

Crossposting to devlist.

Hi Igniters!

It's might be a nice feature to have - get list of registered
continuous
queries with ability to deregister them.

What do you think?

Thanks!
-Dmitry

20.12.2017 16:59, fefe пишет:

For sanity checks or tests. I want to be sure that I haven't forgot
to
deregister any listener.

Its also very important metric to see how many continuous
queries/listeners
are currently running.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[GitHub] ignite pull request #3291: IGNITE-7173 SQL: implement optional row cache

2017-12-26 Thread tledkov-gridgain

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/3291

IGNITE-7173  SQL: implement optional row cache



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7173

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3291


commit c383ff6fc4e84b6c1c21a37a18a52558e712fdc6
Author: tledkov-gridgain 
Date:   2017-12-26T12:18:31Z

IGNITE-7173: save the progress

commit d3e6ee2c137c41daa8069ef7382e6b63f76951dc
Author: tledkov-gridgain 
Date:   2017-12-26T12:25:44Z

IGNITE-7173: save the progress

commit 551312877afeecadff115a1a080ca18a0404401f
Author: tledkov-gridgain 
Date:   2017-12-26T12:31:10Z

IGNITE-7173: save the progress

commit 9d4edbce23979ffb94a6f9e90610d1022e6f8e84
Author: tledkov-gridgain 
Date:   2017-12-26T15:02:48Z

IGNITE-7173: save the progress




---

Re: Handling slashes in cache names

2017-12-26 Thread Igor Sapego

There are also some international features that you might want to
address. For example, instead of backslash some other characters
may be used on Windows - ¥ on the Japanese version, ₩ on the
Korean version.
See [1] for more info.

Here is the citation:
Security Considerations for Character Sets in File Names
Windows code page and OEM character sets used on
Japanese-language systems contain the Yen symbol (¥) instead of
a backslash (\). Thus, the Yen character is a prohibited character for
NTFS and FAT file systems. When mapping Unicode to
a Japanese-language code page, conversion functions map both
backslash (U+005C) and the normal Unicode Yen symbol (U+00A5)
to this same character. For security reasons, your applications should
not typically allow the character U+00A5 in a Unicode string that
might be converted for use as a FAT file name.

[1] - https://msdn.microsoft.com/en-us/library/dd374047(v=vs.85).aspx


Best Regards,
Igor

On Tue, Dec 26, 2017 at 5:01 PM, Dmitriy Setrakyan 
wrote:

> Looks good to me. Is this going to be an exception on startup? If yes, is
> it safe to release it, or should we wait till 3.0?
>
> On Tue, Dec 26, 2017 at 2:08 AM, Stanislav Lukyanov <
> stanlukya...@gmail.com>
> wrote:
>
> > Thanks for the feedback.
> >
> > It seems that another thing to handle is case-insensitive FS – “mycache”
> > and “MyCache” is the same on Windows, so it might be reasonable to
> disallow
> > having two caches with names that are equal ignoring case.
> > And one more thing is control characters – forbidding at least range of
> > ASCII 0x00-0x20 seems reasonable.
> >
> > To summarize, a possible set of restrictions would be
> > - Whitespace characters (via Character.isWhitespaceCharacter)
> > - Control characters (via Character.isISOCharacter)
> > - Slashes
> > - Characters reserved in Windows (<>:"/\|?*)
> > - Length (say, up to 255)
> > - Distinct names of caches when ignoring case
> > It seems reasonable to enforce that even regardless of persistence
> > directories naming (AFAIU that’s what Dmitry meant by forbidding things
> > altogether), so that’s what I’m going to do.
> > Any concerns?
> > Specifically, would it be OK from backward compatibility point of view to
> > forbid all these characters now for all caches?
> >
> > Thanks,
> > Stan
> >
> >
> > From: Alexey Kuznetsov
> > Sent: 26 декабря 2017 г. 7:51
> > To: dev@ignite.apache.org
> > Subject: Re: Handling slashes in cache names
> >
> > It also make sense to limit cache name length to reasonable length.
> > Because some File systems could have limitations on path length.
> > See: https://en.wikipedia.org/wiki/Filename#Length_restrictions
> >
> > On Tue, Dec 26, 2017 at 1:41 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > My preference would be to prohibit forward and backward slashes in
> cache
> > > names altogether, as they may create a false feeling of some directory
> > > structure, which does not exist. We should also prohibit spaces as
> well.
> > >
> > > D.
> > >
> > > On Mon, Dec 25, 2017 at 7:09 AM, Stanislav Lukyanov <
> > > stanlukya...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I’m looking into https://issues.apache.org/jira/browse/IGNITE-7264,
> > and
> > > I
> > > > need some guidance on what’s the best way to approach it.
> > > >
> > > > The problem is that cache names are not restricted, but if
> persistence
> > is
> > > > enabled the cache needs to have a corresponding directory on the file
> > > > system (“cache-…”) which can’t be created if the cache name contains
> > > > certain characters (or a reserved system name).
> > > >
> > > > A straightforward approach would be to check if a cache name is
> allowed
> > > on
> > > > the local system (e.g. via `Paths.get(name)`) and fail to create
> cache
> > if
> > > > it isn’t, but I’m a bit concerned with the consistency of the
> behavior
> > > (the
> > > > same cache name be allowed on one system and not on another).
> > > > I think a better way would be to replace special characters (say, all
> > > > non-alphanumeric characters) with underscores in file names (not
> > changing
> > > > the cache configuration). Would this be OK? Are there any risks I’m
> not
> > > > considering?
> > > >
> > > > WDYT?
> > > >
> > > > Thanks,
> > > > Stan
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> >
> >
>

[jira] [Created] (IGNITE-7314) IgniteCacheFullApiSelfTestSuite is extremely slow with enabled MVCC

2017-12-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-7314:
---

 Summary: IgniteCacheFullApiSelfTestSuite is extremely slow with 
enabled MVCC
 Key: IGNITE-7314
 URL: https://issues.apache.org/jira/browse/IGNITE-7314
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Vladimir Ozerov
 Fix For: 2.5


It MVCC is enabled most of tests in {{IgniteCacheFullApiSelfTestSuite}} works 
fine. However, they are really *slow*. So slow, that TC cannot cope with them 
and kills the build after 1h30m of execution.

Need to understand what is the root cause of this dramatic slowdown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7313) Recovery process doesn't propagate MVCC version

2017-12-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-7313:
---

 Summary: Recovery process doesn't propagate MVCC version
 Key: IGNITE-7313
 URL: https://issues.apache.org/jira/browse/IGNITE-7313
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Vladimir Ozerov
 Fix For: 2.5


Reproducer: 
{{IgnitePdsContinuousRestartTest.testRebalancingDuringLoad_8000_8000_8_16}}
Root cause: MVCC version is not passed during recovery process, as a result we 
cannot commit transaction properly.

Stack trace:
{code}
[2017-12-26 
17:44:03,011][ERROR][sys-stripe-5-#216%persistence.IgnitePdsContinuousRestartTest3%][G]
 Failed to execute runnable.
java.lang.AssertionError: Mvcc is not initialized: GridDhtTxRemote 
[nearNodeId=5213f13a-541e-41e0-ac30-c0cdc9d0, 
rmtFutId=ea4b6a39061-cf384028-0bcb-46d8-92e1-3c898390d074, 
nearXidVer=GridCacheVersion [topVer=125779403, order=1514300484265, 
nodeOrder=1], storeWriteThrough=false, super=GridDistributedTxRemoteAdapter 
[explicitVers=null, started=true, commitAllowed=1, 
txState=IgniteTxRemoteStateImpl [readMap={}, writeMap={IgniteTxKey 
[key=KeyCacheObjectImpl [part=113, val=2545, hasValBytes=true], 
cacheId=-1368047377]=IgniteTxEntry [key=KeyCacheObjectImpl [part=113, val=2545, 
hasValBytes=true], cacheId=-1368047377, txKey=IgniteTxKey 
[key=KeyCacheObjectImpl [part=113, val=2545, hasValBytes=true], 
cacheId=-1368047377], val=[op=CREATE, val=CacheObjectImpl [val=null, 
hasValBytes=true]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], 
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, 
explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, 
filtersSet=false, entry=GridDhtCacheEntry [rdrs=[], part=113, 
super=GridDistributedCacheEntry [super=GridCacheMapEntry 
[key=KeyCacheObjectImpl [part=113, val=2545, hasValBytes=true], 
val=CacheObjectImpl [val=null, hasValBytes=true], startVer=1514300484409, 
ver=GridCacheVersion [topVer=125779403, order=1514300455870, nodeOrder=2], 
hash=2545, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=null, 
rmts=[GridCacheMvccCandidate [nodeId=91bcaf65-b816-41e9-a74e-ca58c821, 
ver=GridCacheVersion [topVer=125779403, order=1514300484266, nodeOrder=2], 
threadId=285, id=1158826, topVer=AffinityTopologyVersion [topVer=-1, 
minorTopVer=0], reentry=null, otherNodeId=5213f13a-541e-41e0-ac30-c0cdc9d0, 
otherVer=null, mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, 
serOrder=null, key=KeyCacheObjectImpl [part=113, val=2545, hasValBytes=true], 
masks=local=0|owner=1|ready=0|reentry=0|used=1|tx=1|single_implicit=0|dht_local=0|near_local=0|removed=0|read=0,
 prevVer=null, nextVer=null, flags=2]]], prepared=1, locked=false, 
nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, 
partUpdateCntr=0, serReadVer=null, xidVer=null], IgniteTxKey 
[key=KeyCacheObjectImpl [part=17, val=3089, hasValBytes=true], 
cacheId=-1368047377]=IgniteTxEntry [key=KeyCacheObjectImpl [part=17, val=3089, 
hasValBytes=true], cacheId=-1368047377, txKey=IgniteTxKey 
[key=KeyCacheObjectImpl [part=17, val=3089, hasValBytes=true], 
cacheId=-1368047377], val=[op=CREATE, val=CacheObjectImpl [val=null, 
hasValBytes=true]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], 
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, 
explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, 
filtersSet=false, entry=GridDhtCacheEntry [rdrs=[], part=17, 
super=GridDistributedCacheEntry [super=GridCacheMapEntry 
[key=KeyCacheObjectImpl [part=17, val=3089, hasValBytes=true], 
val=CacheObjectImpl [val=null, hasValBytes=true], startVer=1514300484410, 
ver=GridCacheVersion [topVer=125779403, order=1514300459131, nodeOrder=2], 
hash=3089, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=null, 
rmts=[GridCacheMvccCandidate [nodeId=91bcaf65-b816-41e9-a74e-ca58c821, 
ver=GridCacheVersion [topVer=125779403, order=1514300484266, nodeOrder=2], 
threadId=285, id=1158827, topVer=AffinityTopologyVersion [topVer=-1, 
minorTopVer=0], reentry=null, otherNodeId=5213f13a-541e-41e0-ac30-c0cdc9d0, 
otherVer=null, mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, 
serOrder=null, key=KeyCacheObjectImpl [part=17, val=3089, hasValBytes=true], 
masks=local=0|owner=1|ready=0|reentry=0|used=1|tx=1|single_implicit=0|dht_local=0|near_local=0|removed=0|read=0,
 prevVer=null, nextVer=null, flags=2]]], prepared=1, locked=false, 
nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, 
partUpdateCntr=0, serReadVer=null, xidVer=null], IgniteTxKey 
[key=KeyCacheObjectImpl [part=35, val=5155, hasValBytes=true], 
cacheId=-1368047377]=IgniteTxEntry [key=KeyCacheObjectImpl [part=35, val=5155, 
hasValBytes=true], cacheId=-1368047377, txKey=IgniteTxKey 
[key=KeyCacheObjectImpl [part=35, val=5155, hasValBytes=true], 
cacheId=-1368047377],

[jira] [Created] (IGNITE-7312) Make use of plain java.util.Base64 instead of reflective alternatives

2017-12-26 Thread Andrey Kuznetsov (JIRA)

Andrey Kuznetsov created IGNITE-7312:


 Summary: Make use of plain java.util.Base64 instead of reflective 
alternatives
 Key: IGNITE-7312
 URL: https://issues.apache.org/jira/browse/IGNITE-7312
 Project: Ignite
  Issue Type: Task
  Components: general
Reporter: Andrey Kuznetsov


Reflective Base64 encoding should be changed to {{java.util.Base64$Encoder}} as 
soon as language level becomes equal to 8.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3206: IGNITE-6743: Cleaning direct buffer through refle...

2017-12-26 Thread andrey-kuznetsov

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3206


---

[GitHub] ignite pull request #3197: Ignite 6734

2017-12-26 Thread andrey-kuznetsov

GitHub user andrey-kuznetsov reopened a pull request:

https://github.com/apache/ignite/pull/3197

Ignite 6734



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6734

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3197


commit aff7c61f66ed0ba8de051f8ab037aa8f3076f167
Author: Andrey Kuznetsov 
Date:   2017-12-11T12:04:45Z

IGNITE-6734: Switched language level to 8 for entire project.

commit cac8bcc74fc2075a75a74c445757ccce9f86c4f2
Author: Andrey Kuznetsov 
Date:   2017-12-11T12:08:34Z

IGNITE-6734: Replaced sun.misc.BASE64Encoder with standard Java 8 encoder.

commit 9e310825570314afda08cc4d2b2996cdd7aee15d
Author: Andrey Kuznetsov 
Date:   2017-12-26T13:38:03Z

Merge branch 'master' into ignite-6734

commit 5c7913b685eaf659dccec68a20c704d7f2c81ac2
Author: Andrey Kuznetsov 
Date:   2017-12-26T14:08:09Z

IGNITE-6734: Changed to reflective Base64 encoding.




---

Re: IGNITE-7135 needs review

2017-12-26 Thread Dmitriy Setrakyan

Here is the link to the ticket:
https://issues.apache.org/jira/browse/IGNITE-7135

For some odd reason, the ticket is in unassigned state. Alexander,
shouldn't it be assigned to you?

D.

On Mon, Dec 25, 2017 at 11:58 PM, Alexandr Kuramshin 
wrote:

> Hello community!
>
> I've implemented IGNITE-7135 doing two improvements:
>
> 1) control remote node startup (successful or not) through
> IgniteCluster.startNodes();
>
> 2) keep the first Java principle working "Compile once, run everywhere" -
> from now running remotely on Windows also supported.
>
> Committers, please review.
>
> --
> Thanks,
> Alexandr Kuramshin
>

[jira] [Created] (IGNITE-7311) Expiration does not work with enabled MVCC

2017-12-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-7311:
---

 Summary: Expiration does not work with enabled MVCC
 Key: IGNITE-7311
 URL: https://issues.apache.org/jira/browse/IGNITE-7311
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Vladimir Ozerov
 Fix For: 2.5


Reproducer: {{RedisProtocolStringSelfTest#testExpire}}
Root cause: MVCC is not honored when updating entry TTL.

{code}
Exception in thread "pub-#101%redis.RedisProtocolStringSelfTest0%" 
java.lang.AssertionError: Assertion error on search row: SearchRow 
[key=KeyCacheObjectImpl [part=294, val=k1, hasValBytes=true], hash=3366, 
cacheId=0]
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1749)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1337)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:371)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3249)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:2391)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:3191)
at 
org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler$UpdateTllCommand$1$1.process(GridCacheCommandHandler.java:1685)
at 
org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler$UpdateTllCommand$1$1.process(GridCacheCommandHandler.java:1675)
at 
org.apache.ignite.internal.processors.cache.EntryProcessorResourceInjectorProxy.process(EntryProcessorResourceInjectorProxy.java:68)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onEntriesLocked(GridDhtTxPrepareFuture.java:432)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1271)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:683)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1061)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:395)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:519)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTxLocal(IgniteTxHandler.java:320)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.proceedPrepare(GridNearOptimisticTxPrepareFuture.java:604)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepareSingle(GridNearOptimisticTxPrepareFuture.java:410)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepare0(GridNearOptimisticTxPrepareFuture.java:341)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:158)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:95)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3323)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3390)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.optimisticPutFuture(GridNearTxLocal.java:2550)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAsync0(GridNearTxLocal.java:646)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.invokeAsync(GridNearTxLocal.java:431)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter$25.op(GridCacheAdapter.java:2481)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter$25.op(GridCacheAdapter.java:2473)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4085)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.invoke0(GridCacheAdapter.java:2473)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.invoke(GridCacheAdapter.java:2451)
at

Re: Handling slashes in cache names

2017-12-26 Thread Dmitriy Setrakyan

Looks good to me. Is this going to be an exception on startup? If yes, is
it safe to release it, or should we wait till 3.0?

On Tue, Dec 26, 2017 at 2:08 AM, Stanislav Lukyanov 
wrote:

> Thanks for the feedback.
>
> It seems that another thing to handle is case-insensitive FS – “mycache”
> and “MyCache” is the same on Windows, so it might be reasonable to disallow
> having two caches with names that are equal ignoring case.
> And one more thing is control characters – forbidding at least range of
> ASCII 0x00-0x20 seems reasonable.
>
> To summarize, a possible set of restrictions would be
> - Whitespace characters (via Character.isWhitespaceCharacter)
> - Control characters (via Character.isISOCharacter)
> - Slashes
> - Characters reserved in Windows (<>:"/\|?*)
> - Length (say, up to 255)
> - Distinct names of caches when ignoring case
> It seems reasonable to enforce that even regardless of persistence
> directories naming (AFAIU that’s what Dmitry meant by forbidding things
> altogether), so that’s what I’m going to do.
> Any concerns?
> Specifically, would it be OK from backward compatibility point of view to
> forbid all these characters now for all caches?
>
> Thanks,
> Stan
>
>
> From: Alexey Kuznetsov
> Sent: 26 декабря 2017 г. 7:51
> To: dev@ignite.apache.org
> Subject: Re: Handling slashes in cache names
>
> It also make sense to limit cache name length to reasonable length.
> Because some File systems could have limitations on path length.
> See: https://en.wikipedia.org/wiki/Filename#Length_restrictions
>
> On Tue, Dec 26, 2017 at 1:41 AM, Dmitriy Setrakyan 
> wrote:
>
> > My preference would be to prohibit forward and backward slashes in cache
> > names altogether, as they may create a false feeling of some directory
> > structure, which does not exist. We should also prohibit spaces as well.
> >
> > D.
> >
> > On Mon, Dec 25, 2017 at 7:09 AM, Stanislav Lukyanov <
> > stanlukya...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I’m looking into https://issues.apache.org/jira/browse/IGNITE-7264,
> and
> > I
> > > need some guidance on what’s the best way to approach it.
> > >
> > > The problem is that cache names are not restricted, but if persistence
> is
> > > enabled the cache needs to have a corresponding directory on the file
> > > system (“cache-…”) which can’t be created if the cache name contains
> > > certain characters (or a reserved system name).
> > >
> > > A straightforward approach would be to check if a cache name is allowed
> > on
> > > the local system (e.g. via `Paths.get(name)`) and fail to create cache
> if
> > > it isn’t, but I’m a bit concerned with the consistency of the behavior
> > (the
> > > same cache name be allowed on one system and not on another).
> > > I think a better way would be to replace special characters (say, all
> > > non-alphanumeric characters) with underscores in file names (not
> changing
> > > the cache configuration). Would this be OK? Are there any risks I’m not
> > > considering?
> > >
> > > WDYT?
> > >
> > > Thanks,
> > > Stan
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
>
>

Re: Runtime.availableProcessors() returns hardware's CPU count which is the issue with Ignite in Kubernetes

2017-12-26 Thread Yakov Zhdanov

Ilya, agree. I like IGNITE_AVAILABLE_CPU more.

Yakov Zhdanov,
www.gridgain.com

2017-12-26 16:36 GMT+03:00 Ilya Lantukh :

> Hi Yakov,
>
> I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
> confusing, because users might want to reduce amount of available resources
> for ignite node not only because they run multiple nodes per host, but also
> because they run other software. Also, in my opinion all types of system
> resources (CPU, memory, network) shouldn't be scaled using the same value.
>
> So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
> IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.
>
> On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov 
> wrote:
>
>> Cross-posting to dev list.
>>
>> Guys,
>>
>> Suggestion below makes sense to me. Filed a ticket
>> https://issues.apache.org/jira/browse/IGNITE-7310
>>
>> Perhaps, Arseny would like to provide a PR himself ;)
>>
>> --Yakov
>>
>> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk > >:
>>
>> > Hi guys.
>> >
>> > Ignite configures all thread pools, selectors, etc. basing on
>> Runtime.availableProcessors()
>> > which seems not correct in containerized environment. In Kubernetes with
>> > Docker that method returns CPU count of a Node/machine, which is 64 in
>> our
>> > particular case. But those 64 CPU and their timings are shared between
>> > other stuff on the node like other Pods and services. Appropriate value
>> of
>> > available cores for Pod is usually configured as CPU Resource and
>> estimated
>> > basing on different things taking performance into account. General
>> idea,
>> > if you want to run several Pods on the same node, they all should
>> request
>> > less resources then the node provides. So, we give 4-8 cores for Ignite
>> > instance in Kubernetes, but Ignite's thread pools are configured like
>> they
>> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with
>> 4-8
>> > cores available.
>> >
>> > Now we manually set appropriate values for all available properties
>> which
>> > relate to thread pools.
>> >
>> > Would it be correct to have one environment variable, say
>> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
>> those
>> > configurations and by default equals to Runtime.availableProcessors()?
>> >
>> > Thanks.
>> >
>> > 
>> > Arseny Kovalchuk
>> >
>> > Senior Software Engineer at Synesis
>> > skype: arseny.kovalchuk
>> > mobile: +375 (29) 666-16-16
>> > LinkedIn Profile 
>> >
>>
>
>
>
> --
> Best regards,
> Ilya
>

Re: Runtime.availableProcessors() returns hardware's CPU count which is the issue with Ignite in Kubernetes

2017-12-26 Thread Ilya Lantukh

Hi Yakov,

I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
confusing, because users might want to reduce amount of available resources
for ignite node not only because they run multiple nodes per host, but also
because they run other software. Also, in my opinion all types of system
resources (CPU, memory, network) shouldn't be scaled using the same value.

So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.

On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov  wrote:

> Cross-posting to dev list.
>
> Guys,
>
> Suggestion below makes sense to me. Filed a ticket
> https://issues.apache.org/jira/browse/IGNITE-7310
>
> Perhaps, Arseny would like to provide a PR himself ;)
>
> --Yakov
>
> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk :
>
> > Hi guys.
> >
> > Ignite configures all thread pools, selectors, etc. basing on
> Runtime.availableProcessors()
> > which seems not correct in containerized environment. In Kubernetes with
> > Docker that method returns CPU count of a Node/machine, which is 64 in
> our
> > particular case. But those 64 CPU and their timings are shared between
> > other stuff on the node like other Pods and services. Appropriate value
> of
> > available cores for Pod is usually configured as CPU Resource and
> estimated
> > basing on different things taking performance into account. General idea,
> > if you want to run several Pods on the same node, they all should request
> > less resources then the node provides. So, we give 4-8 cores for Ignite
> > instance in Kubernetes, but Ignite's thread pools are configured like
> they
> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> > cores available.
> >
> > Now we manually set appropriate values for all available properties which
> > relate to thread pools.
> >
> > Would it be correct to have one environment variable, say
> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
> those
> > configurations and by default equals to Runtime.availableProcessors()?
> >
> > Thanks.
> >
> > 
> > Arseny Kovalchuk
> >
> > Senior Software Engineer at Synesis
> > skype: arseny.kovalchuk
> > mobile: +375 (29) 666-16-16
> > LinkedIn Profile 
> >
>



-- 
Best regards,
Ilya

RE: Runtime.availableProcessors() returns hardware's CPU count whichis the issue with Ignite in Kubernetes

2017-12-26 Thread Stanislav Lukyanov

Hi Arseny,

This behavior of the `Runtime.availableProcessors()` is actually a recognized 
issue of the Hotspot, see [1]. It was fixed not that long ago for JDK 9 and 
8uX, and I can see correct values returned in my Docker environment with JDK 
8u151, although I believe it depends on a specific way a container is 
configured.

Which Java version do you use? Can you try your code on JDK 8u151?

BTW see also [2] and [3] on more stuff to be fixed in JDK for better container 
support.

Thanks,
Stan

[1] https://bugs.openjdk.java.net/browse/JDK-6515172
[2] https://bugs.openjdk.java.net/browse/JDK-8146115
[3] https://bugs.openjdk.java.net/browse/JDK-8182070


From: Yakov Zhdanov
Sent: 26 декабря 2017 г. 16:05
To: u...@ignite.apache.org; dev@ignite.apache.org
Subject: Re: Runtime.availableProcessors() returns hardware's CPU count whichis 
the issue with Ignite in Kubernetes

Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk :

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on 
> Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> 
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: +375 (29) 666-16-16
> LinkedIn Profile 
>

Re: Runtime.availableProcessors() returns hardware's CPU count which is the issue with Ignite in Kubernetes

2017-12-26 Thread Yakov Zhdanov

Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk :

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on 
> Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> 
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: +375 (29) 666-16-16
> LinkedIn Profile 
>

[jira] [Created] (IGNITE-7310) Ignite should be easily configured to share hardware resources between instances

2017-12-26 Thread Yakov Zhdanov (JIRA)

Yakov Zhdanov created IGNITE-7310:
-

 Summary: Ignite should be easily configured to share hardware 
resources between instances
 Key: IGNITE-7310
 URL: https://issues.apache.org/jira/browse/IGNITE-7310
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov


This may be handy when working with containers or in test environments when 
starting more than 1 node per host.

Suggestion is to introduce Ignite system property IGNITE_NODES_PER_HOST which 
is 1 by default and calculate all resource related config parameters using this 
value, e.g. offheap memory, pool threads, nio workers in TcpCommunicationSpi, 
etc. dividing total amount of available resource by IGNITE_NODES_PER_HOST.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3285: IGNITE-7261 MVCC enabled

2017-12-26 Thread gvvinblade

Github user gvvinblade closed the pull request at:

https://github.com/apache/ignite/pull/3285


---

[GitHub] ignite pull request #3284: IGNITE-7261 MVCC disabled

2017-12-26 Thread gvvinblade

Github user gvvinblade closed the pull request at:

https://github.com/apache/ignite/pull/3284


---

[GitHub] ignite pull request #3197: Ignite 6734

2017-12-26 Thread andrey-kuznetsov

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3197


---

Re: IGNITE-7107 Apache Ignite RPM packages

2017-12-26 Thread Petr Ivanov

Removed replacement for default-config.xml.
Also reimplemented service to be able to run multiple instances of Ignite.

See updated PR [1].


[1] https://github.com/apache/ignite/pull/3171 




> On 25 Dec 2017, at 18:57, Pavel Tupitsyn  wrote:
> 
> PDS and discovery settings are unrelated to the packaging task.
> Andrey is right, just use existing default config.
> 
> On Mon, Dec 25, 2017 at 6:51 PM, Petr Ivanov  wrote:
> 
>> Can we change default configuration file then?
>> So that both binary and package deliveries contained PDS turned on by
>> default?
>> 
>> And what about Multicast Discovery?
>> 
>> 
>>> On 25 Dec 2017, at 18:41, Dmitriy Setrakyan 
>> wrote:
>>> 
>>> I agree with Andrey. The product should be identical regardless of how
>> you
>>> download and install it.
>>> 
>>> On Mon, Dec 25, 2017 at 7:18 AM, Andrey Gura  wrote:
>>> 
 Hi,
 
 I think we should provide the same default configuration for all
 binary builds. From my point of view RPM/DEB/etc packages should use
 default configuration file like to standard binary release.
 
 On Mon, Dec 25, 2017 at 4:39 PM, Ilya Kasnacheev
  wrote:
> Hello Igniters!
> 
> What's your take on enabling PDS in 2.4 in default config? This way we
> could enable it in RPM build with easy heart.
> 
> The rest of your answers seem spot on. I'll happily review your amended
> patch when it is ready (later this week?)
> 
> --
> Ilya Kasnacheev
> 
> 2017-12-22 18:27 GMT+03:00 vveider :
> 
>> Ilya Kasnacheev wrote
>>> I have noticed that both spec and DEVNOTES are made to use tabs
 alongside
>>> with spaces. I thought we are avoiding tabs? Please confirm that
 usage of
>>> tabs is desired in this case.
>> 
>> My fault - got used to editing such files in default vim. Fixed and
 updated
>> vim settings to indent with spaces.
>> 
>> 
>> Ilya Kasnacheev wrote
>>> Maybe it's my CentOS but initially build will fail with the following
>>> message, which I had to fix in spec
>>> + cd 'apache-ignite-*'
>>> /var/tmp/rpm-tmp.KwOoyl: line 34: cd: apache-ignite-*: No such file
>> or
>>> directory
>> 
>> Strange error - shell expansion is not working. Anyway, replaced with
 more
>> universal variant.
>> 
>> 
>> Ilya Kasnacheev wrote
>>> We absolutely should not inline configuration files (such as
>>> default-config.xml) into build scripts. We should just copy over
>>> default-config.xml from config/ to /etc, with dependencies if needed.
>> 
>> Extracted corresponding sections into separate files.
>> 
>> 
>> Ilya Kasnacheev wrote
>>> I'm not sure PDS should be ON by default. Better provide it in
 examples
>>> but disable by default.
>> 
>> AFAIK, PDS is actively pushing forward and has to be enabled by
>> default
 so
>> that Ignite users start working with PDS from the very beginning.
>> 
>> 
>> Ilya Kasnacheev wrote
>>> I cannot comment on firewall rules validity, but firewall and service
>>> configurations should be stored in sources, as files, supplied with
>>> release (somewhere in packages/) and installed from there
>> 
>> Unfortunately, I did not find any other way to open ports and
>> multicast,
>> except applying direct iptables rules though firewall-cmd -- so there
 can
>> be
>> no separate service firewall rules file (as can be found here:
>> /usr/lib/firewalld/services). Applied rules from package go straight
>> to
>> /etc/firewalld/direct.xml. If we agree not to put Multicast-enabled by
>> default configuration file and will stick to IP Discovery, I'll try to
>> revise firewall configuration and create separate service firewall
>> rules
>> file (but, I guess, much later).
>> 
>> 
>> Ilya Kasnacheev wrote
>>> * sqlline.sh fails to connect initially, needs to be specified
>>> jdbc:ignite:thin://localhost, then it works. I think that sqlline
 should
>>> become available as /usr/bin/apache-ignite-sqlline for example, to be
 in
>>> $PATH. And figure out how to connect by itself.
>>> * ignitevisorcmd.sh becomes confused. first it scans
>>> /usr/share/apache-ignite for configs, then it fails to write to
>>> /usr/share/apache-ignite/work. We should consider how it can be fixed
 (a
>>> symlink from /usr/share/apache-ignite/config to /etc/apache-ignite,
>>> perhaps, and work dir in /tmp?), and made available as
>>> /usr/bin/apache-ignite-visorcmd.
>> 
>> The problem exists mainly because of sqlline.sh|ignitevisorcmd.sh
>> unaware-of-package design and I see the following 3-step way to
>> overcome
>> this limitation.

RE: Handling slashes in cache names

2017-12-26 Thread Stanislav Lukyanov

Thanks for the feedback.

It seems that another thing to handle is case-insensitive FS – “mycache” and 
“MyCache” is the same on Windows, so it might be reasonable to disallow having 
two caches with names that are equal ignoring case.
And one more thing is control characters – forbidding at least range of ASCII 
0x00-0x20 seems reasonable.

To summarize, a possible set of restrictions would be
- Whitespace characters (via Character.isWhitespaceCharacter)
- Control characters (via Character.isISOCharacter)
- Slashes
- Characters reserved in Windows (<>:"/\|?*)
- Length (say, up to 255)
- Distinct names of caches when ignoring case
It seems reasonable to enforce that even regardless of persistence directories 
naming (AFAIU that’s what Dmitry meant by forbidding things altogether), so 
that’s what I’m going to do. 
Any concerns?
Specifically, would it be OK from backward compatibility point of view to 
forbid all these characters now for all caches?

Thanks,
Stan


From: Alexey Kuznetsov
Sent: 26 декабря 2017 г. 7:51
To: dev@ignite.apache.org
Subject: Re: Handling slashes in cache names

It also make sense to limit cache name length to reasonable length.
Because some File systems could have limitations on path length.
See: https://en.wikipedia.org/wiki/Filename#Length_restrictions

On Tue, Dec 26, 2017 at 1:41 AM, Dmitriy Setrakyan 
wrote:

> My preference would be to prohibit forward and backward slashes in cache
> names altogether, as they may create a false feeling of some directory
> structure, which does not exist. We should also prohibit spaces as well.
>
> D.
>
> On Mon, Dec 25, 2017 at 7:09 AM, Stanislav Lukyanov <
> stanlukya...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I’m looking into https://issues.apache.org/jira/browse/IGNITE-7264, and
> I
> > need some guidance on what’s the best way to approach it.
> >
> > The problem is that cache names are not restricted, but if persistence is
> > enabled the cache needs to have a corresponding directory on the file
> > system (“cache-…”) which can’t be created if the cache name contains
> > certain characters (or a reserved system name).
> >
> > A straightforward approach would be to check if a cache name is allowed
> on
> > the local system (e.g. via `Paths.get(name)`) and fail to create cache if
> > it isn’t, but I’m a bit concerned with the consistency of the behavior
> (the
> > same cache name be allowed on one system and not on another).
> > I think a better way would be to replace special characters (say, all
> > non-alphanumeric characters) with underscores in file names (not changing
> > the cache configuration). Would this be OK? Are there any risks I’m not
> > considering?
> >
> > WDYT?
> >
> > Thanks,
> > Stan
> >
>



-- 
Alexey Kuznetsov

[GitHub] ignite pull request #3223: IGNITE-7189: Fixed ODBC table names in metadata

2017-12-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3223


---

[GitHub] ignite pull request #3290: Ignite 7308

2017-12-26 Thread agoncharuk

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/3290

Ignite 7308



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7308

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3290


commit e7ca9b65a68de7752195c8f4d2b5180f3c77d19f
Author: Dmitriy Govorukhin 
Date:   2017-11-13T18:52:47Z

ignite-blt-merge -> ignite-2.4.1

commit cc8168fc184bb7f5e3cc3bbb0743397097f78bfb
Author: Dmitriy Govorukhin 
Date:   2017-11-13T19:13:01Z

merge ignite-pitr-rc1 -> ignite-2.4.1

commit 87e6d74cf6a251c7984f9e68c391f790feccc281
Author: Dmitriy Govorukhin 
Date:   2017-11-14T12:49:33Z

ignite-gg-12877 Compact consistent ID in WAL

commit 9f5a22711baea05bd37ab07c8f928a4837dd83a4
Author: Ilya Lantukh 
Date:   2017-11-14T14:12:28Z

Fixed javadoc.

commit d5af2d78dd8eef8eca8ac5391d31d8c779649bb0
Author: Alexey Kuznetsov 
Date:   2017-11-15T08:09:00Z

IGNITE-6913 Baseline: Added new options to controls.sh for baseline 
manipulations.

commit 713924ce865752b6e99b03bd624136541cea5f9f
Author: Sergey Chugunov 
Date:   2017-11-15T09:03:12Z

IGNITE-5850 failover tests for cache operations during BaselineTopology 
changes

commit b65fd134e748d496f732ec2aa0953a0531f544b8
Author: Ilya Lantukh 
Date:   2017-11-15T12:54:35Z

TX read logging if PITR is enabled.

commit 9b2a567c0e04dc33116b51f88bee75f76e9107d1
Author: Ilya Lantukh 
Date:   2017-11-15T13:45:16Z

TX read logging if PITR is enabled.

commit 993058ccf0b2b8d9e80750c3e45a9ffa31d85dfa
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:51:54Z

ignite-2.4.1 optimization for store full set node more compacted

commit 1eba521f608d39967aec376b397b7fc800234e54
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:52:22Z

Merge remote-tracking branch 'professional/ignite-2.4.1' into ignite-2.4.1

commit 564b3fd51f8a7d1d81cb6874df66d0270623049c
Author: Sergey Chugunov 
Date:   2017-11-15T14:00:51Z

IGNITE-5850 fixed issue with initialization of data regions on node 
activation, fixed issue with auto-activation when random node joins inactive 
cluster with existing BLT

commit c6d1fa4da7adfadc80abdc7eaf6452b86a4f6aa4
Author: Sergey Chugunov 
Date:   2017-11-15T16:23:08Z

IGNITE-5850 transitionResult is set earlier when request for changing 
BaselineTopology is sent

commit d65674363163e38a4c5fdd73d1c8d8e1c7610797
Author: Sergey Chugunov 
Date:   2017-11-16T11:59:07Z

IGNITE-5850 new failover tests for changing BaselineTopology up (new node 
added to topology)

commit 20552f3851fe8825191b144179be032965e0b5c6
Author: Sergey Chugunov 
Date:   2017-11-16T12:53:43Z

IGNITE-5850 improved error message when online node is removed from baseline

commit 108bbcae4505ac904a6db774643ad600bfb42c21
Author: Sergey Chugunov 
Date:   2017-11-16T13:45:52Z

IGNITE-5850 BaselineTopology should not change on cluster deactivation

commit deb641ad3bdbf260fa60ad6bf607629652e324bd
Author: Dmitriy Govorukhin 
Date:   2017-11-17T09:45:44Z

ignite-2.4.1 truncate wal and checkpoint history on move/delete snapshot

commit 3c8b06f3659af30d1fd148ccc0f40e216a56c998
Author: Alexey Goncharuk 
Date:   2017-11-17T12:48:12Z

IGNITE-6947 Abandon remap after single map if future is done (fixes NPE)

commit ba2047e5ae7d271a677e0c418375d82d78c4023e
Author: devozerov 
Date:   2017-11-14T12:26:31Z

IGNITE-6901: Fixed assertion during 
IgniteH2Indexing.rebuildIndexesFromHash. This closes #3027.

commit abfc0466d6d61d87255d0fe38cbdf11ad46d4f89
Author: Sergey Chugunov 
Date:   2017-11-17T13:40:57Z

IGNITE-5850 tests for queries in presence of BaselineTopology

commit f4eabaf2a905abacc4c60c01d3ca04f6ca9ec188
Author: Sergey Chugunov 
Date:   2017-11-17T17:23:02Z

IGNITE-5850 implementation for setBaselineTopology(long topVer) migrated 
from wc-251

commit 4edeccd3e0b671aa277f58995df9ff9935baa95a
Author: EdShangGG 
Date:   2017-11-17T18:21:17Z

GG-13074 Multiple snapshot test failures after baseline topology is 
introduced
-adding baseline test to suite
-fixing issues with baseline

commit edae228c8f55990c15ef3044be987dcb00d6c81a
Author: EdShangGG 
Date:   2017-11-18T10:36:41Z

hack with sleep

commit b5bffc7580a4a8ffbcc06f60c282e73979179578
Author: Ilya Lantukh

[jira] [Created] (IGNITE-7309) Sutable exception should be return as job results when node is about to stop.

2017-12-26 Thread Andrew Mashenkov (JIRA)

Andrew Mashenkov created IGNITE-7309:


 Summary: Sutable exception should be return as job results when 
node is about to stop.
 Key: IGNITE-7309
 URL: https://issues.apache.org/jira/browse/IGNITE-7309
 Project: Ignite
  Issue Type: Bug
  Components: compute, general
Reporter: Andrew Mashenkov
Priority: Minor


User job can fails with confusing exception when server node is stopping and 
going to leave the grid. 
We should suppress InterruptedException. If node is stopping then user should 
see NodeStoppingException.

{code:java}
[org.apache.ignite.internal.processors.job.GridJobWorker] Failed to serialize 
job response [nodeId=02e1588
c-53eb-454a-99a1-48ac6cb33667, ses=GridJobSessionImpl [ses=GridTaskSessionImpl 
..
org.apache.ignite.IgniteCheckedException: Failed to register class.
  at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9929)
  at 
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:819)
  at 
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:760)
  at 
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:619)
  at 
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:483)
  at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
  at 
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1180)
  at 
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1899)
  at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1519)
  at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1147)
  at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:119)
  at 
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1087)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to register 
class.
  at 
org.apache.ignite.internal.binary.BinaryContext.registerUserClassDescriptor(BinaryContext.java:778)
  at 
org.apache.ignite.internal.binary.BinaryContext.registerClassDescriptor(BinaryContext.java:751)
  at 
org.apache.ignite.internal.binary.BinaryContext.descriptorForClass(BinaryContext.java:621)
  at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:164)
  at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
  at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
  at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:239)
  at 
org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
  at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
  at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9923)
  ... 14 common frames omitted
Caused by: org.apache.ignite.internal.IgniteInterruptedCheckedException: null
  at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
  at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119)
  at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.tryGetAndPut(GridDhtAtomicCache.java:724)
  at 
org.apache.ignite.internal.MarshallerContextImpl.registerClassName(MarshallerContextImpl.java:169)
  at 
org.apache.ignite.internal.MarshallerContextAdapter$1.applyx(MarshallerContextAdapter.java:168)
  at 
org.apache.ignite.internal.MarshallerContextAdapter$1.applyx(MarshallerContextAdapter.java:166)
  at 
org.apache.ignite.internal.MarshallerContextAdapter.computeIfAbsent(MarshallerContextAdapter.java:224)
  at 
org.apache.ignite.internal.MarshallerContextAdapter.registerClass(MarshallerContextAdapter.java:164)
  at 
org.apache.ignite.internal.binary.BinaryContext.registerUserClassDescriptor(BinaryContext.java:775)
  ... 23 common frames omitted
Caused by: java.lang.InterruptedException: null
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
  at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161)
  ... 31 common frames omitted
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7308) Caches info is updated on cluster change finish message in discovery thread

2017-12-26 Thread Alexey Goncharuk (JIRA)

Alexey Goncharuk created IGNITE-7308:


 Summary: Caches info is updated on cluster change finish message 
in discovery thread
 Key: IGNITE-7308
 URL: https://issues.apache.org/jira/browse/IGNITE-7308
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
Reporter: Alexey Goncharuk
 Fix For: 2.4


This leads to an assertion on 'slow' clients when exchange in the cluster 
finished before exchange on the client:
{code}
Exception in thread 
"exchange-worker-#188%database.IgniteDbBaselineTopologySelfTest14%" 
java.lang.AssertionError: CacheGroupDescriptor [grpId=94416770, grpName=null, 
startTopVer=null, rcvdFrom=5a916a46-7d5a-400c-a9bc-5be54340, 
deploymentId=20ec5f19061-8097d64f-4969-4114-bc70-eb3c477953d1, 
caches={cache=94416770}, rcvdFromVer=null, persistenceEnabled=true, 
cacheName=cache]
at 
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$CachesInfo.updateCachesInfo(CacheAffinitySharedManager.java:2620)
at 
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:743)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:842)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:595)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2284)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3289: IGNITE-5580

2017-12-26 Thread SomeFire

GitHub user SomeFire opened a pull request:

https://github.com/apache/ignite/pull/3289

IGNITE-5580



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SomeFire/ignite ignite-5580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3289


commit bb9f8490140de45cbad7989ee6b2c7d3644ea85f
Author: Dmitrii Ryabov 
Date:   2017-12-19T17:13:48Z

IGNITE-5580: Basic information about fail for TcpDiscoveryNodeFailedMessage.

commit 973039c82ff94a36e6ce5a5125b1709eb5803fdf
Author: Dmitrii Ryabov 
Date:   2017-12-21T11:17:19Z

IGNITE-5580: Discovery history.

commit 93509d7f01d0e87b106b5973d7045c357517fcb4
Author: Dmitrii Ryabov 
Date:   2017-12-26T08:42:47Z

IGNITE-5580: Test added.




---

[jira] [Created] (IGNITE-7307) H2DynamicIndexingComplexServerTransactionalReplicatedTest.testOperations fails with assertion

2017-12-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-7307:
---

 Summary: 
H2DynamicIndexingComplexServerTransactionalReplicatedTest.testOperations fails 
with assertion
 Key: IGNITE-7307
 URL: https://issues.apache.org/jira/browse/IGNITE-7307
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Vladimir Ozerov
 Fix For: 2.5


{code}
[2017-12-26 11:40:53,187][ERROR][main][root] Test failed.
java.lang.AssertionError: localNode = cde0ed21-5bfa-48ee-8e47-53d85880, 
dhtNodes = [TcpDiscoveryNode [id=49318149-4b22-40d1-b68f-8b6b2d52, 
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=3, 
intOrder=3, lastExchangeTime=1514277649002, loc=false, 
ver=2.4.0#19700101-sha1:, isClient=false], TcpDiscoveryNode 
[id=cde0ed21-5bfa-48ee-8e47-53d85880, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1514277652494, loc=true, ver=2.4.0#19700101-sha1:, 
isClient=false], TcpDiscoveryNode [id=7530f231-3bd4-4cc0-b295-0e2f11b3, 
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=4, 
intOrder=4, lastExchangeTime=1514277649456, loc=false, 
ver=2.4.0#19700101-sha1:, isClient=false]]
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.map(GridDhtTxPrepareFuture.java:1608)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1274)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:678)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1056)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3619)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:257)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearPessimisticTxPrepareFuture.prepareLocal(GridNearPessimisticTxPrepareFuture.java:256)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearPessimisticTxPrepareFuture.preparePessimistic(GridNearPessimisticTxPrepareFuture.java:406)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearPessimisticTxPrepareFuture.prepare(GridNearPessimisticTxPrepareFuture.java:190)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3323)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3390)
at 
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commit(GridNearTxLocal.java:3350)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:429)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:177)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsDistributed(DmlStatementsProcessor.java:212)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1809)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1645)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$6.applyx(GridQueryProcessor.java:2034)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$6.applyx(GridQueryProcessor.java:2030)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2527)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFieldsNoCache(GridQueryProcessor.java:2039)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFieldsNoCache(GridQueryProcessor.java:1995)
at 
org.apache.ignite.internal.processors.cache.index.H2DynamicIndexingComplexTest.executeSql(H2DynamicIndexingComplexTest.java:333)
at 
org.apache.ignite.internal.processors.cache.index.H2DynamicIndexingComplexTest.executeSql(H2DynamicIndexingComplexTest.java:342)
at 
org.apache.ignite.internal.processors.cache.index.H2DynamicIndexingComplexTest.executeSqlSingle(H2DynamicIndexingComplexTest.java:351)
at 
org.apache.ignite.internal.processors.cache.index.H2DynamicIndexingComplexTest.testOperations(H2DynamicIndexingComplexTest.java:141)
at

[jira] [Created] (IGNITE-7306) Incorrect force key request processing when MVCC is enabled

2017-12-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-7306:
---

 Summary: Incorrect force key request processing when MVCC is 
enabled
 Key: IGNITE-7306
 URL: https://issues.apache.org/jira/browse/IGNITE-7306
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Vladimir Ozerov
 Fix For: 2.5


Reproducer: {{IgniteCacheMultiTxLockSelfTest#testExplicitLockManyKeys}}
Root cause: when {{GridDhtForceKeysRequest}} is processed locally, we obtain 
{{GridCacheEntryInfo}} through {{GridCacheMapEntry.info}}. Returned instance is 
unaware of MVCC version. Need to return {{GridCacheMvccEntryInfo}} instead.

{code}
java.lang.AssertionError
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccInitialValue(IgniteCacheOffheapManagerImpl.java:1433)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccInitialValue(IgniteCacheOffheapManagerImpl.java:396)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2624)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture$MiniFuture.onResult(GridDhtForceKeysFuture.java:537)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onResult(GridDhtForceKeysFuture.java:199)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processForceKeyResponse(GridDhtCacheAdapter.java:176)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$11.onMessage(GridDhtTransactionalCacheAdapter.java:193)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$11.onMessage(GridDhtTransactionalCacheAdapter.java:191)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$MessageHandler.apply(GridDhtCacheAdapter.java:1406)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$MessageHandler.apply(GridDhtCacheAdapter.java:1388)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1567)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1195)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:128)
at 
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092)
at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

53 matches

Mail list logo