Re: Ignite query performance with lots of joins

Hi, I checked provided test data. I was able to speedup a query execution with Ignite about 2 times on my machine by using extra configuration property System.setProperty("IGNITE_MAX_INDEX_PAYLOAD_SIZE", "256"); See a following documentation section about configuring index inline size [1]. You

Re: DataStreamer addData takes lot of time after 500 million writes

Hi, Is the problem specific to DataStreamer? Will the problem disappear if insert the same data using IgniteCache.put (or putAll batches)? вс, 15 сент. 2019 г. в 15:49, KR Kumar : > > Hi all - Why data streamer take lot of time randomly after 500+ million > writes, it frequently and very

Re: Topology snapshot explanation

Hi Rick, 1. Clients should not "reserve" offheap memory. 2. Near caches do not use offheap. Could you please check that a client process does not request offheap memory from OS? I hope this is just a trouble with a "Topology snapshot" misleading message. сб, 14 сент. 2019 г. в 12:36, rick_tem :

Re: Sudden node failure on Ignite v2.7.5

Unfortunately I do not know. Currently there is no release activity on 2.8. пт, 19 июл. 2019 г. в 09:39, ihalilaltun : > > Hi Ivan > > Thanks for the reply. I've checked the jira issue and it says it will be > released in v2.8, when do you think v2.8 will be released? > > > > - > İbrahim

Re: Sudden node failure on Ignite v2.7.5

Hi, It seems that the issue [1] was already fixed. [1] https://issues.apache.org/jira/browse/IGNITE-11953 вт, 16 июл. 2019 г. в 09:30, ihalilaltun : > > Hi Pavel, > > Thanks for you reply. Since we use the whole sysyem on production > environment we cannot apply the second solution. > Do you

Re: onheapCacheEnabled enormous heap consumption

Andrey, Yes it is a little bit complicated for understanding. CacheConfiguration.evictionPolicy takes it roots back in history where there were neither offheap nor persisence, cache data was stored in java heap only. Also, as far as I know, today CacheConfiguration.evictionPolicy (as already

Re: How to improve the performance of COPY commands?

Hi, Currently COPY is a mechanism designed for the fastest data load. Yes, you can try separate your data in chunks and execute COPY in parallel. By the way, where is your input located and what is it size in bytes (Gb)? Is persistence enabled? Does a DataRegion have enough memory to keep all

Re: Spring Indexing example

Hi Rick, You can try following:

Re: unable to query the cache after restart

Have no clever idea, waiting for a reproducer. вт, 25 июн. 2019 г. в 05:59, goutham manchikatla : > > It didn't work as expected, tried disable/enable WAL too, still I see the > same behavior. > > On Mon, Jun 24, 2019 at 7:32 AM Павлухин Иван wrote: >> >> Hi Goutha

Re: Canceling a running query on ignite

Hi, How do you execute your SQL queries? Java API SqlFieldsQuery has a setTimeout method. Also there is a QueryCursor.cancel method. Also KILL query SQL command [1] is targeted for 2.8. [1] https://issues.apache.org/jira/browse/IGNITE-11564 пт, 21 июн. 2019 г. в 18:41, Ph Tham : > > Hello All,

Re: Can we avoid the PME when restarts a node in cluster.

It worth noting that IGNITE-9420 seems to be relevant for clusters with enabled persistence. пт, 21 июн. 2019 г. в 12:57, Ilya Kasnacheev : > > Hello! > > 2.7.5 definitely did not include it and there is no set date for 2.8. > > Regards, > > пт, 21 июн. 2019 г., 6:50 Justin Ji : >> >> 2.7.5 may

Re: unable to query the cache after restart

Hi Goutham, I did not get from your last message does it work now as expected? If not does it work without a trick with disable/enable WAL? ср, 19 июн. 2019 г. в 19:18, goutham manchikatla : > > Hi Denis, > > I tried removing Default_Region name property from config, still I see the > same

Re: Stop JVM on network Segmenation

Hi Taruk, There is no such thing out of box. You can try to use org.apache.ignite.Ignition#addListener and handle org.apache.ignite.IgniteState#STOPPED_ON_SEGMENTATION state change according to your needs. Perhaps, if you can describe why do you need a specific handling then the Community might

Re: unable to query the cache after restart

Hi, First bet is that it is a problem with configuring persistence for a cache, but I have not found any problems in config. I can suggest to check a directory with database files to check that the same directory is used among node restarts and that it contains files for the cache in question.

Re: FW: class loading, peer class loading, jars, fun times in ignite

Hi Scott, As far as I know, peer class loading does not work for data classes (which are stored in a cache). It works for tasks sended for execution using IgniteCompute. It is only a partial answer. Could you describe your use case in more details? вт, 28 мая 2019 г. в 23:35, Scott Cote : > >

Re: Behavior of Ignite during PME on new nodes added to the cluster

Hi Evangelos and Matt, As far as know there were issues with a join of a client node in previous Ignite versions. In new versions a joining client should not cause any spikes. In fact PME is (unfortunately) a widely known beast in the Ignite world. Fundamentally PME can (and should) perform

Re: sending event on update of a specific cache

Hi, You can try a following combination: 1. Use IgniteEvents to setup event listeners. 2. Disable events for all other caches via CacheConfiguration.setEventsDisabled. вс, 7 апр. 2019 г. в 15:18, matanlevy : > > Hi, > > I am using ignite cache and I would like to know if there is any mechanishm

Re: Primary partitions return zero partitions before rebalance.

20:22, Koitoer : > > Sorry but what is exactly the PME ? > > On Mon, Apr 1, 2019 at 1:55 AM Павлухин Иван wrote: >> >> Hi, >> >> Sorry for the late answer. An observed result seems expected to me. I >> suppose following: >> 1. EVT_CACHE_REBALANC

Re: quick Q- date type in cache is converted to ticks. is this based on time since from 1970?

Yes, internally it is stored as number of milliseconds since 1970-01-01 00:00:00.000 пт, 29 мар. 2019 г. в 16:17, wt : > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ -- Best regards, Ivan Pavlukhin

Re: Primary partitions return zero partitions before rebalance.

nt=ErrorOnTimerWait message={}", e.getMessage(), e); > } > } > }).start(); > > > After a couple of attempts (some seconds), the `primaryPartitions` is > returning the correct set of partitions assigned to a node. I will check the > AffinityAssignment

Re: How to use atomic operations on C++ thin client?

Hi Jack, Should be included into next version [1]. Stay tuned. [1] https://issues.apache.org/jira/browse/IGNITE-9904 пт, 15 мар. 2019 г. в 01:32, jackluo923 : > > After digging deeper, it appears that thin-client atomic cache operations are > not implemented. I have implemented and tested the

Re: Primary partitions return zero partitions before rebalance.

Hi, What Ignite version do you use? How do you register your listener? On what object do you call primaryPartitions/allPartitions? It is true that Ignite uses late affinitly assignment. And it means that for each topology change (node enter or node leave) parttion assigment changes twice. First

Re: GridTimeoutProcessor - Timeout has occurred - Too frequent messages

Hi, Yes, perhaps there is no reach documentation for mentioned classes. But on the other hand they are internal classes which could be changed at any time. I will try to outline roles for 2 classes. 1. CancellableTask is used internally as a general way to cancel some action sheduled in the

Re: GridTimeoutProcessor - Timeout has occurred - Too frequent messages

Hi, There is nothing bad with message you see. GridTimeoutProcessor is used internally by Ignite for scheduling a task execution after a delay. Debug message "Timeout has occurred" is logged every time scheduling delay has ended. It is a debug message and a purpose of it is checking that

Re: Why does write behind require to enable write through?

Hi, Ignite CacheConfiguration inherits setWriteThrough method from JCache API [1]. Enabled writeThrough in JCache means that cache is backed by some other storage to which writes are propagated along with writes to to the cache. Unfortunately (or not?) JCache uses write-through term while Ignite

Re: Access a cache loaded by DataStreamer with SQL

Hi Mike, You can find a simple example of loading data with data streamer and querying it with SQL in a following gist [1]. It is possible but somehow tricky to load table created by DDL using data streamer. Perhaps, SQL COPY could be handy for you [2]. It uses data streamer under the hood. [1]

Re: same cache cannot update twice in one transaction

Hi, MVCC in Ignite is targeted to provide transactional consistency guarantees. I suppose that with eventually consistent 3rd party store it would be impossible to give much guarantees in general. Do think that such eventually consistent store will be widely used? What kind of guarantees should

Re: What is the correct way of keeping ignite alive in a console app

Hi, Your application exits because Ignite node is started in try-with-resources block. So, the node is stopped upon leaving try block. If you write simply public static void main(String[] args) { Ignition.start(igniteConfiguration); } application will continue running after main method

Re: Native memory tracking of an Ignite Java 1.8 Application

.sleep(6); > unsafe.freeMemory(address); > System.out.println("3)Now going to sleep"); > Thread.sleep(6); > System.out.println("Exited."); > } > > > > Thanks., > Prasad > > On Mon, Feb 18, 2019 at 7:56 PM Павлухин Иван wrot

Re: Native memory tracking of an Ignite Java 1.8 Application

Prasad, Someone has already posted a snippet [1]. [1] https://gist.github.com/prasanthj/48e7063cac88eb396bc9961fb3149b58 пн, 18 февр. 2019 г. в 17:23, Павлухин Иван : > > Hi Prasad, > > As far as I remember offheap memory allocated with use of Unsafe is > not reflected in Native

Re: Native memory tracking of an Ignite Java 1.8 Application

Hi Prasad, As far as I remember offheap memory allocated with use of Unsafe is not reflected in Native Memory Tracking report. You are right that documentation is not verbose about reported categories [1]. It might be the case that memory allocated by ByteBuffer.allocateDirect falls into

Re: I have a question about Java scan ignite cache

Hi, But what result do you observe in your experiment? ср, 13 февр. 2019 г. в 05:16, chengpei : > > import org.apache.ignite.Ignite; > import org.apache.ignite.IgniteCache; > import org.apache.ignite.Ignition; > import org.apache.ignite.cache.CacheAtomicityMode; > import

Re: DataStreamer not loading complete data.

Hi, Have you called "Close" after last item of data was fed to your streamer? If it is not the case could you please provide your code to check? вт, 12 февр. 2019 г. в 16:10, Hemasundara Rao < hemasundara@travelcentrictechnology.com>: > Hi, > While using DataStreamer (.net ) to load data ,

Re: Unable to form connection between ignite(v 2.7) node inside kubernetes-1.11.3

Hi Lalit, Usually topics related to some sort of contribution are discussed on dev list. I added user list to recipients list. You will get an answer for usability questions on user list quicker. вт, 29 янв. 2019 г. в 00:00, Lalit Jadhav : > > While starting one node it gets up with time delay

Re: Cache updates slow on Linux Vs Windows

I suppose that Windows is faster in particular case. Am I wrong? пн, 14 янв. 2019 г. в 18:00, ilya.kasnacheev : > > Hello! > > Did you figure out anything? I went through your log but did not have any > exact ideas. Is it possible that Windows node is slowed down by very active > I/O? > >

Re: How to dump thread stacks in Ignite docker container

Hi Justin, One way you can do it: 1. Access to standard output of a default process started in an Ignite container. 2. Find pid of a java process in the container (run ps in the container). 3. Run "kill -3 ${java_pid}" in the container. 4. Observe a thread dump in the container output (mentioned

Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7

Hi Prasad, > javax.cache.CacheException: Only pessimistic repeatable read transactions are > supported at the moment. Exception mentioned by you should happen only for cache with TRANSACTIONAL_SNAPSHOT atomicity mode configured. Have you configured TRANSACTIONAL_SNAPSHOT atomicity for any cache?

Re: How to define a cache template?

Hi, Perhaps following docs section can help you [1]. [1] https://apacheignite.readme.io/docs/cache-template сб, 29 дек. 2018 г. в 04:56, yangjiajun <1371549...@qq.com>: > > Hello! > > I want to make some cache settings global.It means I need to define a cache > template and then use

Re: Ignite.close method blocked indefinitely

Hi userx, How do you start you client? You use a term IgniteClient, but it looks like that you are using Ignite instance running in client mode. By IgniteClient so-called "thin client" is assumed (java interface for thin client has name "IgniteClient"). >From the thread dump I can see that

Re: CAP Theorem (CP? or AP?)

Hi Jose, First of you refer to a slide about Data Center Replication, an commercial feature of GridGain. Ignite does not provide such feature. Also, SQL and Cache API could behave different. You can check how Cache API shows itself in your experiments. CacheAtomicityMode and PartitionLossPolicy

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi summasumma, > Means, i should have minimum 16 Gb of RAM (8 dataregion+ 8 directmem) for > Ignite to run properly i guess. Not quite. Actually, I am not aware that Ignite requires some special tuning of MaxDirectMemorySize. If direct memory causes OOME then the exception message usually points

Re: Do we require to set MaxDirectMemorySize JVM parameter?

GMT+03:00, Павлухин Иван : > Hi collnc, > > Perhaps, documentation can answer you question [1]. > > [1] https://apacheignite.readme.io/docs/durable-memory-tuning > > 2018-12-21 20:39 GMT+03:00, summasumma : >> In the above example, >> >> is setting "

Re: Do we require to set MaxDirectMemorySize JVM parameter?

Hi collnc, Perhaps, documentation can answer you question [1]. [1] https://apacheignite.readme.io/docs/durable-memory-tuning 2018-12-21 20:39 GMT+03:00, summasumma : > In the above example, > > is setting " > " > in xml config file same as adding a jvmoption "-XX:MaxDirectMemorySize=8g" > ? >

Re: How Ignite transfer cached data between nodes in the cluster

Hi, Actually, all code paths are not trivial. If you would like to dig into data retrieval process for IgniteCache.get you can explore GridPartitionedSingleGetFuture and check where GridNearSingleGetRequest is created and how is it handled. ср, 19 дек. 2018 г. в 18:26, vyhc...@hotmail.com : > > I

Re: Can i use SQL query and Cache Operations in same transaction (JTA)

Hi Hyungbai, Please be aware that MVCC is included into Ignite release as kind of experimental feature as stated in release notes [1]. I do not know plans for providing releases with MVCC fixes. But perhaps you can try nightly builds [2] once the related ticket [3] is resolved. And I believe that

Re: Ignite in docker (Native Persistence)

Hi Rahul, Could you please share an ignite configuration and how do you launch a docker container with Ignite? Do you see something in your ignitedata/persistence ignitedata/wal ignitedata/wal/archive after container stop? I guess you can configure a consistentId by configuring

Re: Migrate from 2.6 to 2.7

Hi Andrey, It looks like your persisted data was read incorrectly by upgraded Ignite. It would be great if you provide runnable reproducer. Regarding Optimistic Serializable transactions. They are still supported by caches with TRANSACTIONAL atomicity mode. In your error it looks like that your

Re: Events question

Hi Mikael, In order to use event storage you should configure EventStorageSpi. By default an event storage is not enabled. Which fits your case. You can find more details about event storage in [1]. [1]

Re: Unable to load more than 5g data through sqlline

Hi Debashis, Sorry for late answer. How much RAM does your server have? You configured your data region with 7 gb max size. This size defines how much RAM could be allocated for the region. If your server has not enough RAM the OS cannot allocate enough for Ignite and kills it. With persistence

Re: Unable to load more than 5g data through sqlline

Hi Debashis, Is sqlline started on the same machine? Perhaps sqlline ate all the available memory but the system decided to kill Ignite. Could you split incoming data into relatively small chunks and try it out? вт, 30 окт. 2018 г. в 23:07, debashissinha : > < >

Re: how to handle dataregion out of memmory gracefully

Hi Wayne, You can see a message written to a console during Ignite startup with a calculated amount of memory required for a server. It looks as follows: Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease

Re: Writing binary [] to ignite via memcache binary protocol

.exit(0); } } And I see expected output: [1, 2, 3] 123 [1] https://mvnrepository.com/artifact/net.spy/spymemcached/2.12.3 ср, 17 окт. 2018 г. в 10:25, Павлухин Иван : > Hi Michael, > > Answering one of your questions. > > Does ignite internally have a way to store the data type when cache > e

Re: Writing binary [] to ignite via memcache binary protocol

Hi Michael, Answering one of your questions. > Does ignite internally have a way to store the data type when cache entry is stored? Yes, internally Ignite maintains data types for stored keys and values. Could you confirm that for real memcached your example works as expected? I will try

Re: Ignite Events

Hi drosso, Luckily there is no mystery. By the way, what version of Ignite do you use? The clue to strange behavior here is topology change during your test execution. As I see, Putter node is a server data node as well, so it will hold some data partitions on it and consequently will receive

Re: Ignite Events

Hi drosso, Indeed looks strange. If you provide a reproducer I will take a look. ср, 10 окт. 2018 г. в 16:44, drosso : > Hi, > I've been playing with Ignite events for a couple of days but there's a > behavior of my sample programs that I really can't understand. > I've prepared 2 sample

Re: lost partition recovery with native persistence

Please ignore my message. I misunderstood the problem. пт, 5 окт. 2018 г. в 14:16, Павлухин Иван : > Hi Roman, > > Actually, Ignite with enabled persistence supports crash recovery. It is > mentioned in [1]. > > [1] > https://apacheignite.readme.io/v2.6/docs/distributed-per

Re: lost partition recovery with native persistence

Hi Roman, Actually, Ignite with enabled persistence supports crash recovery. It is mentioned in [1]. [1] https://apacheignite.readme.io/v2.6/docs/distributed-persistent-store#section-transactional-guarantees пт, 5 окт. 2018 г. в 13:30, Maxim.Pudov : > Great idea, I like it. However, it's

Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture

Hi Subash, Correct me if I am wrong, but in current Ignite version persistence is not mandatory and it is even disabled by default. With disabled persistence nothing will be written to disk. 2.x version storage architecture was developed as an improvement of the previous one. Ignite 2.x uses

Re: SQL query and Indexes architecture

Hi Eugene, In community wiki there are several "under the hood" documents like [1]. Unfortunately there is no document about SQL. It seems that such document would be useful for you and many others. Perhaps, if you have habit to document your findings (in blogs or somewhere else) it could become

Re: How to set query timeout or cancel query when submit SQL query from SQL tool like DBweaver and sqlline?

Hi Ray, As far as I know query timeout can be configured only for SqlQuery/SqlFieldsQuery API. Mentioned global timeout configuration through "jdbc:h2..." URL was not implemented.

Re: IgniteUtils NoClassDefFoundError

Hi Jack, Could you provide logs and full console output? NoClassDefFoundError -- can be thrown when class in question is on classpath but fails to initialize (e.g. exception thrown from static initializer). 2018-09-11 6:05 GMT+03:00 Jack Lever : > Hi All, > > I'm getting an error on application

Re: Simulate Read Only Caches

Hi Steve, It is an interesting question. I am not aware of any development in this direction. Perhaps, more experienced community members could tell more. I think that it would be great to experiment around a topic disabling some checks here and where. Then it worth to run some kind of benchmark

Re: How to run a job every 5 seconds in Ignite

Hi Lokesh, You could try out extended cron syntax implemented by Ignite [1]. [1] https://apacheignite.readme.io/docs/cron-based-scheduling#section-syntax-extension 2018-08-28 10:51 GMT+03:00 Lokesh Sharma : > Is it possible to run the job every few seconds? As far as I know, cron > API

Re: values retrieved from the cache are wrapped with JdkDynamicAopProxy while using springboot and JCache

Hi daya, Yes it is a serialization issue. Field AdvisedSupport.methodCache is marked transient and method readObject which should initialize this field on deserialization is not called. So, it has value null and NPE is thrown. I am not sure whether it is expected behavior or not. Also, it does

Re: The Apache Ignite Book

Hi Shamim, In 1st chapter Introduction among Ignite possible usages it is mentioned: "From version 2.5 Apache Ignite will support transactions at SQL level" It is not true. Transactional SQL was not released yet. There is plan to release it in near future, but as for me it is better to not

Re: values retrieved from the cache are wrapped with JdkDynamicAopProxy while using springboot and JCache

Hi, Looks like Spring itself wraps result into proxy. If you could provide a reproducer it will help to find a reason faster. 2018-08-07 21:09 GMT+03:00 daya airody : > Values retrieved from cache are wrapped with JdkDynamicAopProxy. This > throws > below NPEs > > --- >

Re: OOM on connecting to Ignite via JDBC

Hi Orel, Are you sure that correct port is used? By default 10800 port is used for JDBC connections. You have 8080 in your command line. The error could be caused by reading unexpected input from server and interpreting it as very huge packet size. Attempt to allocate buffer of such size could

Re: Transaction return value problem

Hi, Denis I wonder if NESTED transaction propagation will work with Ignite? 2018-08-02 18:07 GMT+03:00 Denis Mekhanikov : > Here you can find how to use Spring transaction management together with > Ignite: https://ignite.apache.org/releases/latest/javadoc/ >

Re: 3rd party persistence with Hive

Hi engrdean, I see 2 points here: 1. Log says that load was failed in multi-threaded mode. But then it says that loading was finished, which means that it succeeded in single-threaded mode. 2. Indeed Ignite generates query which is not supported by Hive. And not supported thing is 'mod' operator,