Re: what does CurrentCpuLoad in ClusterLocalNodeMetricsMXBeanImpl means?

2017-05-15 Thread Andrey Mashenkov
Hi, Actualy, it returns OS process CPU load. On Fri, May 12, 2017 at 12:17 PM, bluehu wrote: > what does CurrentCpuLoad in ClusterLocalNodeMetricsMXBeanImpl means? > > "the TOTAL system cpuload" or "cpuload the node consume"? > > > > -- > View this message in context:

Re: Cannot create cache in EVT_CLIENT_NODE_RECONNECTED event

2017-05-15 Thread Andrey Mashenkov
Hi David, Seems, this predicate called from sensitive code and cache creation end up with deadlock. Try to move cache creation to separate thread. On Mon, May 15, 2017 at 11:44 AM, David Li wrote: > Hi guys, > > In the code snippet below, I am trying to re-create a cache

Re: Ignite2.0 memory policy limit

2017-05-15 Thread Andrey Mashenkov
Hi Ajay, Would you please share a full stacktrace? On Mon, May 15, 2017 at 11:00 AM, Ajay wrote: > Hi, > > Is Ignite having any limit to create the memory region on single server > node. Because i created server node with following > > > >

Re: PutIfAbsent issues

2017-05-15 Thread Andrey Mashenkov
Hi Rick, This should be impossible. Please, fill the ticket if you will have a success to reproduce it. On Thu, May 11, 2017 at 3:42 PM, rick_tem wrote: > Hi, > > I am using Ignite 2.0 and wondering if there are any known putIfAbsent(key, > value) issues where on

Re: Performance issue with Replicated Cache among 7 nodes

2017-05-15 Thread Andrey Mashenkov
Hi Ramzinator, 1. There is no need to se number of backups when using replicated cache. All data nodes will have full cache data copy. 2. For FULL_SYNC mode, Ignite will send updates to all nodes and waits for they being finished. 3. Is it possible there is slow connection? How much time takes to

Re: Group indices and group by query

2017-05-11 Thread Andrey Mashenkov
> /* "PersonWithIndex"."merge_scan" */ > GROUP BY __C0_0] > > Although it seems that it uses it when I get times, seems the opposite. > > 2017-05-11 13:55 GMT+02:00 Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> Query use PERSONWITHINDEX_MON

Re: Group indices and group by query

2017-05-11 Thread Andrey Mashenkov
ySqlField > public String dni; > @QuerySqlField(index = true, orderedGroups = > {@QuerySqlField.Group(name="index_group_month_age", order=0)}) > public String month; > > > > > > public class PersonWithoutIndex implements Serializable { >

Re: GroupBy with index is really slow.

2017-05-11 Thread Andrey Mashenkov
SONWITHOUTINDEX.__SCAN_ */] > [SELECT DISTINCT > __C0_0 AS AGE > FROM PUBLIC.__T0 > /* "PersonWithoutIndex"."merge_scan" */] > > Age is a value between 0-100. I tried with a value between 0-1000 with the > same result. > It seems that it's not us

Re: GroupBy with index is really slow.

2017-05-10 Thread Andrey Mashenkov
e with a real dataset(22M rows, > 7gb) but it takes so long that query is aborted. > > > 2017-05-08 12:12 GMT+02:00 Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> Hi, >> >> Why do you think query is slow? >> What its execution time? What is expected t

Re: How to correctly shut down Ignite Application

2017-05-10 Thread Andrey Mashenkov
hable(IgniteUtils.java:2102) > ~[ignite-core-2.0.0.jar:2.0.0] > > at > org.apache.ignite.internal.util.IgniteUtils$18.run(IgniteUtils.java:1884) > ~[ignite-core-2.0.0.jar:2.0.0] > > ... 5 more > > > > These errors disappeared after setting:

Re: How to correctly shut down Ignite Application

2017-05-09 Thread Andrey Mashenkov
Looks like client node can't connect to server [id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.89].. *isClient=false*] Would you please attach full logs? On Tue, May 9, 2017 at 2:55 AM, baozipu wrote: > Hi, > > Thanks for the

Re: How to balance many service grids in the cluster?

2017-05-09 Thread Andrey Mashenkov
Hi, You can use affinity deployment via setting affinity key in serviceConfiguration. On Tue, May 9, 2017 at 2:38 PM, ght230 wrote: > I want to deploy 12 service grids in a cluster with 2 ignite hosts. > > Every service grid only running one instance. > > I have though that 6

Re: How to correctly shut down Ignite Application

2017-05-08 Thread Andrey Mashenkov
Hi, Try to set same option for all nodes either -Djava.net.preferIPv4Stack=true or -Djava.net.preferIPv6Stack=true. There is an example in ignite.sh. On Tue, May 9, 2017 at 12:19 AM, baozipu wrote: > Hello, here is my topology configuration: > > 1. One Ignite server

Re: How to monitor and alert for server counts

2017-05-08 Thread Andrey Mashenkov
Looks like you need a 3rd party monitoring software. E.g. Nagios\Icinga or Zabbix. On Mon, May 8, 2017 at 1:19 PM, tysli2016 wrote: > we have a couple of Ignite servers serve as key-value store and want to get > email notification when any server went down. > > anyone

Re: Ignite Version 2.0 startup H2 error

2017-05-08 Thread Andrey Mashenkov
Hi Rick, Looks weird. Ignite-indexing module should has H2 as dependency. Ignite-2.0 should work fine with H2 1.4.195. On Mon, May 8, 2017 at 1:13 PM, rick_tem wrote: > Hi, > > I am trying to move from 1.9 to 2.0 version of Ignite and running into the > following error.

Re: Group indices and group by query

2017-05-08 Thread Andrey Mashenkov
Group indices description can be found here [1]. Please, check if you read docs for your ignite version, as configuration for ignite 1.x and 2.0 can differs. Group indices in Ignite a similar to composite or multi-column indixes in databases. Yes, it is work that way. Group index for (groupFiled,

Re: GroupBy with index is really slow.

2017-05-08 Thread Andrey Mashenkov
Hi, Why do you think query is slow? What its execution time? What is expected time? How many nodes do you use? How much data does cache contains? On Mon, May 8, 2017 at 1:47 AM, Guillermo Ortiz wrote: > I have a simple cache where I have Person object (id, age and so

Re: Query search performance and scalability

2017-05-08 Thread Andrey Mashenkov
Hi, Network roundtrip costs 100-200mcs. As you can see error is very high, lower and higher value differ in more than 2 times. So, query runinng in 280mcs looks ok. Netwok latencies often unpredictable, I would think queries per second metric can give more clear picture than comparing of micro

Re: Nodes hanging when accessing queue

2017-05-05 Thread Andrey Mashenkov
Hi, There is known issue, sometimes one of nodes use IPv6 for communication that results in whole grid hanging with series of similar messages in logs. Yes, it should be applied to both client and server nodes. There is a commented line with preferIPv4 option in ignite.sh. Would you please

Re: Ignite TLS with TrustStore

2017-05-04 Thread Andrey Mashenkov
Hi, You have to set either trustManagers or trustStoreFilePath. Do you see any other errors on node start? Can Ignite find trustStore? Does trustStore contains Root cert and all CA certificates, so certificate path validation can be performed? Store with root certificates locates in

Re: Network Discovery inside docker containers

2017-05-04 Thread Andrey Mashenkov
Hi, Yes, addressResolver should be helpful, but seems, there is a bug [1] [1] https://issues.apache.org/jira/browse/IGNITE-5117 On Thu, Apr 27, 2017 at 10:15 AM, Lukas Lentner wrote: > Ok, > > > > I heard from chat that I should use an https://ignite.apache.org/ >

Re: Nodes hanging when accessing queue

2017-05-04 Thread Andrey Mashenkov
Hi, 1. Yes, adding backups for datastructures cache (via AtomicConfiguration) shoud help. 2. Seems, one or more of your nodes register with ipv6 address. Try to use JVM option for all nodes: either -Djava.net.preferIPv4Stack=true or -Djava.net.preferIPv6Stack=true. On Tue, May 2, 2017 at 5:17

Re: Combined off heap size?

2017-05-04 Thread Andrey Mashenkov
Hi, 1. Ignite will throw GridOffHeapOutOfMemoryException if you try to allocate more than 10G, and will remove entries from cache if it number exceeds 10k (according to eviction policy). If you want Ignite remove entries when its occupy more than 10G, then you need to configure eviction policy

Re: Ignite SQL Indexing Performance problems

2017-05-03 Thread Andrey Mashenkov
plan. > > Sergi > > > 2017-05-03 15:02 GMT+03:00 Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> Hi, >> >> As you use REPLICATED cache, the query will run on one node in single >> thread. This should be fixed in 2.0 >> The only workaroun

Re: Write-behind Flush Size

2017-05-03 Thread Andrey Mashenkov
Hi Ricky, WriteBehindFlushSize is number of entries that queue can accumulate before it will be flushed. Are you update a entry with same key or different entries? Entry in writeBehind queue can be overwritten before it will be persisted. You can use time-based flushing or queue-sized flushing

Re: Ignite SQL Indexing Performance problems

2017-05-03 Thread Andrey Mashenkov
Hi, As you use REPLICATED cache, the query will run on one node in single thread. This should be fixed in 2.0 The only workaround for now is to convert cache to PARTITIONED, to make the query to be run on several nodes, that will significantly increase performance. Also, you can try to add more

Re: "object not serializable " when initializing igniteContext with Zookeeper Configuration

2017-05-02 Thread Andrey Mashenkov
Hi, Please properly subscribe to the mailing list so that the community can receive email notifications for your messages. To subscribe, send empty email to user-subscr...@ignite.apache.org and follow simple instructions in the reply. Seems, you provide anonymous class to IgniteContext that has

Re: tcp-disco-ip-finder-cleaner

2017-05-02 Thread Andrey Mashenkov
Hi, Looks like a bug. Here is a ticket [1]. [1] https://issues.apache.org/jira/browse/IGNITE-5117 On Thu, Apr 27, 2017 at 4:01 PM, gatorjh wrote: > Here is the ignite config > > > > > -- > View this message in context: http://apache-ignite-users. >

Re: tcp-disco-ip-finder-cleaner

2017-04-27 Thread Andrey Mashenkov
Hi Would you please share ignite config? On Tue, Apr 25, 2017 at 9:55 PM, gatorjh wrote: > We are running Ignite 1.8.0 within Docker on an AWS instance. Ignite > registers with Zookeeper. We use a BasicAddressResolver in the Ignite > config > so that only the Docker host IP

Re: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing - Query execution is too long

2017-04-27 Thread Andrey Mashenkov
Hi, 1. You may want to make Ignite use index on APPLICATION_DATE or even composite index of two fields: EXPIRATION_DT, APPLICATION_DATE. Ignite does not support HINTs in 1.x version, but it will be added in 2.0. 2. I see no query plan for second query. Try to run in from console with explain

Re: setting query timeout and getting partial result instead of exception.

2017-04-26 Thread Andrey Mashenkov
Hi, You can try to increase SqlQuery.setTimeout() (30 sec by default) or connectorConfiguration.setIdleTimeout() (7 sec by default) It may slightly help if you use sorting, as you need to have pages of rows from all nodes to make correct merge. Also, you can try to increase SqlQuery.pageSize()

Re: Basic question on scaling of reads with ignite sql queries

2017-04-26 Thread Andrey Mashenkov
Hi Tuco, Your case has one lack, you have 2 of 4 machines that contains full dataset and both nodes has non-intersecting partition set. But what if we have thousand nodes? There are number of additional questions... How do we find node group that has full partition set, how much node group we

Re: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing - Query execution is too long

2017-04-26 Thread Andrey Mashenkov
" > > + > CONSUMER_SELECT_FEILDS_QUERY + " from CRDecisionConsumerEntity where > LAST_NAME = ? > > + " UNION " > + CONSUMER_SELECT_FEILDS

Re: Basic question on scaling of reads with ignite sql queries

2017-04-26 Thread Andrey Mashenkov
Replicated cache works in different way. SQL query on replicated cache will always run on single node. Also, joins to replicated cache will always be local as all data is collocated. There is no way to run query in multiple threads on replicated cache as it has single segment for now. On Tue,

Re: Weird behaviour about enum with methods

2017-04-26 Thread Andrey Mashenkov
Hi David, Looks like a bug. As a workaround, you can either use optimized marshaller or compare enum ordinal. I've created an issue IGNITE-5087 [1]. [1] https://issues.apache.org/jira/browse/IGNITE-5087 On Mon, Apr 24, 2017 at 10:57 AM, David Li wrote: > Hi all, > > I

Re: Basic question on scaling of reads with ignite sql queries

2017-04-25 Thread Andrey Mashenkov
Hi Tuco, Seems, there is a misunderstanding. Ignite does not have master and slave nodes. All nodes are equals. There are primary and backup partitions distributed among grid nodes. SQL queries are running on index of primary partitions on all data nodes. Actually, index contains backup data,

Re: Input data is no significant change in multi-threading

2017-04-25 Thread Andrey Mashenkov
<ignite.charl...@gmail.com> wrote: > Same data set mean that I separate original data into 2 parts & input > them from 2 separate programs. > E.g. a data set with id 1 - 100. Program A input id 1-50. Program B input > 51 - 100. > > 2017-04-21 17:24 GMT+08:00 Andrey Mashenkov

Re: Basic question on scaling of reads with ignite sql queries

2017-04-25 Thread Andrey Mashenkov
Hi Tuco, Backup partition is full copy of primary partition, so you will run sql query on same data. There is no need to run query on backups as we can run in on primaries in multiple thread with same effect. We already test approach with running query in multiple threads when every thread works

Re: IgniteCacheProxy connection failure in REPLICATAED mode

2017-04-24 Thread Andrey Mashenkov
Hi Rick, It 100% reproduces on ignite-1.9, but ignite-2.0 looks fine. Looks like put and remove can fails regardless of cache modes. PFA the reproducer I've made. Seems, it can failed when node tries to register key class in marshaller cache while other node leaves topology. System marshaller

Re: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing - Query execution is too long

2017-04-24 Thread Andrey Mashenkov
, Abhijit Prusty <abhijitpru...@fico.com> wrote: > Hi Andrey, > > > > So, is there a limitation on the threads for query parallelism . Could you > pass any sample configuration ? > > > > Thanks, > > Abhijit > > > > *From:* Andrey Mashenkov [mailto:

Re: IgniteCacheProxy connection failure in REPLICATAED mode

2017-04-24 Thread Andrey Mashenkov
Hi Rick, Is it full stacktrace? Looks like transaction commit was failed when primary node has left the grid. Looks weird. Could you check this transaction was successfully restarted and commited at last? On Fri, Apr 21, 2017 at 6:21 PM, rick_tem wrote: > Hey Andre, > >

Re: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing - Query execution is too long

2017-04-24 Thread Andrey Mashenkov
Hi, Looks like your query use unapropriate index "CustomIgniteCacheConsumer"." SUPPRESSION_FLAG_idx": You can try to add index with better selectivity. It is possible you already have it and issue is caused by H2 query planner. Index hint support will be added in ignite-2.0 that come very soon.

Re: stdout - Message queue limit is set to 0, potential OOMEs

2017-04-24 Thread Andrey Mashenkov
Hi Sam, If you see messageQueueSize is often close to limit and low CPU pressure on nodes, then limit value increasing can be helpful. On Fri, Apr 21, 2017 at 11:36 PM, javastuff@gmail.com < javastuff@gmail.com> wrote: > Added below to TcpCommunicationSpi and message went away, able to

Re: disable fault tolerance

2017-04-24 Thread Andrey Mashenkov
able and disable. > > thanks > > 2017-04-20 17:41 GMT-03:00 Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> Hi Andre, >> >> What fault tolerance do you mean? Ignite has a number of modules. >> >> On Thu, Apr 20, 2017 at 9:54 PM, André Luis Alm

Re: Distributed Closures VS Executor Service

2017-04-24 Thread Andrey Mashenkov
Hi, 1. Your query uses index "PPSP-IMDG-CACHE"."type_idx" that included single field. Most probably it is due to H2 planner lack. Unfortunately, current H2 version does not support index hints. This should be fixed in ignite-2.0. 2. SQL queries are running in single thread by default. But it is

Re: Index

2017-04-21 Thread Andrey Mashenkov
Hi Sam, Default is 'null' obviously, it is by bad design and here is a ticket [1]. It is already fixed in 2.0, where index is SORTED by default. [1] https://issues.apache.org/jira/browse/IGNITE-4511 On Fri, Apr 21, 2017 at 10:00 PM, javastuff@gmail.com < javastuff@gmail.com> wrote: >

Re: IgniteCacheProxy connection failure in REPLICATAED mode

2017-04-21 Thread Andrey Mashenkov
Hi Rick, Looks weird. Cache operation on replicated cache on server node should not failed when other node leaved. Would you please attach logs? Ignite 2.0 ETA should be release on next week. But seems, it may delays for 1-2 weeks as there is neither node freeze happened nor vote started. On

Re: Input data is no significant change in multi-threading

2017-04-21 Thread Andrey Mashenkov
Hi Woo, DataStreamer is designed to fill cache with maximum throughput. By default, streamer will not rewrite cache data, until allowOverwite option is set. Why you need to input same set of data? Why do you expected data input time will change significantly with 2 programs compared to 1 if data

Re: disable fault tolerance

2017-04-20 Thread Andrey Mashenkov
Hi Andre, What fault tolerance do you mean? Ignite has a number of modules. On Thu, Apr 20, 2017 at 9:54 PM, André Luis Almeida wrote: > Igniters, > > Is it possible to disable fault tolerance in Apache Ignite? > > thanks, > > > André Luís B. Almeida > -- Best

Re: FailureDetectionTimeOut not working

2017-04-20 Thread Andrey Mashenkov
Hi, Ignite uses ring topology. Almost all network exchange done via TcpDiscoverySPI [1] (see, you bind it to port 47500). Topology updates, cluster hearbeat use it. Failure detection timeout is a time window when every node should send update to the NEXT node in topology via Discovery. Also,

Re: How to do write-behind caching?

2017-04-20 Thread Andrey Mashenkov
Hi Ricky, Here is a link to documentation [1]. Write behind will perform in async way. Yes, you should wait 5 sec for flush. [1] https://apacheignite.readme.io/docs/persistent-store#write-behind-caching On Thu, Apr 20, 2017 at 5:56 AM, Ricky Nauvaldy wrote: >

Re: Input data is no significant change in multi-threading

2017-04-20 Thread Andrey Mashenkov
& buffer in server > node. is that correct? > If I correct, this data will buffer in random server node or only the one > it directly connected? > > 2017-04-19 18:33 GMT+08:00 Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> It may have effect if you prep

Re: IgniteCacheProxy connection failure in REPLICATAED mode

2017-04-20 Thread Andrey Mashenkov
Hi Looks like master has gone ahead. Try this one. On Tue, Apr 18, 2017 at 4:51 PM, rick_tem wrote: > Hi, > > I appreciate the last response to this. I've checked out the 1.9 branch > (ie. git clone -b ignite-1.9 --single-branch > https://github.com/apache/ignite.git

Re: stdout - Message queue limit is set to 0, potential OOMEs

2017-04-20 Thread Andrey Mashenkov
Actual 'outbound queue size' are printed in node mertics messages of accessible via mbean ClusterLocalNodeMetricsMXBean. On Wed, Apr 19, 2017 at 11:13 AM, javastuff@gmail.com < javastuff@gmail.com> wrote: > What do you mean by messages? I am not using Ignite messaging. Are these >

Re: Ignite errors in log

2017-04-19 Thread Andrey Mashenkov
Hi Rishi, This means your node found another node with different 'java.net.preferIPv4Stack' value. Please, check if there is no other node running? On Wed, Apr 19, 2017 at 4:58 AM, Rishi Yagnik wrote: > Hello Andrew, > > I have applied IPv4 setting on both ignite

Re: Input data is no significant change in multi-threading

2017-04-19 Thread Andrey Mashenkov
at 12:29 PM, woo charles <ignite.charl...@gmail.com> wrote: > Is that mean the performance of input data will not affect if I use 2 > IgniteDataStreamer(2 > client program) to input data as they use the same queue in remote nodes? > > 2017-04-19 10:02 GMT+08:00 Andrey Mas

Re: Use of Ignition.allGrids()

2017-04-19 Thread Andrey Mashenkov
Hi, There are number of methods with confising names. Actually, gridName means ignite instance name. It will be fixed in 2.0 release. 1. Ignition.allGrids() return local JVM node instances. If you need all cluster nodes, see Ignite.cluster() method. 2. Both of you nodes belongs to same cluster

Re: stdout - Message queue limit is set to 0, potential OOMEs

2017-04-19 Thread Andrey Mashenkov
Hi Sam, Ignite use messages for inter-node communication. You have to configure TcpCommunicationSPI in IgniteConfiguretion. Method TcpCommunicationSPI.setMessageQueueLimit() is what you need. On Wed, Apr 19, 2017 at 11:13 AM, javastuff@gmail.com < javastuff@gmail.com> wrote: > What do

Re: stdout - Message queue limit is set to 0, potential OOMEs

2017-04-18 Thread Andrey Mashenkov
Hi Sam, Slow node can cause OOME, as messages for slow node can make queue grows infinitly. Also, you can receive messages with higher rate than node able to process, that also can results in OOME. On Tue, Apr 18, 2017 at 10:13 PM, javastuff@gmail.com < javastuff@gmail.com> wrote: >

Re: Input data is no significant change in multi-threading

2017-04-18 Thread Andrey Mashenkov
Hi Woo, IgniteDataStreamer uses per node buffer to make bulk cache updates that shows much better throughput than single updates. Also, IgniteDataStreamer send jobs to remote nodes, to utilize multiple threads on remote nodes. In multi-node grid IgniteDataStreamer usually shows better results

Re: ignite-indexing IncompatibleClassChangeError

2017-04-18 Thread Andrey Mashenkov
Hi David, Looks like your H2 version is not compatible with ignite. Please, check if correct version of H2 is in classpath. On Wed, Apr 19, 2017 at 4:37 AM, David Li wrote: > I have a working ignite cluster and clients. However, then I added the > ignite-indexing to the

Re: Node left topology

2017-04-18 Thread Andrey Mashenkov
Hi Anil 1. It shouldn't. Grid heartbeat uses separate thread. However, high CPU or network pressure with low failure detection timeout can be a reason. 2. Do you mean client doesn't reconnect to grid at all, or it reconnect to smaller part after split-brain? If first, then there is a bug [1] that

Re: Ignite errors in log

2017-04-18 Thread Andrey Mashenkov
Hi Rishi, Would you please check if both nodes has same either java.net.preferIPv4Stack=true or java.net.preferIPv6Stack=true option? "remote_host/remote_host:47102" looks weird. Would you also check if all dns names are correctly resolved on both nodes? On Tue, Apr 18, 2017 at 9:29 PM, Rishi

Re: Basic question on scaling of reads with ignite sql queries

2017-04-18 Thread Andrey Mashenkov
Hi Tuco, ReadFromBackup works only for cache operations. On Tue, Apr 18, 2017 at 10:02 AM, tuco.ramirez wrote: > There is an option 'readFromBackup' in CacheConfiguration. > Does it work for both cache.get() calls and SQL queries or only for > cache.get() calls? >

Re: Basic question on scaling of reads with ignite sql queries

2017-04-17 Thread Andrey Mashenkov
Hi, There is no sense to set QueryParallelism higher than number of available processors. On Mon, Apr 17, 2017 at 9:07 PM, tuco.ramirez wrote: > Thanks, > The default value of queryParallelism is 1. What is the recommended value? > Does it depend on cores etc of the

Re: Basic question on scaling of reads with ignite sql queries

2017-04-17 Thread Andrey Mashenkov
Hi, Actually, you cache is splitted to partitions that distributed among cache data nodes. With no of backups > 0, you will just have additional partition copies in grid and there is a guarantee you will never have more than 1 copy of data within every single node. Also, every node can be either

Re: Repeatable cache updates

2017-04-17 Thread Andrey Mashenkov
Hi, There is no need to add some reciever. IgniteDataStreamer.addData() will send data automatically to affinity node. Streamer use buffers to fill cache in efficient way, so data may be put to cache with few delay. You can flush Streamer buffer manually if needed and do not forget to close

Re: Ignite Node Discovery IP config

2017-04-17 Thread Andrey Mashenkov
Hi, Every node can participate in single grid only. If you need to connect to 2 (or more) grids, then you should create a client node per grid. If you need to divide grid into zones, then AffinityFunction should be used. See [1] and [2] for more details. [1]

Re: Basic question on scaling of reads with ignite sql queries

2017-04-14 Thread Andrey Mashenkov
Hi, Please properly subscribe to the mailing list so that the community can receive email notifications for your messages. To subscribe, send empty email to user-subscr...@ignite.apache.org and follow simple instructions in the reply. Yes, for now, query will be send to all Item cache data

Re: Ignite Node Discovery IP config

2017-04-14 Thread Andrey Mashenkov
Hi Ramzinator, This will not work in that way. All nodes in grid should be able to see each other as via dicovery ports as via communication ports. Node1 starts and found no one. So, you have first grid. Node2 starts and found no one. Now, you have second grid. Node3 starts and join to one of

Re: Repeatable cache updates

2017-04-12 Thread Andrey Mashenkov
Hi 2) Also you can subscribe to local node events and update other caches when event is received on primary node for key to avoid excessive calls. On Wed, Apr 12, 2017 at 7:35 PM, Andrey Mashenkov < andrey.mashen...@gmail.com> wrote: > Hi, > > 3) Initial query is not mandato

Re: Repeatable cache updates

2017-04-12 Thread Andrey Mashenkov
Hi, 3) Initial query is not mandatory, you can set it to null to receive updates only. Also, you can try CacheInterceptor to update other caches synchronously. It can cause performance issues as CacheInterceptor methods are called from sensitive part of code inside synchronized block, but it

Re: Client got stucked on get operation

2017-04-12 Thread Andrey Mashenkov
10) > at java.lang.Thread.run(Thread.java:745) > > After that exchange-worker dies and all put and get operations blocks. > > 1 - What does above error mean? When does it happen? > 2 - Is there a way to avoid or recover from that state? > > Regards. > > On Thu, Apr 6, 2

Ignite server showing this error class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues)

2017-04-12 Thread Andrey Mashenkov
Hi, Please properly subscribe to the mailing list so that the community can receive email notifications for your messages. To subscribe, send empty email to user-subscr...@ignite.apache.org and follow simple instructions in the reply. Stopping node should cancel all its tasks automatically.

Re: Ignite may start more service instances per node than maxPerNodeCount in case of manual redeploy

2017-04-12 Thread Andrey Mashenkov
Hi Evgeniy, I've fixed the issue. PFA a patch. On Tue, Apr 4, 2017 at 12:23 AM, Andrey Mashenkov < andrey.mashen...@gmail.com> wrote: > Hi Evgeniy, > > Looks like a bug. I've created a ticket [1]. > Seems, this reporoduced only when service deployment with dynamic > approac

Re: Local Mode - Kafka Streamer unable to write data

2017-04-10 Thread Andrey Mashenkov
Hi Santosh, KafkaStreamer has 10 seconds retry timeout. Try to reduce it. Please, share you config and sample code if it would not helpful. On Fri, Apr 7, 2017 at 5:12 PM, Santosh Pingale wrote: > To add to this, I have set autoFlushFrequency to 10. This frequency

Re: External data sources

2017-04-07 Thread Andrey Mashenkov
Hi Dmitry, You can try to create additional table with only row ID and field for join. So, you will be able to make joins, but you will need to retrieve detailed data from external store manually. To speed up external store lookups, you can create one more cache with eviction policy backed with

Re: IgniteCacheProxy connection failure in REPLICATAED mode

2017-04-06 Thread Andrey Mashenkov
Hi Rick, Looks like it is fixed [1] in master and will be available in 2.0 version. You can try to apply a patch to 1.9 version. PFA patch. [1] https://issues.apache.org/jira/browse/IGNITE-4473 On Thu, Apr 6, 2017 at 12:05 PM, rick_tem wrote: > Hi, > > I have a cache

Re: External data sources

2017-04-06 Thread Andrey Mashenkov
Hi Dmitri, There is no way to make SQL join with external data source. SQL engine requires all data should be in cache. On Thu, Apr 6, 2017 at 1:48 AM, Dmitri Bronnikov wrote: > What's the best way to implement SQL that joins an Ignite cache with an > external

Re: Client got stucked on get operation

2017-04-06 Thread Andrey Mashenkov
the attachment. > > Interruption exception is on 30/Mar/2017 15:56:03. starvation log is on > the second node. > > I do not have threaddump. I can provide if the problem repeats. > > Thanks for your helps! > > On Thu, Apr 6, 2017 at 4:27 PM, Andrey Mashenkov < >

Re: Client got stucked on get operation

2017-04-06 Thread Andrey Mashenkov
Hi Alper, Would you please provide full treaddump and full log? On Thu, Apr 6, 2017 at 4:08 PM, nragon wrote: > Hi Andrew, > > Please note that the same flink job without ignite runs around 30k/s with > ignite get method goes to 2k/s. If you don't mind

Re: Client got stucked on get operation

2017-04-06 Thread Andrey Mashenkov
Hi, Looks like not related. Most of time spent in non-ignite methods, but in flink serializer. Also I see a contention in flink Serializer. On Thu, Apr 6, 2017 at 2:27 PM, nragon wrote: > Hi, > > Might be related with: >

Re: Client got stucked on get operation

2017-04-06 Thread Andrey Mashenkov
al...@evam.com> wrote: > Hi Andrey. > > You can find configuration files and basic structure in the attachment. > App servers connects cache server as client programmatically so no > configuration for that. > > > On Tue, Apr 4, 2017 at 6:15 PM, Andrey Mashenkov < &

Re: Error in executing hadoop job using ignite

2017-04-05 Thread Andrey Mashenkov
Hi, Please properly subscribe to the mailing list so that the community can receive email notifications for your messages. To subscribe, send empty email to user-subscr...@ignite.apache.org and follow simple instructions in the reply. It looks like you have outdated "objectweb-asm" jar library

Re: ScanQuery With BinaryObject

2017-04-05 Thread Andrey Mashenkov
Hi David, Thanks for reproducer. I've create a ticket [1] for this bug. [1] https://issues.apache.org/jira/browse/IGNITE-4918 On Wed, Apr 5, 2017 at 5:03 PM, Andrey Mashenkov <andrey.mashen...@gmail.com > wrote: > I've creted a ticket [1]. > > [1] https://issues.apache.org/jir

Re: Apache ignite persistant store implementation

2017-04-05 Thread Andrey Mashenkov
o" ? > > вт, 4 апр. 2017 г. в 13:52, Andrey Mashenkov <andrey.mashen...@gmail.com>: > >> Client can have only LOCAL cache and NearCache that is frontend to >> partitioned, see [1]. Moreover, client node requires live server node in >> topology. >> >>

Re: Apache Ignite WriteBehind Guarantee during a Graceful Shutdown

2017-04-05 Thread Andrey Mashenkov
Hi Jaipal, All data should be flushed to back store automatically on graceful shutdown. On Wed, Apr 5, 2017 at 10:44 AM, Jaipal Reddy wrote: > Hii, > > We are using Apache Ignite for caching and using its writeBehind feature > for pushing updated data asynchronoulsy

Re: Local node SEGMENTED. Data Lost on a Partitioned Cache with backups of 3

2017-04-04 Thread Andrey Mashenkov
Hi Raja, All should work out-of-box. Would you please share grid configuration and logs? On Tue, Apr 4, 2017 at 7:22 PM, Raja wrote: > Yes we fill the cache manually after a cluster is started. > > We load data to Ignite for use it for ultra fast access and fast updates

Re: Client got stucked on get operation

2017-04-04 Thread Andrey Mashenkov
pplication servers or while > accessing caches on cache server. > > Our cached on application servers are affinity based so we only access > records on local server. > > > > > On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov < > andrey.mashen...@gmail.com> wrote: > &g

Re: Client got stucked on get operation

2017-04-04 Thread Andrey Mashenkov
Hi Alper, Thread is waiting for result of remote get operation. Are you sure your server 1 is standalone and has its own topology? I can't understand, how 1 and 3 connects to 1 as client. Would you please clarify? On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp wrote: > Hi. > >

Re: Ignite Logger error

2017-04-04 Thread Andrey Mashenkov
Hi Rick, Program aborts with ClassNotFoundException or NoClassDefFoundError? Would you please attach full stacktrace? On Tue, Apr 4, 2017 at 12:47 PM, rick_tem wrote: > Hi, > > I have the following code where log4JFile is a valid log4j2 conf file...If > however,

Re: Apache ignite persistant store implementation

2017-04-04 Thread Andrey Mashenkov
rs and use LOCAL cache with read-through. [1] https://apacheignite.readme.io/docs/clients-vs-servers#creating-distributed-caches On Tue, Apr 4, 2017 at 10:56 AM, ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com > wrote: > client node doesnt store cache entries, at all ? > > вт, 4 ап

Re: Local node SEGMENTED. Data Lost on a Partitioned Cache with backups of 3

2017-04-04 Thread Andrey Mashenkov
Hi Raja, You are right, when node left the grid new backup should be created if at least one copy of data exists. I mean that you can loose data if you start to load data on a node which left the grid unexpectedly, so some of data would be on segmented node only. As you say you use Hadoop/Yarn,

Re: Apache ignite persistant store implementation

2017-04-04 Thread Andrey Mashenkov
Hi Sweta, You should to start server node to Ignite could keep cache data in memory. On Tue, Apr 4, 2017 at 8:43 AM, sweta Das wrote: > Hi > I have few clarifications on the read and write through implementation of > ignite cache. > I have a mysql database with a simple

Re: Local node SEGMENTED. Data Lost on a Partitioned Cache with backups of 3

2017-04-03 Thread Andrey Mashenkov
Hi Raja, Data loss can happend if rebalance is not finished by some reason between nodes failures. Also you can loss data that is being loaded from segmented node when failure happens. Would you please attach ignite configuration? On Mon, Apr 3, 2017 at 9:43 PM, Raja

Re: Ignite may start more service instances per node than maxPerNodeCount in case of manual redeploy

2017-04-03 Thread Andrey Mashenkov
Hi Evgeniy, Looks like a bug. I've created a ticket [1]. Seems, this reporoduced only when service deployment with dynamic approach. When Ignite starts with service configured - all looks fine for me. [1] https://issues.apache.org/jira/browse/IGNITE-4907 On Mon, Apr 3, 2017 at 4:26 PM, Evgeniy

Re: ScanQuery With BinaryObject

2017-04-03 Thread Andrey Mashenkov
Hi David, Scan query results are never cached. It looks like your IgniteBiPredicate implementation is cached on server side. Try to move this class to upper level or make it inner and make "prefix" configurable with class constructor. This should work. On Mon, Apr 3, 2017 at 9:24 AM, David Li

Re: Storing JSON data on Ignite cache

2017-04-03 Thread Andrey Mashenkov
Hi Austin, There is no way to query binary data as Ignite does not know its structure. You should deserialize data before store it with Ignite or store it as BinaryObjects. The first approach required all Ignite nodes should have data structure classes in their classpathes. The second one

Re: SpringBoot Integration

2017-04-03 Thread Andrey Mashenkov
Hi Vishal, See Ignite exmples source code: - org.apache.ignite.examples.misc.springbean.SpringBeanExample. - org.apache.ignite.examples.datagrid.store.spring.CacheSpringStoreExample Also test project [1] may be helpful [1]

Re: ScanQuery With BinaryObject

2017-03-31 Thread Andrey Mashenkov
ta into the cache, the newly added data > will be returned as per the first query. It looks like the first query has > been kept somewhere. > > > > > On Thu, Mar 30, 2017 at 9:02 PM, Andrey Mashenkov < > andrey.mashen...@gmail.com> wrote: > >> Hi David, >> >

<    1   2   3   4   5   6   >