Re: Native persistence and upgrading

2020-12-14 Thread Stanislav Lukyanov
Yes, you can upgrade from an older version to a newer one and keep the data, it will just work. You don't really need snapshots for that, although I assume snapshots would also work. > On 10 Dec 2020, at 16:20, xero wrote: > > Hi Dimitry and community, > Is this still true? My intention is to

Re: High availability of local listeners for ContinuousQuery or Events

2020-11-13 Thread Stanislav Lukyanov
Hi, It's true that currently you need to implement something for Continuous Queries failover in your application code. Continuous Queries have setInitialQuery API to help with that. How it's supposed to work is: you define an initial query (e.g. an SQL query) which fetches the data that's

Re: Ignite timeouts and trouble interpreting the logs

2020-11-12 Thread Stanislav Lukyanov
Hi, This looks weird but with the right logs we should figure it out. One thing that I don't like about these settings is the asymmetry of the server's and client's timeouts. The server will use clientFailureDetectionTimeout=30s when talking to the client. The client will use

[Webinar] Networking in Apache Ignite - Talk at Sep 9 2020

2020-09-09 Thread Stanislav Lukyanov
Hi Igniters, I'll be giving a webinar titled Networking in Apache Ignite. We'll look at Apache Ignite's networking components - Discovery and Communication - to see how they work together to implement various networking functions. The webinar will be held at 10 AM PT / 1 PM ET / 5 PM GMT.

Re: apache-ignite compatibility with armhf(32-bit arm linux)

2020-07-24 Thread Stanislav Lukyanov
It should work fine. I ran Ignite on various platforms non-Intel platforms, including arm. There were some issues in the past but modern versions work well. But you do have to keep in mind that release testing for Ignite is done on Intel platforms. Also arm can bring some surprises in terms of

[Meetup] The Role and Specifics of Networking - Talk at June 11 2020 for Bay Area IMC Meetup

2020-06-10 Thread Stanislav Lukyanov
Hi Igniters, Tomorrow I'll be talking at an online meetup of the Bay Area In-Memory Computing community. The subject is The Role and Specifics of Networking in Distributed Systems. We'll use Apache Ignite's protocols as an example - experienced Ignite users will guess that we'll be looking at

Re: PutAll Behavior Single vs Multiple Servers

2019-12-11 Thread Stanislav Lukyanov
This is a very common pitfall with distributed systems - comparing 1 node vs 3 nodes. In short, this is not correct to compare them. When you write to one node each write does the following: 1) client sends the request to the server 2) server updates data 3) server sends the response to the

Re: Local node terminated after segmentation

2019-12-11 Thread Stanislav Lukyanov
In Ignite a node can go into "segmented" state in two cases really: 1. A node was unavailable (sleeping. hanging in full GC, etc) for a long time 2. Cluster detected a possible split-brain situation and marked the node as "segmented". Yes, split-brain protection (in GridGain implementation and in

Re: Client (weblogic) attempting to rejoin cluster causes shutdown.

2019-12-10 Thread Stanislav Lukyanov
Ok, there is a lot to dig through here but let me try with establishing simple things first. 1. If two nodes (client or server) have the same cache specified in the configuration, the configs must be identical. 2. If one node has a cache configuration, it will be shared between all nodes

Re: How sql works for near cache

2019-12-10 Thread Stanislav Lukyanov
Not out of the box but you could use SQL or ScanQuery for that. With SQL: SELECT _key FROM mycache (given that your cache is SQL-enabled). With ScanQuery: cache.query(new ScanQuery(), Cache.Entry::getKey) (may need to fix type errors to compile this) Stan On Wed, Dec 4, 2019 at 2:36 AM

Re: Error while adding the node the baseline topology

2019-11-05 Thread Stanislav Lukyanov
This message actually looks worrisome: 2019-10-22 10:31:42,441][WARN ][data-streamer-stripe-3-#52][PageMemoryImpl] Parking thread=data-streamer-stripe-3-#52 for timeout (ms)=771038 It means that Ignite's throttling algorithm has decided to put a thread to sleep for 771 seconds. Can you share

Re: How to insert data?

2019-11-05 Thread Stanislav Lukyanov
There are multiple ways to configure a cache to use SQL. The easiest is to use @QuerySqlField annotation. Check out this doc https://www.gridgain.com/docs/8.7.6/developers-guide/SQL/sql-api#querysqlfield-annotation . On Tue, Nov 5, 2019 at 5:52 PM BorisBelozerov wrote: > I have 3 nodes, and I

Re: ignite-rest-http in classpath error

2019-11-05 Thread Stanislav Lukyanov
Hi, Web Console requires ignite-rest-http module to be enabled. It is not enabled by default in Ignite binaries nor Docker image. The steps that you've taken are done while the container is running - so, AFTER the Ignite process has started. That's why copying the module has no effect. Try

Re: Ignite singel instance

2019-11-05 Thread Stanislav Lukyanov
First, 1700 TPS given your transaction structure is 17 simple operations per second, which is quite substantial - especially if you're doing that from a single thread / single ODBC client. Second, note that TRANSACTIONAL_SNAPSHOT is in beta and is not ready for production use. There are no

Re: Cluster in AWS can not have more than 100 nodes?

2019-11-05 Thread Stanislav Lukyanov
Each node is supposed to add its own IP and port to the S3 bucket when it starts. That said, I wouldn't check the cluster state based on the contents of the bucket alone. Check your logs for errors. Try using some tools (e.g. check out Web Console - either the one in Ignite

Re: How does Apache Ignite distribute???

2019-11-05 Thread Stanislav Lukyanov
I believe that the correct answer to your question - don't do that. The strength of distributed systems is that you have a number of identical pieces which you can scale out virtually with no limits. If your cluster is heterogenous - i.e. all the nodes are different in size, amount of data and

Re: Query execution in ignite

2019-11-01 Thread Stanislav Lukyanov
This is not exactly correct. When you do an SQL query with only PARTITIONED tables, or with a mix of PARTITIONED and REPLICATED, the data will be taken from the primary partions of PARTITIONED tables and *all* partitions of REPLICATED tables. When you do an SQL query with only REPLICATED tables,

Re: Throttling getAll

2019-11-01 Thread Stanislav Lukyanov
The right answer to this is probably not to use getAll in such cases. If you want to load data in batches then you should either split the keys yourself or use Query APIs, like ScanQuery or SqlQuery. Stan On Mon, Oct 28, 2019 at 10:36 PM Abhishek Gupta (BLOOMBERG/ 919 3RD A) <

Re: Drop index do not release memory used ?

2019-11-01 Thread Stanislav Lukyanov
Hi, What version do you use? There was an issue with recycling pages between data and indexes which has been fixed in 2.7 https://issues.apache.org/jira/browse/IGNITE-4958. In AI 2.7 and later this should be working fine. Stan On Sat, Oct 26, 2019 at 5:22 PM yann Blazart wrote: > Yes the data

Re: Question on submitted post

2019-08-26 Thread Stanislav Lukyanov
es into the > libs folder in ignite but that has also not helped. > > On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov > wrote: > >> Hi, >> >> It looks like the issue is that you're ending up sending an instance of >> your gRPC server inside your service.

Re: Ignite WAL and WAL archive size estimation

2019-08-26 Thread Stanislav Lukyanov
Hi, In normal circumstances checkpoint is triggered on timeout, e.g. every 3 minutes (controlled by checkpointFrequency). So, the size of the checkpoint is the amount of data written/updated in a 3-minute interval. The best way to estimate it in your system is to enable data storage metrics

Re: HTTP/2 implementations do not robustly handle abnormal traffic and resource exhaustion

2019-08-26 Thread Stanislav Lukyanov
Hi, AFAICS this is not about the *protocol*, this is about *implementations* of the protocol. I've followed the links and found this matrix of vulnerable technologies: https://vuls.cert.org/confluence/pages/viewpage.action?pageId=56393752 >From this matrix, Ignite uses only Node.js in WebConsole,

Re: Support for latest version of MongoDB in Ignite web console

2019-08-26 Thread Stanislav Lukyanov
Hi, I believe support for MongoDB 4.x is already implemented in https://issues.apache.org/jira/browse/IGNITE-10847. Also, I believe Ignite doesn't require a specific version of MongoDB. Have you tried to install the latest 3.4.x version? Thanks, Stan On Sun, Aug 25, 2019 at 7:04 PM Ashfaq

Re: Cache Miss Using Thin Client

2019-08-23 Thread Stanislav Lukyanov
Hi, I'm thinking this could be related to differences in the binary marshaller configuration. Are you using Java thin client? What version? What is the cache key type? Are you setting a BinaryConfiguration explicitly on the client or server? Thanks, Stan On Fri, Aug 23, 2019 at 3:38 PM wrote:

Re: Question on submitted post

2019-08-23 Thread Stanislav Lukyanov
Hi, It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect. What you should do is - not pass gRPC to the service instance - add an init() method implementation to your service - in your init() start your

Re: ZooKeeper Discovery - Handling large number of znodes and their cleanup

2019-08-23 Thread Stanislav Lukyanov
Hi Abhishek, What's your Ignite version? Anything else to note about the cluster? E.g. frequent topology changes (clients or servers joining and leaving, caches starting and stopping)? What was the topology version when this happened? Regarding the GC. Try adding

Re: One of Ignite pod keeps crashing and not joining the cluster

2019-08-22 Thread Stanislav Lukyanov
Hi, Please share - Ignite version you're running - Exact steps and events (a node was restarted, a client joined, etc) - Logs of all three servers Thanks, Stan On Mon, Aug 19, 2019 at 3:27 PM radha jai wrote: > Hi , > Ignite being deployed on the kubernetes, there were 3 replicas of ignite >

Re: Ignite backup/restore Cache-wise

2019-08-22 Thread Stanislav Lukyanov
GridGain Snapshots allow you to take a backup on a live, working cluster. If you can allow to stop the cluster activity while snapshot is being taken you can: - Deactive the cluster (e.g. control.sh --deactivate) - Copy the persistence files (you would need work/binary_meta, work/marshaller,

Re: SQL delete command is slow and can cause OOM

2019-05-08 Thread Stanislav Lukyanov
Hi, You have to also fetch values to do a "compare-and-delete". Before deleting each entry you check if it has been concurrently modified. If it was then it's possible that the entry doesn't match your WHERE anymore. So yes, for now deleting a large number of entries is heap-intensive. It should

Re: Ignite DataStreamer Memory Problems

2019-04-24 Thread Stanislav Lukyanov
Can you share your full configuration (Ignite config and JVM options) and the server logs of Ignite? Which version of Ignite you use? Can you confirm that on this version and configuration simply disabling Ignite persistence removes the problem? If yes, can you try running with walMode=NONE? It

Re: Ignite DataStreamer Memory Problems

2019-04-21 Thread Stanislav Lukyanov
I've put a full answer on SO - https://stackoverflow.com/questions/55752357/possible-memory-leak-in-ignite-datastreamer/55786023#55786023 . In short, so far it doesn't look like a memory leak to me - just a misconfiguration. There is a memory pool in JVM for direct memory buffers which is by

RE: GridGain perfomance

2019-03-27 Thread Stanislav Lukyanov
First, this is a mailing list for Apache Ignite, although the results would be more or less equal as GridGian is based on Ignite. Second, the question is too broad. You shouldn’t really think about running on 1 core as Ignite is for scaling to many cores and machines. The performance will vary

Re: Ignite Client getting OOM, GridAffinityProcessor grows in size

2019-03-27 Thread Stanislav Lukyanov
The memory leak looks very much like https://issues.apache.org/jira/browse/IGNITE-7918. Can you check on 2.7? Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Event listeners on servers only

2019-03-26 Thread Stanislav Lukyanov
The options I see 1. Register a local listener on each node; you can call localListen() from a broadcast() job or when the node starts. 2. Deploy a cluster-singleton service that calls remoteListen() in its initialize(). I guess the first one will perform better. Stan From: maros.urbanec

Re: Ignoring model fields

2019-02-20 Thread Stanislav Lukyanov
Regular Java `transient` keyword should suffice. Add it to all fields that shouldn't be serialized. Alan Ward wrote > Is there a way (preferably annotation-based) to exclude certain fields in > user-defined model classes from Ignite (cache, query, etc.), similar to > how > Jackson has a

Re: External multiple config files for Docker installation

2019-02-20 Thread Stanislav Lukyanov
Ignite's Docker image doesn't have a parameter for that. You can create your own Dockerfile extending Ignite's image and define and handle a new parameter there. Stan kyro wrote > Hi akurbanov, > > Yes, I have the lines as you mentioned in my config. I was asking as to > how > do I pass the

Re: Index inline size

2019-02-20 Thread Stanislav Lukyanov
Depends on the use case. Sometimes you want to save the memory as much as possible, and then you would use a lower inline size. However, in most cases you actually need a higher value because that will greatly improve the performance. Starting 2.7 there are warnings with a recommended size

RE: is peerClassLoadingEnabled in client mode makes difference

2019-01-23 Thread Stanislav Lukyanov
Yes, that’s actually the intended usage. Stan From: shivakumar Sent: 23 января 2019 г. 20:47 To: user@ignite.apache.org Subject: is peerClassLoadingEnabled in client mode makes difference when peerClassLoadingEnabled is enabled in client node which joins the cluster of servers, if any class/jar

RE: Baselined node rejoining crashes other baseline nodes - DuplicateKeyError

2019-01-23 Thread Stanislav Lukyanov
Hi, I’ve reproduced this and have a fix – I guess it’ll be available with 2.8. Meanwhile I can only suggest not to create indexes without an explicit name. Stan From: mahesh76private Sent: 16 января 2019 г. 12:39 To: user@ignite.apache.org Subject: RE: Baselined node rejoining crashes other

RE: sql fields visibility without sql indexing

2019-01-18 Thread Stanislav Lukyanov
There are implicit indexes needed to make SQL work – one on primary key, one on affinity key. They are also stored in index.bin, so it’s expected to see it growing. Stan From: Yuriy Sent: 18 января 2019 г. 13:37 To: user@ignite.apache.org Subject: sql fields visibility without sql indexing Hi.

RE: server always shutdown

2019-01-17 Thread Stanislav Lukyanov
Hi, You have to provide the details – configs, usage scenarios, error messages, logs. Without that no one will be able to tell what’s wrong with your cluster. Stan From: hulitao198758 Sent: 17 января 2019 г. 8:42 To: user@ignite.apache.org Subject: server always shutdown Ignite. Sh start

RE: Failing to create index on Ignite table column

2019-01-16 Thread Stanislav Lukyanov
Don’t think there is any Ignite API for that yet. Stan From: Shravya Nethula Sent: 16 января 2019 г. 12:42 To: user@ignite.apache.org Subject: RE: Failing to create index on Ignite table column Hi Stan, Thank you! This information is helpful. Do you know any Ignite API through which I can get

RE: Baselined node rejoining crashes other baseline nodes - Duplicate KeyError

2019-01-16 Thread Stanislav Lukyanov
Hi, Left a comment in the issue. In short, the problem is that you got a duplicate index on one of your nodes somehow, even though it shouldn’t happen. Need to figure out, how. Can you tell what you do with the cluster when it is running? I’m particularly interested in any of the actions

RE: Does set streaming off command flush data?

2019-01-16 Thread Stanislav Lukyanov
Yes. It closes the underlying streamers, which in turn flushes the data. Stan From: yangjiajun Sent: 16 января 2019 г. 11:41 To: user@ignite.apache.org Subject: Does set streaming off command flush data? Hello. The ignite's doc says we should close the JDBC/ODBC connection so that all data is

RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES

2019-01-16 Thread Stanislav Lukyanov
Yes. You can also use an environment variable instead of the system property: IGNITE_BPLUS_TREE_LOCK_RETRIES=10 ignite.sh … Stan From: mahesh76private Sent: 16 января 2019 г. 11:29 To: user@ignite.apache.org Subject: RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES how do I set it ?

RE: ignite continuous query with XML

2019-01-16 Thread Stanislav Lukyanov
No, you have to use actual code. Stan From: shivakumar Sent: 16 января 2019 г. 11:08 To: user@ignite.apache.org Subject: ignite continuous query with XML is there a way to configure continuous query using spring XML? is there any example or reference for configuring continuous query with XML?

RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES

2019-01-16 Thread Stanislav Lukyanov
It means that Ignite couldn’t find the place it needed in a B+ tree in 1000 iterations. It could mean either that there is a high contention on the tree (it changes a lot, and one thread is unlucky and couldn’t keep up with the speed), or that the tree is corrupted. Try to set a larger value

RE: Is there a way to allow overwrite when set streaming on?

2019-01-16 Thread Stanislav Lukyanov
Use `SET STREAMING ON ALLOW_OVERWRITE ON`. It’s a shame it’s not documented. Filed https://issues.apache.org/jira/browse/IGNITE-10952 for that. Stan From: yangjiajun Sent: 16 января 2019 г. 9:19 To: user@ignite.apache.org Subject: Is there a way to allow overwrite when set streaming on? Hello.

RE: SQLFieldsQuery timeout is not working

2019-01-15 Thread Stanislav Lukyanov
Hi, What’s your Ignite version? Can you share Ignite and cache configs and the query SQL? Thanks, Stan From: garima.j Sent: 15 января 2019 г. 14:18 To: user@ignite.apache.org Subject: SQLFieldsQuery timeout is not working Hello, I'm using the below code to execute a SQL fields query :

RE: Extra console output from logs.

2019-01-15 Thread Stanislav Lukyanov
Hi, First, try to disable IGNITE_QUIET. If still seeing duplicated messages after that, make sure you don’t have multiple slf4j adapters in the classpath. Let me know if that helps. Thanks, Stan From: javadevmtl Sent: 9 января 2019 г. 21:43 To: user@ignite.apache.org Subject: Re: Extra

RE: Thread got interrupted while trying to acquire table lock & Gotinterrupted while waiting for future to complete

2019-01-15 Thread Stanislav Lukyanov
Looks like the thread just ended. Do you see a similar issue? Do you have a reproducer? Stan From: bintisepaha Sent: 14 января 2019 г. 23:07 To: user@ignite.apache.org Subject: Re: Thread got interrupted while trying to acquire table lock & Gotinterrupted while waiting for future to complete

RE: SQL performance very slow compared to KV in pure-in-memory cache?

2019-01-14 Thread Stanislav Lukyanov
Do you use the same config for both runs? If you use ignite-sql.xml for SQL and ignite.xml for key-value than key-value keys indeed should be much faster, because SQL features are not used there and Ignite doesn’t have to maintain all the SQL indexes, etc. Stan From: summasumma Sent: 7 января

RE: Failed to read data from remote connection

2019-01-14 Thread Stanislav Lukyanov
Not really. The amount of direct memory needed doesn’t grow with the node count nor the amount of data you store. Stan From: wangsan Sent: 12 января 2019 г. 9:30 To: user@ignite.apache.org Subject: RE: Failed to read data from remote connection Yeath, set a larger MaxDirectMemorySize . But, I

RE: Ignite 2.7 Persistence

2019-01-11 Thread Stanislav Lukyanov
Running the query the first time isn’t really like loading all data into memory and then doing the query. I would assume that it is much less efficient – all kinds of locking and contention may be involved. Also, the reads are done via random disk access, while when reading from CSV you’re

RE: ignite-cassandra-store module has incorrect dependencies

2019-01-11 Thread Stanislav Lukyanov
FTR there is a JIRA issue for that fixed in 2.8 https://issues.apache.org/jira/browse/IGNITE-10856. Stan From: Serg Sent: 2 января 2019 г. 15:51 To: user@ignite.apache.org Subject: Re: ignite-cassandra-store module has incorrect dependencies Unfortunately I could not just change my pom

RE: TcpCommunicationSpi failed to establish connect to node node will bedropped from cluster.

2019-01-11 Thread Stanislav Lukyanov
Well, it means that node 6 didn’t answer to the node 2. Was the node 6 dropped from the cluster after that? It could happen because of a network issue, GC on the node 6, perhaps some OS/VM issue. Can’t say anything more specific with that little info. Stan From: sehii Sent: 4 января 2019 г.

RE: Do elements with EternalExpiryPolicy in cache ever get evicted?

2019-01-11 Thread Stanislav Lukyanov
Yes, that’s how EternalExpiryPolicy works. You can achieve the same by just not using an expiry policy at all – Ignite doesn’t evict entries by default. Stan From: rick_tem Sent: 7 января 2019 г. 12:38 To: user@ignite.apache.org Subject: Re: Do elements with EternalExpiryPolicy in cache ever

RE: How to debug network issues in cluster

2019-01-11 Thread Stanislav Lukyanov
+1 to all points. Generally, the message “Local node SEGMENTED” generally means that the cluster decided that the node is dead and kicked it out. The next time the node tried to send a message to the cluster, it received an answer “you’re segmented” meaning “we’ve kicked you out, sorry”. It

RE: Failing to create index on Ignite table column

2019-01-11 Thread Stanislav Lukyanov
I know one way to do that – connect via SQLLine and execute !indexes command. Check out this doc https://apacheignite-sql.readme.io/docs/sqlline Stan From: Shravya Nethula Sent: 7 января 2019 г. 13:49 To: user@ignite.apache.org Subject: Re: Failing to create index on Ignite table column Hi, Is

RE: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

2019-01-11 Thread Stanislav Lukyanov
for running a spark app using ignite in a spark cluster?  Sent from my iPhone On Jan 10, 2019, at 08:25, Stanislav Lukyanov wrote: Hi,   Were you able to solve this?   It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530.   Does it work

RE: Amazon S3 Based Discovery NOT USING BasicAWSCredentials

2019-01-10 Thread Stanislav Lukyanov
Hi, Were you able to solve this? It seems that your config is actually fine… The feature was added by https://issues.apache.org/jira/browse/IGNITE-4530. Does it work if you replace `ref` with just a value? Like Stan From: Max Barrios Sent: 12 декабря 2018 г. 23:51 To:

RE: Question about add new nodes to ignite cluster.

2019-01-10 Thread Stanislav Lukyanov
Here “cache start” is a rather internal wording. It means “cache adapter machinery will be initialized”. In case of ASYNC rebalancing the cache will first appear on the node as existing but storing no data until it is rebalanced. In practice, ASYNC rebalancing means that the node will start

Re: Do we require to set MaxDirectMemorySize JVM parameter?

2019-01-10 Thread Stanislav Lukyanov
> In my case, I have configured swap storage > (https://apacheignite.readme.io/docs/swap-space) but *not* Ignite durable > memory. If DataRegion maxSize is say 100GB and my physical RAM is 50GB > then > the swap file will be 100GB but Ignite will also use some portion (<50GB) > of > the

RE: Pain points of Ignite user community

2019-01-10 Thread Stanislav Lukyanov
: Re: Pain points of Ignite user community Thank you Stan. Denis, I don’t intend to speak for my employer. The content will be my personal opinion. Regards, Rohan On Nov 28, 2018, at 8:05 PM, Stanislav Lukyanov wrote: Hi,   I expect a write-up on some of the Ignite pitfalls to be out soon

RE: There is no property called StartSize in CacheConfiguration

2019-01-10 Thread Stanislav Lukyanov
The .Net page seems to be outdated. The startSize property isn’t there anymore. Check out the main one - https://apacheignite-net.readme.io/docs/performance-tips. Stan From: Peter Sham Sent: 9 декабря 2018 г. 8:22 To: user@ignite.apache.org Subject: There is no property called StartSize in

RE: Ignite 2.7 Persistence

2019-01-10 Thread Stanislav Lukyanov
Hi, That’s right, Ignite nodes restart “cold” meaning that they become operational without the data in the RAM. It allows to restart as quickly as possible, but the price is that the first operations have to load data from the disk, meaning that the performance will be much lower. Here is a

RE: How to define a cache template?

2018-12-31 Thread Stanislav Lukyanov
What do you mean by “global”? The settings of a cache template are not automatically applied to every cache. Cache template is just a way to reuse the same settings in multiple CREATE TABLE SQL commands. There is no such thing as “global cache settings” in Ignite. Stan From: yangjiajun Sent:

RE: Ouch! Argument is invalid: Cache name must not be null or empty.

2018-12-31 Thread Stanislav Lukyanov
As I’ve said in a thread nearby, there is no such thing as “global cache settings”. Usually you’ll need to repeat the configuration paratemers for all caches. You can avoid code/config duplication in multiple ways though. If you explain your use case in detail, I can suggest a way to do that.

RE: CAP Theorem (CP? or AP?)

2018-12-27 Thread Stanislav Lukyanov
There were two bugs actually, but the problem is basically the same, just in different cases. SQL + Partition Loss Policies issue (fixed in 2.7) - https://issues.apache.org/jira/browse/IGNITE-8927 (the issue says “hang” but the visible behavior actually varies) SQL + Partition Loss Policies +

RE: First write to new cache on new cluster.

2018-12-21 Thread Stanislav Lukyanov
Hi, Are you sure it is only the first write? I’m not aware of any lazy initialization there, so I wouldn’t expect the first write to be slow. Which version do you use? And what is the performance difference of the first transaction compared to the second? I have two ideas why it may work

RE: adding into baseline topology

2018-12-20 Thread Stanislav Lukyanov
Hi, Can you show the logs? Thanks, Stan From: Som Som Sent: 20 декабря 2018 г. 12:08 To: user@ignite.apache.org Subject: adding into baseline topology i am trying to add node into baseline topology:  using (var ignite = Ignition.StartFromApplicationConfiguration())             {    

RE: I encountered a problem when restarting ignite

2018-12-19 Thread Stanislav Lukyanov
Hi, Sending a SIGQUIT signal forces VM to print a thread dump to its stdout: kill -3 Stan From: Justin Ji Sent: 13 декабря 2018 г. 5:20 To: user@ignite.apache.org Subject: Re: I encountered a problem when restarting ignite Akurbanov - Thank for your reply! I have tried to dump the

RE: Effective way to pre-load data around 10 TB

2018-12-19 Thread Stanislav Lukyanov
The problem might be in HDD not performing fast enough, and also suffering from random reads (IgniteCache::preloadPartition at least tries to read sequentially). Also, do you have enough RAM to store all data? If not, you shouldn’t preload all the data, just the amount that fits into RAM.

RE: Question about add new nodes to ignite cluster.

2018-12-19 Thread Stanislav Lukyanov
Well, in short - it does, don’t worry about it :) Unfortunately I’m not aware of a proper design document explaining the process in detail. But simply put, Ignite will wait for the new node to obtain all of the data it needs to store. While that’s happening, the node doesn’t serve any requests.

RE: Failed to read data from remote connection

2018-12-19 Thread Stanislav Lukyanov
“OOME: Direct buffer memory” means that MaxDirectMemorySize is too small. Set a larger MaxDirectMemorySize value. Stan From: wangsan Sent: 18 декабря 2018 г. 5:08 To: user@ignite.apache.org Subject: Re: Failed to read data from remote connection Now the cluster have 100+ nodes, when 'Start

RE: Did anyone write custom Affinity function?

2018-12-19 Thread Stanislav Lukyanov
You could write a custom affinity function, and some people do, but as far as I can see you don’t need it. You just chose a poor affinity key. You need to have MANY affinity keys, much more than there are partitions, and have MANY partitions, much more than nodes. That will make sure that the

Re: BinaryObjectImpl.deserializeValue with specific ClassLoader

2018-12-04 Thread Stanislav Lukyanov
Hi, Unfortunately, it's not going to be near future. It's definitely not 2.7 (which was frozen a long time ago), almost definitely not 2.8 (because it would probably take to much time to do and 2.8 is supposed to figure out other stuff related to Java 9+ support). You can always workaround this

RE: When will Apache Ignite support Java 11?

2018-12-03 Thread Stanislav Lukyanov
The support for Java 9 *should* mean support for Java 11, the compatibility gap between the two is not big. Moreover, I would (and going to) push for almost completely skipping the testing on Java 9 – it is in end-of-life already, so providing support for it is kind of pointless. Java 11 is

RE: How to filter ip interfaces in TcpDiscoveryJdbcIpFinder

2018-12-03 Thread Stanislav Lukyanov
Currently you can only use IGNITE_LOCAL_HOST or TcpDiscoverySpi.localAddress for this. You can automate setting these addresses via an external script, like MY_IP=`ifconfig | grep ` java -DIGNITE_LOCAL_HOST=$MY_IP or put it into the Ignite config like Stan From: Luckyman Sent: 3

RE: Create index got stuck and freeze whole cluster.

2018-12-03 Thread Stanislav Lukyanov
Hi, The only thing I can say is that your troubles seem to have started way before. I see a bunch of “Found long running cache future” repeating, and then exchange for stopping SQL_PUBLIC_USERLEVEL cache that never completes. Would need logs going further (at least minutes) into the past to

RE: Avoiding Docker Bridge network when using S3 discovery

2018-12-03 Thread Stanislav Lukyanov
Hi, Have you been able to solve this? I think specifying TcpDiscoverySpi.localAddress should work. Stan From: Dave Harvey Sent: 17 октября 2018 г. 20:10 To: user@ignite.apache.org Subject: Avoiding Docker Bridge network when using S3 discovery When we use S3 discovery and Ignite containers

RE: Ignite can't activate

2018-12-03 Thread Stanislav Lukyanov
Hi, Reproduced that and filed https://issues.apache.org/jira/browse/IGNITE-10516. Thanks for reporting. Stan From: yangjiajun Sent: 29 ноября 2018 г. 10:52 To: user@ignite.apache.org Subject: Re: Ignite can't activate Hello. Here is a reproducer for my case: 1.Start a node with persistence

RE: Fair queue polling policy?

2018-12-03 Thread Stanislav Lukyanov
I think what you’re talking about isn’t fairness, it’s round-robinness. You can’t distribute a single piece of work among multiple nodes fairly – one gets it and others don’t. Yes, it could be using different node each time, but it I don’t really a use case for that. The queue itself isn’t a

RE: Ignite not scaling as expected !!! (Thread dump provided)

2018-12-03 Thread Stanislav Lukyanov
How many cores does each node have? Numbers of which threads do you increase? The ones doing the get() calls? Thread dump from the client isn’t that interesting. Better to look what’s going on the servers. You need to monitor your resources – CPU, Network IO, Disk IO. You may hit the limit on

RE: Ignite cache.getAll takes a long time

2018-12-03 Thread Stanislav Lukyanov
I guess it could be caused by https://issues.apache.org/jira/browse/IGNITE-5003 which is mentioned in that thread. Also, make sure that your cache store code doesn’t cause you troubles – that you don’t open a new connection every time, don’t have unnecessary blocking, etc. Stan From: Justin

RE: Query regarding Ignite unit tests

2018-12-03 Thread Stanislav Lukyanov
Hi, This is better to be asked on the dev-list – added that to the To, and Bcc’ed user-list. I actually don’t think you can run tests for a specific module – either a single test, or a single test suite, or all of them. I would usually either run a single test from IDEA or run all tests via

RE: Effective way to pre-load data around 10 TB

2018-11-29 Thread Stanislav Lukyanov
Hi, Currently the best option is IgniteCache::preloadPartition method added in https://issues.apache.org/jira/browse/IGNITE-8873. There is a JIRA ticket to allow pre-loading data before the node joins the cluster: https://issues.apache.org/jira/browse/IGNITE-10152. Stan From: Naveen Sent: 29

RE: some cases (IF NOT EXISTS) in the CREATE TABLE statement does notwork

2018-11-29 Thread Stanislav Lukyanov
A bug for this is filed: https://issues.apache.org/jira/browse/IGNITE-10414 Stan From: Qingping Sent: 27 ноября 2018 г. 4:26 To: user@ignite.apache.org Subject: some cases (IF NOT EXISTS) in the CREATE TABLE statement does notwork Question=== When testing Ignite 2.6.0 (2018-07-16),

RE: Client stucks and doesn't connect

2018-11-29 Thread Stanislav Lukyanov
Hi, The interesting (and disappointing) part is the NPE: java.lang.NullPointerException: null     at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:666)     at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)     at

RE: Ignite cache.getAll takes a long time

2018-11-29 Thread Stanislav Lukyanov
Hi, Start by looking at what’s going on in the cluster when you see these long reads. Collecting a JFR record or a GC log would be nice. Also, do you see that with concurrent reads and writes, i.e. do you have getAll and putAll executing in parallel? Perhaps there is some sort of contention

RE: Pain points of Ignite user community

2018-11-28 Thread Stanislav Lukyanov
Hi, I expect a write-up on some of the Ignite pitfalls to be out soon – ping me next week. Stan From: Rohan Honwade Sent: 29 ноября 2018 г. 0:42 To: user@ignite.apache.org Subject: Pain points of Ignite user community Hello, I am currently creating some helpful blog articles for Ignite

RE: java.lang.ClassCastException:org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl

2018-11-28 Thread Stanislav Lukyanov
Hi, Try a full fresh start: - stop all nodes - clean work directory (D:\ApacheIgnite2_6\ApacheIgnite\work) - make sure all nodes use the same configuration - start nodes again Stan From: userx Sent: 20 ноября 2018 г. 8:11 To: user@ignite.apache.org Subject: Re:

RE: How to restart Apache Ignite nodes via command line

2018-11-28 Thread Stanislav Lukyanov
Well, as you said you need to write some scripts. To shutdown the cluster you kill Ignite processes, to load new configurations – copy the configuration files. No magic here, and nothing specific to Ignite. Stan From: Max Barrios Sent: 28 ноября 2018 г. 0:40 To: user@ignite.apache.org

RE: PublicThreadPoolSize vs FifoQueueCollisionSpi.ParallelJobsNumber

2018-11-28 Thread Stanislav Lukyanov
Hi, 1) No. With publilcThreadPoolSize=16 and parallelJobsNumber=32 you’ll have 32 jobs submitted to executor with 16 threads, which means that 16 jobs will be executing and 16 will be waiting in the public thread pool’s queue. 2) Yes – see setWaitingJobsNumber. Stan From: Prasad Bhalerao

RE: Slow Data Insertion On Large Cache : Spark Streaming

2018-11-12 Thread Stanislav Lukyanov
Hi, Do you use persistence? Do you have more data on disk than RAM size? If yes, it’s almost definitely https://issues.apache.org/jira/browse/IGNITE-9519. If no, it still can be the same issue. Try running on 2.7, it should be released soon. Stan From: ApacheUser Sent: 5 ноября 2018 г. 20:10

RE: Cannot query cache by affinity key when custom cache template is used

2018-10-22 Thread Stanislav Lukyanov
Also crossposting from SO :) https://stackoverflow.com/questions/52879064/cannot-query-cache-by-affinity-key-when-custom-cache-template-is-used/52935802#52935802 Apparently, it's a bug in Ignite. Filed https://issues.apache.org/jira/browse/IGNITE-9964. Thanks for reporting! The issue only

RE: IGNITE-8386 question (composite pKeys)

2018-10-17 Thread Stanislav Lukyanov
index for (customer_id, date)? Cheers, Eugene On Mon, Sep 17, 2018 at 10:52 AM Stanislav Lukyanov wrote: Hi,   The thing is that the PK index is currently created roughly as     CREATE INDEX T(_key) and not     CREATE INDEX T(customer_id, date).   You can’t use the _key column in the WHERE clause

RE: Ignite complains for low heap memory

2018-10-17 Thread Stanislav Lukyanov
Put your -X* options before -jar. That’s how java command line works. Stan From: Lokesh Sharma Sent: 17 октября 2018 г. 14:08 To: user@ignite.apache.org Subject: Re: Ignite complains for low heap memory I get the same output in the logs when I run the application with Xms set to 2 GB. I ran

RE: How can I unsubscribe from the email server?

2018-10-16 Thread Stanislav Lukyanov
See the “unsubscribe” links here: https://ignite.apache.org/community/resources.html They’ll open your mail client with a draft of the message – just send it. Stan From: Lucky Sent: 16 октября 2018 г. 11:56 To: user@ignite.apache.org Subject: How can I unsubscribe from the email server? There

RE: Priority on affinityCalls

2018-10-16 Thread Stanislav Lukyanov
I personally feel that this should be handled on higher level. In my view, Ignite Compute subsystem is about simple delivery of a job to a remote server. It has failover and all, but to really use that you have to go for ComputeTaskSplitAdapter or similar things, and they’re hard to get right.

  1   2   3   >