Re: Renaming a cache

Thanks for the answer. It is then much easier to add a level of indirection and to have a random name for the cache. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Renaming-a-cache-tp10943p10987.html Sent from the Apache Ignite Users mailing list archive at

Affinity based on the fields NOT in the key

Hello, first let me apologize if there is any double submit. I think that my first post on this subject was not visible because I did not do the subscription completely. My understanding is that affinities can only be based on fields in the key. Is this correct? This seems counter-intuitive

Re: Affinity based on the fields NOT in the key

Hello Alexey, yes it does thanks. No that I know that it is the way of doing it, it's fine. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-based-on-the-fields-NOT-in-the-key-tp9457p9460.html Sent from the Apache Ignite Users mailing list archive at

What are typical load time for millions of records

I'm currently trying to improve the load time. I would like to have some feedbacks from other Ignite users as for the actual load performances. In my setup, I have a DB in the same subnet with an Ignite node with 8 cores. It loads 20,000,000 records in 1h with the standard loadCache

Re: LoadCache Performance decreases with the size of the cache

Hi Val and thanks for the reply. I narrowed it down a little bit. The problem comes from the indexing. I tried with no indexing/query fields and then half the fields queryable and it turns out that with indexing the loading performances decrease over time. First, I would like to know whether it

Re: LoadCache Performance decreases with the size of the cache

Hello guys, I finally understood what the "problem" is. With 20 millions of records, the indexing time is very expensive. Is there a way, like for a database, to disable indexing during load time and to re-enable it after loading? I guess that rebalancing the b-trees (or whatever is used to

Re: LoadCache Performance decreases with the size of the cache

Hello Val, you're right I was too quick to jump to a conclusion. Actually, the problem comes from my code but it was not obvious to me. I create the indexes with the following code: - Collection idxs = new ArrayList<>(); QueryIndex

Re: LoadCache Performance decreases with the size of the cache

Sure here is the code QueryEntity qryEntity = new QueryEntity(); qryEntity.setKeyType("myapp.bpepoc.model.MyModelKey"); qryEntity.setValueType("myapp.bpepoc.model.MyModel"); LinkedHashMap fields = new LinkedHashMap<>();

Re: LoadCache Performance decreases with the size of the cache

Hello Val, Yes I'll try to upload something on github tomorrow. Thanks for the help -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/LoadCache-Performance-decreases-with-the-size-of-the-cache-tp9645p9710.html Sent from the Apache Ignite Users mailing list

LoadCache Performance decrease with the size of the cache

Hello, I am trying to increase the performance of the cache loading. I am witnessing a strange behavior: as the number of objects increase in the cache the number of objects loaded per seconds decrease. The database server seems not to be the problem. To get some numbers I copy pasted the

Re: LoadCache Performance decreases with the size of the cache

Hello, while investigating, I understood why I do have a lot of locks on Lucene Documents in java mission control. That is because as soon as there is String in the index, this is handled by Lucene even if you do not want full text search (The string being a identifier). --- From

Re: LoadCache Performance decreases with the size of the cache

Hi Val, first of all, let me thank you for this great product and incredible mailing list. Anonymizing our code will take a bit longer but I can give the following numbers: load time 10M records without indexes 10m2s load time 10M records with a simple row id type index : 12m load time 10M

Re: What are typical load time for millions of records

Hi, I am using the partition capability with multiple threads to improve the data loading, but referring to my other post : I see that the performance are not constant with respect to the size of the cache. Regards -- View this message in context:

Re: Identical objects on the same node

Hi Val, I definitly consider this alternative ("normalizing" the objects like with foreign keys) but it requires more work from my side to cope with graphes of objects. In other terms, if I want to see data from the cache as "pure" objects, I do have to implement the "ORM" layer on top of it with

Re: Identical objects on the same node

Hello Val, thanks for the answer. Would it be theoritecally possible from your point of view? I mean having some sort of serialized pointers to other byte arrays on the same node? What is the rational of serializing everything on the node? I get that internode communication requires

Identical objects on the same node

Hello, I would to understand Ignite's behavior when I put identical objects (from strings up to graphs of objects) in the same cache (as part of the graph of different values) on the same node. In other words, is there some sort of flyweight pattern implemented to reduce memory consumption? Many

RE: Write behind and eventual consistency

Hi Val, the use case is the following 1) Load data into the database from an external system 2) Once ready load it into the grid 3) Process something that does massive write behinds 4) Take a snapshot of the results (or) Do a backup of the tables <<--- At this point I need the eventual

Write behind and eventual consistency

Hello, Iwould like to enable the write-behind mode but since Ihave to sync the overall process with other jobs (extract to the datawarehouse), I would like to know how to make sure that there are no more write behind operations waiting for completion. I had a look at the API but I do not see

Apache Ignite, Memory Class Storage, GPUs, and all that

Hello, Having to explain the choice of Ignite internally, I wonder what is the "official" position of Apache Ignite towards Storage Class Memory and using GPUs. On the SCM story, I guess it is just another way of allocating/freeing memory in a kind of off-heap mode but on disk. On the GPUs

Re: copyOnRead to false

Hello, thanks for the answer. The benchmark is actually our application stressed with several volumes. Some quite complex to describe. However, for these benchmarks we are only using one node. Basically we are loading a set of caches from the database, do a lot of querying both ScanQuery (on

copyOnRead to false

Hello, we did some benchmarks and did set the copyOnRead flag to false and it did increase the processing time and the memory a little bit (10%). I cannot figure out why this would be case since my understanding is that if anything it should reduce the processing time by avoiding a clone. One

Re: Ignite and Persistent Collections

Will do thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Simulate Read Only Caches

Hello, I do have a bunch of caches that I would like to have replicated but to keep them "read only" after a certain point. It is a fairly standard use case. There are master (exchange rates) that are fixed once and for all for a given (set of processes). Once loaded there is no reason to bother

BinaryMarshaller (micro) Benchmark

Hello, I am puzzled by a micro-benchmark I made and that you can find here: https://github.com/hostettler/IgnitePerfTest.git To reproduce the below results, just run: $ mvn clean install $ cd target $ java -Xmx512m -Xmx512m -XX:+UseG1GC -jar

Ignite and Persistent Collections

Hello, I would like to know what is the position of the committers towards Persistent Collections (https://pcollections.org/)? Apparently it could massively reduce the marshalling/unmarshalling time. Best Regards -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: BinaryMarshaller (micro) Benchmark shows that keepBinary is a bit slower than systematically unmarshall the object

Hello all, I actually found what the problem was. I accessed the fields of the BinaryObject in that way: IgniteCache cacheAsBinary = cache.withNoRetries().withSkipStore().withKeepBinary(); ///Do a million times BinaryObject bo = cacheAsBinary .get(key); Long field1 = o.field("field1"); but I

Re: Simulate Read Only Caches

Precisely my intention :) Will switch to the developer list to ask for some guidance. Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Business Intelligence connection to Ignite

Hello, is there any best practice/recommendation on how to connect 3rd business intelligente tools to Ignite. For instance, is it possible to connect a BO universe to ignite? Thansk for your help -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Affinity function and partition aware database loading

Hello, I would like to enable partition aware data loading. I do have a composite business key in the database (Oracle and SQL Server) that happen to be the key of the object in the key. The most important part of that key is a string I can very easily compute a good affinity from that key, the

Re: K8S on Azure

Hi all, sorry for the confusion, I was using the wrong ip finder (using a wrong docker image of mine) on some of the nodes. It worked with K8S on docker-desktop but not with K8S on Azure because of the way networking is managed. Apologies -- Sent from:

K8S on Azure

Hello, I am trying to deploy an ignite based application on an Azure K8S using helm charts. I test my application locally on a docker-desktop based K8S with obviously only one node (my laptop) That works quite well and when I increase the replicas, the new pods join the cluster and it works. The

Re: K8S on Azure

I continued my investigations and I started with only one pod and then scale up. I get the following message and on the new pod that proves that the two started to communicate but it did not result in the new pod to join the cluster as the topology did not change. 2019-12-14

REPLICATED caches network overhead?

Hello, first let me wish you a happy new Year! I am currently working with replicated and local caches and I observe a strange behavior in my application. I do not know whether it is related to my application or to ignite at this point and thus looking for information. I have a REPLICATED cache

Re: REPLICATED caches network overhead?

So in the mean time, I was able to confirm that there are messages sent to another node about queries on replicated caches? I get the following messages sent from node 1509c11f-c627-4696-a508-6b2c6bc99904 to 6b3b721d-611b-488e-bdbf-35a9dc16b8f0 GridH2QueryRequest [reqId=663310,

Re: REPLICATED caches network overhead?

So to close this one, the problem was because I did not set the replicatedOnly flag on the query. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Good metrics to assess affinity quality

Hello Igniters, first let me wish you happy holidays. I am working to testing different affinities and I was wondering what would be good metrics to assess affinity quality. I have a near cache to deal with less than perfect affinities. I assume network traffic is a good one but is there

Re: Good metrics to assess affinity quality

Hello, my objective is to come up with a good affinity to optimise data distribution.To reduce local misses by improving co-location. My data structure is however legacy and quite complex and I do not have a clear candidate. Therefore, I would like to test several affinities and to measure what

Communication between Ignite Clusters

Hello, I would like to know what is the best way to integrate two different Apache Ignite Clusters? I can see the ignite client and the jdbc integration. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Communication between Ignite Clusters

Hello and thank you for your answer. Assuming we would have 2 clusters (e.g., we want to scale them independently, upgrade them at different pace) that are manipulating different datasets. The output of the first clusters being the input of the second one. -- Sent from:

Re: Communication between Ignite Clusters

Thanks a lot! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Communication between Ignite Clusters

Thanks a lot for your answer, So you would use a kafka for instance and then how to push the data? Using a data streamer to balance the data on the right nodes of the second cluster? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Optimise replicated caches in multinodes

Hello, in order to provide "easy" multinode, we have set all of our caches to REPLICATED and FULL_ASYNC. We get an improvment (Around 20%) with a second node, but it is very clear that somehow the grid is waiting (95% cpu utilization with 1 node, 75% with 2 nodes). I understand that I should

Re: Ignite on AKS and RBAC issue

I found my mistake, it was indeed a glitch in the deployment yaml as I forgot to specify the service account. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite on AKS and RBAC issue

Hello, I am deploying an embeded version of ignite on AKS and I am getting this error: Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite at

Re: Ignite on AKS and RBAC issue

Hello Alex, thanks for the tip but putting everything on the namespace ignite does not help. I also rechecked the documentation. I still get the 403. Additional question : how does the service account and the service relate? So I have a 1) service account kubectl describe serviceaccount ignite

Re: Using native persistence to "extend" memory

Thank you both for clarifying this. I usually also have large objects 3KB so I thought about increasing the page size to 32KB to reduce the number of pages and thus reduce the speed at which we get to the 2/3 of dirty pages. Good idea? On top of that during the process at some I generate a

Re: Ignite persistence and activation

thanks a lot -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Using native persistence to "extend" memory

Hello, I am trying to use ignite persistence to "extend" memory. That is to replace the swap that is not working very well. Therefore since I do not care about recovery I disabled the WAL. Are there other things you would recommend to configure to use the ignite persistence as a sort of swap.

Re: Using native persistence to "extend" memory

Thanks a lot for the recommendation. So keeping the WAL, disabling archiving. I understand all records are kept on disk. Thanks again. Anything else? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite persistence and activation

Hello. I am trying to implement ignite persistence but I stumbled upon the following problems/questions. It is required to activate the cluster, that much is clear but I have bootstrap code that is using technical caches that I do not want to persist and more problematic I need to use

Re: EXCEPTION_ACCESS_VIOLATION when Swap is enabled

No I will try that now -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Kubernetes : Connection timed out (Connection timed out)

So it appears that since this morning I am not able to access the cluster from within a pod in Azure. Very very strange / # wget http://kubernetes.default.svc.cluster.local:443/api Connecting to kubernetes.default.svc.cluster.local:443 (10.0.0.1) -- Sent from:

Re: Kubernetes : Connection timed out (Connection timed out)

Hello Jose, thanks for the answer. Are you on azure as well by any chance? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Kubernetes : Connection timed out (Connection timed out)

Running curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept: application/json"

Kubernetes : Connection timed out (Connection timed out)

Hello, Since this morning I get a strange Timeout on the connection to the cluster even with only one node. It used to work just fine until yesterday evening. At that point I destroyed the clustered and recreated it this morning. I do so with a batch so it is supposed to be the same all the

Re: Kubernetes : Connection timed out (Connection timed out)

Hello, making some progress I added the missing hostNetwork: true and now I get Caused by: java.net.UnknownHostException: kubernetes.default.svc.cluster.local -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Kubernetes : Connection timed out (Connection timed out)

Hello Denis, same code, same config, today it works so it was an transient AKS thing. no idea what Best Regards -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Kubernetes : Connection timed out (Connection timed out)

Hello Dennis, so it works again, it was apparently a transient Azure problem when creating new clusters. FYI the documentation and https://github.com/apache/ignite/tree/master/modules/kubernetes/config/az is not up to date and gave errors when executing them (from a K8S perspective) -- Sent

How to evaluate memory consumption of indexes

Hello, after having added a new index, I see a surge of off-heap consumption (+10GB) but since it is a small index, I am surprised by it. Is there a way to have more details about what's consuming off-heap? Thanks in advance -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How to evaluate memory consumption of indexes

Thanks for the answer, did not thing of using the persistence to assess the size :) -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/