Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Unfortunately, db work is out of the scope of my responsibility, and I was just told to work around the issue Hence me asking about Colocation and MapReduce -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

> Colocation by Partition, but that API is not available for C# Partition-based APIs were added in Ignite 2.9 [1] > Is it possible to speed up loading cache by using Binary Object + Colocating by Key, or Binary Object + MapReduce API? Probably yes. But have you sorted out the DB connection bottlen

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Thank you for the clarification! For this use case where data is being pulled from an underlying RDBMS and instantiated as an Associate object, is it possible to work the Associate object as a Binary Object similar to what you've done here? Also, I see than in Vladamir's loading example, he uses t

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Vladimir, > I do not use SQL because my interest is to get max performance, I use key-value API instead > The same reason is why I use Java I don't think we should downplay SQL and C# APIs like this - it can be misleading to our users. SQL can be faster than key-val API (e.g. index lookups), and

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

> are there C# equivalencies to the Java interfaces BinaryObject and Affinity Yes, please see IIgnite.GetAffinity [1] and GetBinary [2]. [1] https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.IIgnite.html#Apache_Ignite_Core_IIgnite_GetAffinity_System_String_ [2] https://ig

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hi,I do not use SQL because my interest is to get max performance, I use key-value API instead. The same reason is why I use Java - it is native for Ignite. I think there should be wrappers forBinaryObject and Affinity in c#.Vladimir 4:39, 3 декабря 2020 г., adumalagan :I see, thanks for the clarif

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I see, thanks for the clarification! I also have another question - are there C# equivalencies to the Java interfaces BinaryObject and Affinity? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Yes, once the data is loaded into Ignite you can query it with SQL. - Denis On Wed, Dec 2, 2020 at 9:41 AM adumalagan wrote: > Hi Vlad, I've started to read through your link and I just wanted to know > that, if I were to implement your tutorial, could I perform SQL queries on > the cache afte

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hi Vlad, I've started to read through your link and I just wanted to know that, if I were to implement your tutorial, could I perform SQL queries on the cache after loading? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hi,may be my recent tutorial [1] will shed some light to the question. The story is about my experience in loading big tables to Ignite. Hope it will help despite the Java language insideRegards,Vladimir[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Run your application in the same data center as the database, so that network costs are minimized. On Tue, Nov 24, 2020 at 8:54 PM ABDumalagan wrote: > I see - do you have any suggestions of how I can work around the bottleneck > and speed up data loading into cache? > > > > -- > Sent from: http

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I see - do you have any suggestions of how I can work around the bottleneck and speed up data loading into cache? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

The code looks ok to me. Since the query is so simple, most likely your connection to the DB is the bottleneck. On Tue, Nov 24, 2020 at 8:30 PM ABDumalagan wrote: > So I am currently running this > > code to just pull al

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

So I am currently running this code to just pull all the rows from the underlying database, and the diagnostic session is hitting 20 mins and counting - Is this definitively a database issue or is it in the way I've wri

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

> how would I check to see if the underlying database is a bottleneck Check how much time it takes to read 500 000 rows from there, without loading into Ignite. Or run a profiler. On Tue, Nov 24, 2020 at 6:25 PM ABDumalagan wrote: > I'll try a local scenario and stream with fake data right now,

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I'll try a local scenario and stream with fake data right now, but how would I check to see if the underlying database is a bottleneck? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1) Yes 2) On my local machine with 2 nodes I can stream 500 000 entries in *2* seconds (I tried to recreate the Associate class with some fake data). Are you sure that Oracle is not a bottleneck? Can you try a local scenario with fake data - how much time does it take? On Tue, Nov 24, 2020 at 5:46

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I see - I'm actually going to scrap the LoadCache approach and implement your suggestion of DataStream - I just have a couple of questions regarding that: 1) Is it possible to use both DataStream and CacheStoreFactory together? 2) Right now, my Program.cs

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

NullPointerException usually indicates a bug. Please provide full exception stack trace. On Tue, Nov 24, 2020 at 4:03 PM ABDumalagan wrote: > Also, I tried changing the code from cache.LocalLoadCache(null) to > cache.LoadCache(null) and I received a NullPointerException. > > Am I supposed to pas

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Also, I tried changing the code from cache.LocalLoadCache(null) to cache.LoadCache(null) and I received a NullPointerException. Am I supposed to pass a non-null parameter into LoadCache? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

DataStreamer is the best approach for loading large amounts of data into Ignite On Tue, Nov 24, 2020 at 3:50 PM ABDumalagan wrote: > I see, that makes sense - I actually have a question regarding that: > > If I wanted to store a whole table from an underlying database as > key-value pairs in cac

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I see, that makes sense - I actually have a question regarding that: If I wanted to store a whole table from an underlying database as key-value pairs in cache, but the database had over 500,000 entries, how would you suggest going about that? -- Sent from: http://apache-ignite-users.70518.x6.

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Ok, here is the problem, per documentation [1]: "In case of partitioned caches, keys that are not mapped to this node, either as primary or backups, will be automatically discarded by the cache." Since you have two nodes in the cluster, but call localLoadCache only on one node, part of the cache e

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1. I currently just have 2 Ignite nodes--first one remotely to start the cluster and the second one (this one) started programmatically with C#. 2. Adding Thread.Sleep(5000) doesn't change the result, unfortunately. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1. How many Ignite nodes do you have? 2. What if you add Thread.Sleep(5000) before the last Console.WriteLine? Does the resulting number change? On Mon, Nov 23, 2020 at 6:01 PM ABDumalagan wrote: > 1. Your program worked for me! > > 2. I added something to my LoadCache(Action, params > object[]

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

1. Your program worked for me! 2. I added something to my LoadCache(Action, params object[] args) method in OracleStore.cs. I added the following 3 lines after the while loop: reader.Dispose(); cmd.Dispose(); con.Dispose(); Console returned a non-zero cache size of 5136, however, the queries I wa

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Your code seems to be correct. It works for me in a simplified form: https://gist.github.com/ptupitsyn/a64c899b32b73ab55cb706cd4a09e6e9 1. Can you try the program above - does it work for you? 2. Can you confirm that the Oracle query returns a non-empty result set? On Mon, Nov 23, 2020 at 3:00 A

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

I see - am I dealing with 1.) in my case? When I hover over the method LoadCache(Action, params object[] args) in Visual Studio, it says that *"...This method is called whenever Apache.Ignite.Core.Cache.ICache.LocalLoadCache(Apache.Ignite.Core.Cache.ICacheEntryFilter, params object[]) method is in

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Looks like we mixed up two different delegates: 1. ICache.LoadCache method takes an *optional *filter delegate in case you want to skip some of the entries from cache store. You can pass null there when filtering is not required. 2. ICacheStore.LoadCache method gives you (an implementor of the int

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Got *(1)* to work - thank you! However, for *(2)*, I tried to implement a test Action delegate that writes the value of the entry in the console and I got an error stating: Argument 1: cannot convert from 'System.Action' to 'Apache.Ignite.Core.Cache.ICacheEntryFilter' This error occurred at line c

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hello, *> (1)* Regarding the Load method, I was under the impression that calling Get(key) would inherently call the Load(key) method from my OracleStore.cs It would if you set CacheConfiguration.ReadThrough = true. Similarly, enable WriteThrough if you want Ignite to call OracleStore.Write on ca

C# CacheStoreAdapter - Customizing Load, LoadCache methods

Hello, I have some confusion about implementing my own Load and LoadCache methods - I am trying to load data from an underlying Oracle database into cache, and the entries are key-value pairs where the key is the Associate ID and the value is the entire Associate object. *(1)* Regarding the Load