Re: Loading cache from Oracle Table

[yossi]

Hi Prasad, 

Any update on this? I have a similar use case when I have to load 10million
of records  from a mysql database table but I keep getting the error 
"(jvm-pause-detector-worker) Possible too long JVM pause" even ignite
persistence is enabled (which should write in disk the data when the data
does not fit in RAM).
 
I followed  https://apacheignite.readme.io/docs/jvm-and-system-tuning
   tips for
tuning GC and I have also increased the heap size.

For testing purposes I am using a single node with the following JVM
settings:

-server -Xms8g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC 

So, I would like to know how you manage to use multiple threads to retrieve
rows by batches and how to use partition aware for loading big tables
because it is not clear at all to me. It would be easier If I see an
example.

Thank you!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading cache from Oracle Table

[Ilya Kasnacheev]

Hello!

20 minutes for 40 mln entries seems in reasonable ballpark to me. What's
the CPU usage on Ignite nodes at that time?

Regards,
-- 
Ilya Kasnacheev


ср, 19 февр. 2020 г. в 19:43, Prasad Bhalerao :

> Hi,
> I am using partition aware loading to load big tables using 4-8 threads on
> each nodes.
> I have around 16 caches. 2 caches have 38-40 million entries. It takes
> around approx 21 minutes to load the caches on 6 nodes.
> I have kept the backup count to 2.
>
> Will share the code tomorrow.
>
> Regards,
> Prasad
>
>
> On Tue 18 Feb, 2020, 6:08 PM nithin91 <
> nithinbharadwaj.govindar...@franklintempleton.com wrote:
>
>> Hi Prasad,
>>
>> Is there a improvement in performance. If so can you please share the
>> sample
>> code as i am also looking for a similar solution.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Loading cache from Oracle Table

[Prasad Bhalerao]

Hi,
I am using partition aware loading to load big tables using 4-8 threads on
each nodes.
I have around 16 caches. 2 caches have 38-40 million entries. It takes
around approx 21 minutes to load the caches on 6 nodes.
I have kept the backup count to 2.

Will share the code tomorrow.

Regards,
Prasad


On Tue 18 Feb, 2020, 6:08 PM nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com wrote:

> Hi Prasad,
>
> Is there a improvement in performance. If so can you please share the
> sample
> code as i am also looking for a similar solution.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Loading cache from Oracle Table

[nithin91]

Hi Prasad,

Is there a improvement in performance. If so can you please share the sample
code as i am also looking for a similar solution.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: CamelCase to lowerCase conversion of column names while loading cache from postgres .

[Harshal Patil]

Any suggestions please let me know

On Wed, Mar 20, 2019, 11:45 AM Harshal Patil 
wrote:

> Hi ,
> I have enabled persistant store as Postgres .
> This is my cacheConfiguration ,
>
> public static CacheConfiguration cacheIgniteTableCache() throws Exception {
>
> CacheConfiguration ccfg = new CacheConfiguration();
>
>
> ccfg.setName("IgniteTableCache");
>
> ccfg.setCacheMode(CacheMode.*PARTITIONED*);
>
> ccfg.setAtomicityMode(CacheAtomicityMode.*ATOMIC*);
>
> ccfg.setCopyOnRead(true);
>
>
> CacheJdbcPojoStoreFactory cacheStoreFactory = new
> CacheJdbcPojoStoreFactory();
>
>
> cacheStoreFactory.setDataSourceFactory(new Factory() {
>
> */** {**@inheritDoc**} **/*
>
> @Override public DataSource create() {
>
> return DataSources.*INSTANCE_dsPostgreSQL_Rein*;
>
> };
>
> });
>
>
> cacheStoreFactory.setDialect(new BasicJdbcDialect());
>
> cacheStoreFactory.setBatchSize(1);
>
>
> cacheStoreFactory.setTypes(*jdbcTypeIgniteTable*(ccfg.getName()));
>
>
> ccfg.setCacheStoreFactory(cacheStoreFactory);
>
>
> ccfg.setReadThrough(true);
>
> ccfg.setWriteThrough(true);
>
>
> ArrayList qryEntities = new ArrayList<>();
>
>
> QueryEntity qryEntity = new QueryEntity();
>
>
>
> qryEntity.setKeyType("com.gmail.patil.j.harshal.model.IgniteTableKey");
>
> qryEntity.setValueType("com.gmail.patil.j.harshal.model.IgniteTable");
>
> qryEntity.setTableName("ignite_table");
>
> qryEntity.setKeyFieldName("idCol");
>
>
> HashSet keyFields = new HashSet<>();
>
>
> keyFields.add("idCol");
>
>
> qryEntity.setKeyFields(keyFields);
>
>
> LinkedHashMap fields = new LinkedHashMap<>();
>
>
> fields.put("nameCol", "java.lang.String");
>
> fields.put("idCol", "com.gmail.patil.j.harshal.model.IgniteTableKey");
>
>
> qryEntity.setFields(fields);
>
>
> HashMap aliases = new HashMap<>();
>
>
> aliases.put("idCol", "idCol");
>
> aliases.put("nameCol", "name_col");
>
>
> qryEntity.setAliases(aliases);
>
> qryEntities.add(qryEntity);
>
>
> ccfg.setQueryEntities(qryEntities);
>
>
> return ccfg;
>
> }
>
>
> But I am getting exception ,
>
> * org.postgresql.util.PSQLException: ERROR: column "idcol" does not exist*
>
>
> how I can solve this ? i can see that preparedStatement don't have quotes
> around column .
>
>  Current Query =  *select idCol from ignite_table *
>
> instead of *select "idCol" from ignite_table*
>
>
>
>
>
>
>

CamelCase to lowerCase conversion of column names while loading cache from postgres .

[Harshal Patil]

Hi ,
I have enabled persistant store as Postgres .
This is my cacheConfiguration ,

public static CacheConfiguration cacheIgniteTableCache() throws Exception {

CacheConfiguration ccfg = new CacheConfiguration();


ccfg.setName("IgniteTableCache");

ccfg.setCacheMode(CacheMode.*PARTITIONED*);

ccfg.setAtomicityMode(CacheAtomicityMode.*ATOMIC*);

ccfg.setCopyOnRead(true);


CacheJdbcPojoStoreFactory cacheStoreFactory = new
CacheJdbcPojoStoreFactory();


cacheStoreFactory.setDataSourceFactory(new Factory() {

*/** {**@inheritDoc**} **/*

@Override public DataSource create() {

return DataSources.*INSTANCE_dsPostgreSQL_Rein*;

};

});


cacheStoreFactory.setDialect(new BasicJdbcDialect());

cacheStoreFactory.setBatchSize(1);


cacheStoreFactory.setTypes(*jdbcTypeIgniteTable*(ccfg.getName()));


ccfg.setCacheStoreFactory(cacheStoreFactory);


ccfg.setReadThrough(true);

ccfg.setWriteThrough(true);


ArrayList qryEntities = new ArrayList<>();


QueryEntity qryEntity = new QueryEntity();


qryEntity.setKeyType("com.gmail.patil.j.harshal.model.IgniteTableKey");

qryEntity.setValueType("com.gmail.patil.j.harshal.model.IgniteTable");

qryEntity.setTableName("ignite_table");

qryEntity.setKeyFieldName("idCol");


HashSet keyFields = new HashSet<>();


keyFields.add("idCol");


qryEntity.setKeyFields(keyFields);


LinkedHashMap fields = new LinkedHashMap<>();


fields.put("nameCol", "java.lang.String");

fields.put("idCol", "com.gmail.patil.j.harshal.model.IgniteTableKey");


qryEntity.setFields(fields);


HashMap aliases = new HashMap<>();


aliases.put("idCol", "idCol");

aliases.put("nameCol", "name_col");


qryEntity.setAliases(aliases);

qryEntities.add(qryEntity);


ccfg.setQueryEntities(qryEntities);


return ccfg;

}


But I am getting exception ,

* org.postgresql.util.PSQLException: ERROR: column "idcol" does not exist*


how I can solve this ? i can see that preparedStatement don't have quotes
around column .

 Current Query =  *select idCol from ignite_table *

instead of *select "idCol" from ignite_table*

Re: Loading Cache

[Skollur]

Tried below code, but code is not complaining in IDE..Keep seeing error
message asking to change the method loadcache() to method localloadCache().
When changed to method localloadcache(), asking to change method to
loadCache().


ignite.cache("CustomerCache").loadCache(new
IgniteBiPredicate(){
@Override
public boolean apply(Object key, Customer 
customer) {
return 
customer.field("dwTimestamp").after(fromDate) &&
customer.field("dwTimestamp").after(toDate);
}}, null);



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading Cache

[Skollur]

Thank you for reply. Is there example can you provide to load the data with
criteria i.e load data (select query) is based on date range is restricted.

i.e cache.loadcache(select * from table name where date = '02/02/108').



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading Cache

[akurbanov]

Hi,

Call a method like this and pass closure that returns true for data entries
that match your criteria in apply() method:

cache.loadCache(new P2() {
@Override public boolean apply(Integer key, String val) {
// Accept only even numbers.
return key % 2 == 0;
}
}, cnt);

Variable cnt is an number you may use to limit entry count, you may omit it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Loading Cache

[Skollur]

Currently I am loading data with one line in java. Below ocmmand will load
all data from database to Cache.
Example: ignite.cache("CustomerCache").loadCache(null);

I would like to load data for specific period. How would I do that?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading cache from Oracle Table

[Prasad Bhalerao]

Hi Vinokurov,

I was thinking to use Java ExecutoreService to submit the task (task to
invoke CacheStore.loadCache method). Is it ok to use ExecutorService or
ignite provides some way to submit the task?

Thanks,
Prasad

On Fri, Feb 9, 2018 at 6:04 PM, Vinokurov Pavel 
wrote:

> Hi Prasad,
>
> Within your implementation of CacheStore.loadCacheYou you could use
> multiple threads to retrieve rows by batches.
> Note that each thread should use different jdbc connection.
>
> 2018-02-09 13:57 GMT+03:00 Prasad Bhalerao :
>
>> Hi,
>>
>> I have multiple oracle tables with more 50 million rows. I want to load
>> those table in cache.To load the cache I am using CacheStore.loadCache
>> method.
>>
>> Is there anyway where I can load a single table in multithreaded way to
>> improve the loading performance?
>> What I actually want to do is,
>> 1) Get all the distinct keys from the table.
>> 2) Divide the list of key in batches.
>> 3) Give each batch of keys to a separate thread which will fetch the data
>> from the same table in parallel mode.
>> e.g. Thread T1 will fetch the data for key 1 to 100 and thread T2 2ill
>> fetch the data for keys 101 to 200 and so on.
>>
>> Does ignite provide any mechanism to do this?
>>
>> Note: I do not have partitionId in my table.
>>
>>
>> Thanks,
>> Prasad
>>
>
>
>
> --
>
> Regards
>
> Pavel Vinokurov
>

Re: Loading cache from Oracle Table

[Vinokurov Pavel]

Hi Prasad,

Within your implementation of CacheStore.loadCacheYou you could use
multiple threads to retrieve rows by batches.
Note that each thread should use different jdbc connection.

2018-02-09 13:57 GMT+03:00 Prasad Bhalerao :

> Hi,
>
> I have multiple oracle tables with more 50 million rows. I want to load
> those table in cache.To load the cache I am using CacheStore.loadCache
> method.
>
> Is there anyway where I can load a single table in multithreaded way to
> improve the loading performance?
> What I actually want to do is,
> 1) Get all the distinct keys from the table.
> 2) Divide the list of key in batches.
> 3) Give each batch of keys to a separate thread which will fetch the data
> from the same table in parallel mode.
> e.g. Thread T1 will fetch the data for key 1 to 100 and thread T2 2ill
> fetch the data for keys 101 to 200 and so on.
>
> Does ignite provide any mechanism to do this?
>
> Note: I do not have partitionId in my table.
>
>
> Thanks,
> Prasad
>



-- 

Regards

Pavel Vinokurov

Loading cache from Oracle Table

[Prasad Bhalerao]

Hi,

I have multiple oracle tables with more 50 million rows. I want to load
those table in cache.To load the cache I am using CacheStore.loadCache
method.

Is there anyway where I can load a single table in multithreaded way to
improve the loading performance?
What I actually want to do is,
1) Get all the distinct keys from the table.
2) Divide the list of key in batches.
3) Give each batch of keys to a separate thread which will fetch the data
from the same table in parallel mode.
e.g. Thread T1 will fetch the data for key 1 to 100 and thread T2 2ill
fetch the data for keys 101 to 200 and so on.

Does ignite provide any mechanism to do this?

Note: I do not have partitionId in my table.


Thanks,
Prasad

Re: Loading cache with DataStreamer

[pragmaticbigdata]

Got it. Thanks for detailing it out.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5429.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Loading cache with DataStreamer

[Alexey Goncharuk]

Note that IgniteDataStreamer implements AutoCloseable, so the code in the
example you are referring to is correct because data streamer is used in
try-with-resources block. It is not required to call flush() before calling
close() because close() will flush the data automatically.​

Re: Loading cache with DataStreamer

[pragmaticbigdata]

Thanks for the replies. 

Calling flush() and later close() made sure all the entries were added to
the cache. I think this step should be added to the  examples
<https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/stream/DataStreamerExample.java>
  
since without this the code cannot assume when would the cache be completely
populated with all the entries.

AutoFlushFrequency and/or the buffer size seem to be performance tuning
parameters.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5426.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Loading cache with DataStreamer

[Alexey Goncharuk]

Hi Amit,

You can also close() the streamer or call flush() explicitly to make sure
all the added data was added to the cache.

2016-06-04 10:57 GMT-07:00 visagan <visagan1...@gmail.com>:

> The Streamer actually buffers the data. Buffer Default Size is 1024, either
> the buffer size is reached Or you set a Flush Frequency for the buffer, it
> does not deliver the data to the nodes.
> And it does a parallel load with multiple threads by reading the data that
> has been buffered.
>
> Try Setting this property to something like 10Seconds and see if all the
> data you add to the streamer are flushed. By default Auto flush frequency
> is
> disabled.
> autoFlushFrequency(1)
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5424.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Loading cache with DataStreamer

[visagan]

The Streamer actually buffers the data. Buffer Default Size is 1024, either
the buffer size is reached Or you set a Flush Frequency for the buffer, it
does not deliver the data to the nodes. 
And it does a parallel load with multiple threads by reading the data that
has been buffered. 

Try Setting this property to something like 10Seconds and see if all the
data you add to the streamer are flushed. By default Auto flush frequency is
disabled.  
autoFlushFrequency(1)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5424.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Loading cache with DataStreamer

[pragmaticbigdata]

I tried that but instead of loading 10 entries it loaded only 99330.
Doesn't datastreamer do a parallel load through multiple threads and hence
return a future object that we need to wait on?

Thanks!




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5423.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Loading cache with DataStreamer

[visagan]

 You should probably do this:  
   IgniteDataStreamer<String, ProductDetails> streamer =
ignite.dataStreamer(cacheName); 
   for (long i=1; i<= 10; i++) { 
ProductDetails phone = new ProductDetails("phone" + i); 
//set other properties 
streamer.addData(phone.getName(), phone); 
} 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421p5422.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Loading cache with DataStreamer

[pragmaticbigdata]

I am using apache ignite version 1.6. Assuming datastreamer is a better
approach to bulk load data performance wise I am trying to execute the below
code that gets hanged

IgniteDataStreamer<String, ProductDetails> streamer =
ignite.dataStreamer(cacheName);

List futures = Lists.newArrayList();

for (long i=1; i<= 10; i++) {
ProductDetails phone = new ProductDetails("phone" + i);
//set other properties 
 
futures.add(streamer.addData(phone.getName(), phone));

}

//wait for future objects
for (IgniteFuture f : futures) {
f.get();
}

What am I missing in the above example?

Thanks,
Amit.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Loading-cache-with-DataStreamer-tp5421.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.