Re: Scheduling Cache Refresh

[nithin91]

Thanks for the inputs.It is really helpful.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[Ilya Kasnacheev]

Hello!

You will need to write your own code for that and expose it via compute
tasks or service. Then you can invoke these via REST.

Regards,
-- 
Ilya Kasnacheev


вт, 19 мая 2020 г. в 20:52, nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com>:

> Hi
>
> Thanks For the inputs.
>
> To the question is this way of refreshing the cache is efficient, i mean
> refreshing the cache using the following process.
>
> We have created a REST-API using Spring Boot which refreshes a particular
> cache when a GET Request is triggered.
>
> *Sample REST-API Url :*
>
> http://domainname.com/api/v1/refresh?cachename="MyCache;.
>
> To refresh the cache, in the Spring Boot Application, we have started a
> client node which stays connected to the cluster as long as the Application
> forever.
>
> Also can you please let me know whether DataSteamer API provided by Ignite
> can be accessed through any of the thin clients like JAVA,Python,JS.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Can anyone please help me with your inputs



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Thanks For the inputs.

To the question is this way of refreshing the cache is efficient, i mean 
refreshing the cache using the following process.

We have created a REST-API using Spring Boot which refreshes a particular
cache when a GET Request is triggered.

*Sample REST-API Url :*

http://domainname.com/api/v1/refresh?cachename="MyCache;.

To refresh the cache, in the Spring Boot Application, we have started a
client node which stays connected to the cluster as long as the Application
forever.

Also can you please let me know whether DataSteamer API provided by Ignite
can be accessed through any of the thin clients like JAVA,Python,JS.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[Ilya Kasnacheev]

Hello!

See below inline:

пт, 15 мая 2020 г. в 23:59, nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com>:

> Hi
>
> Although this seems to be working.
> I have following queries with this approach.Can you please
> provide your inputs/suggestions.
>
>
>
> We are going to expose this SpringBoot
> App as a RestFul API where we are trying to
> refresh the Ignite Cache by sending  request to
> this
> Application whenever the data in Oracle changes.
> .
> As we are trying to expose this APP as a API,
> which means that this  Ignite Client Node
> instance launched from Spring boot Application
> will
> be active for ever.Now my queries are
>
> 1. Is it ok to have a client node which
> is continously in connected state forever.
> If Ok,then how many such client node can exist.
>
Yes, it's OK, you can have dozens of clients connected at all times.

2. Can this Application, be hosted on the
> one of the server node with different port.
>
Yes. No additional configuration is necessary.


> 3. Also will client node connect automatically
> to the cluster if the client node has  lost
> communication with server because if the client
> node is down and if there is an API request
> , then the application will throw an error.
>
Client will try to reconnect automatically. It will throw ClientNode
IgniteClientDisconnectedException in the meantime.


> 4. Is this way of refreshing the cache is
> efficient.
> If no, can you please suggest a better way.
>
What do you mean by refreshing cache?


> 5. Also take provide an example on the possibility
> of  having the REST API calling a
> compute job which would
> execute in a separate server
> https://apacheignite.readme.io/docs/compute-grid
> Also i have few more queries related to Ignite
> Behaviour.It would be really helpful if you can
> provide some inputs.
>
> 1. Currently we are using
> ignite.cache("CacheName").loadCache(null) method
> to load  the cache initially and also increamentally as
> this method loads data very fast almost 0.1 million
> in 1 minute. For increamental load also  we are using this
> method
> by passing an additional argument i.e. Custom
> Sql  to load/update only the records that have
> created/changed.
> but what is happening is if the key doesn't
> exist it is inserting the data  but if the key is present
> it is not updating its value.
> Why is the value not updated if the key is present.
>
This is by design.


>
> 2. To overcome this, i have to build my own custom
> cache store as mentioned in the following link
> https://apacheignite.readme.io/v1.9/docs/data-loading.
> But the problem with this approach is  i need to use JDBC
> result set to load the data which is very slow as
> we have to insert row by row.
> Is there a better way to update the keys.
>
Consider using DataStreamer.


>
>
> 3. How to load data faster with custom cache store
> as loading the data which is very slow as
> we have to insert row by row.
>
See above.

Regards,
-- 
Ilya.


>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Can anyone help me by providing inputs to the questions posted in my
previous message.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Issue got resolved after doing the following changes.
Attached you the updated code,client.xml(will be in src/resources folder)
and pom.xml file.
srcandpomupdated.zip

  

1. Removed the following dependency from pom.xml

org.apache.ignite

ignite-spring-boot-autoconfigure-ext
1.0.0

2.  Modifying the IgniteConfig.class such that  
changing the method igniteInstance to return Ignite 
instance by calling 
Ignition.start("Ignite-Client.xml").
Eariler it used to return IgniteConfiguration
by using the following piece of code.Also earlier 
i haven't used IgniteClient.xml.

IgniteConfiguration cfg = new IgniteConfiguration();

// The node will be started as a client node.
cfg.setClientMode(true);

// Classes of custom Java logic will be transferred over the 
wire
from this app.
cfg.setPeerClassLoadingEnabled(true);

// Setting up an IP Finder to ensure the client can locate the
servers.
TcpDiscoveryMulticastIpFinder ipFinder = new
TcpDiscoveryMulticastIpFinder();
ipFinder.setAddresses(Collections.singletonList("Ipaddress"));
cfg.setDiscoverySpi(new 
TcpDiscoverySpi().setIpFinder(ipFinder));
// Starting the node
  
cfg.setSqlSchemas("PIE");
retrun cfg


Although this seems to be working.
I have following queries with this approach.Can you please
provide your inputs/suggestions.



We are going to expose this SpringBoot
App as a RestFul API where we are trying to
refresh the Ignite Cache by sending  request to this
Application whenever the data in Oracle changes.
.
As we are trying to expose this APP as a API,
which means that this  Ignite Client Node 
instance launched from Spring boot Application will 
be active for ever.Now my queries are

1. Is it ok to have a client node which 
is continously in connected state forever.
If Ok,then how many such client node can exist.
2. Can this Application, be hosted on the   
one of the server node with different port.
3. Also will client node connect automatically 
to the cluster if the client node has  lost 
communication with server because if the client 
node is down and if there is an API request
, then the application will throw an error.
4. Is this way of refreshing the cache is efficient.
If no, can you please suggest a better way.
5. Also take provide an example on the possibility
of  having the REST API calling a
compute job which would
execute in a separate server
https://apacheignite.readme.io/docs/compute-grid
Also i have few more queries related to Ignite 
Behaviour.It would be really helpful if you can 
provide some inputs.

1. Currently we are using 
ignite.cache("CacheName").loadCache(null) method
to load  the cache initially and also increamentally as 
this method loads data very fast almost 0.1 million
in 1 minute. For increamental load also  we are using this 
method 
by passing an additional argument i.e. Custom 
Sql  to load/update only the records that have created/changed. 
but what is happening is if the key doesn't 
exist it is inserting the data  but if the key is present
it is not updating its value.
Why is the value not updated if the key is present.

2. To overcome this, i have to build my own custom 
cache store as mentioned in the following link
https://apacheignite.readme.io/v1.9/docs/data-loading.
But the problem with this approach is  i need to use JDBC
result set to load the data which is very slow as 
we have to insert row by row.
Is there a better way to update the keys.


3. How to load data faster with custom cache store
as loading the data which is very slow as 
  

Re: Scheduling Cache Refresh

[akorensh]

org.apache.ignite.IgniteException: Spring application context resource is not
injected.
at
org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.create(CacheJdbcPojoStoreFactory.java:171)

Take a look at the source: 
https://github.com/apache/ignite/blob/feace547f83181853ca6db8626ab9ac6d372eb83/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStoreFactory.java#L171


This means the app was started w/out an application context.

If you used Ignition.start() then this stuation might occur.

Are you using @SpringApplicationContextResource or any such resource.

I noticed that the app you sent over did not use the xml config.

Can you send over the piece of code that starts the app(s).

Can you try w/out spring boot to see that it works. 
Use Ignition.start(xml config) --- don't inject via annotation

Then add spring boot.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[nithin91]

Please find below the log info 


  .     ___ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  || .__|_| |_|_| |_\__, | / / / /
 =|_|==|___/=/_/_/_/
 :: Spring Boot ::(v2.2.5.RELEASE)

2020-05-15 10:35:32.005  INFO 14684 --- [   main]
org.example.springbootapi.App: Starting App on HYDNBK155967 with
PID 14684 (C:\Users\ngovind\eclipse-workspace\springbootapi\target\classes
started by ngovind in C:\Users\ngovind\eclipse-workspace\springbootapi)
2020-05-15 10:35:32.008  INFO 14684 --- [   main]
org.example.springbootapi.App: No active profile set, falling
back to default profiles: default
2020-05-15 10:35:32.564  INFO 14684 --- [   main]
.s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA
repositories in DEFAULT mode.
2020-05-15 10:35:32.586  INFO 14684 --- [   main]
.s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository
scanning in 13ms. Found 0 JPA repository interfaces.
2020-05-15 10:35:33.450  INFO 14684 --- [   main]
o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s):
3000 (http)
2020-05-15 10:35:33.458  INFO 14684 --- [   main]
o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-05-15 10:35:33.459  INFO 14684 --- [   main]
org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache
Tomcat/9.0.31]
2020-05-15 10:35:33.696  INFO 14684 --- [   main]
org.apache.jasper.servlet.TldScanner : At least one JAR was scanned for
TLDs yet contained no TLDs. Enable debug logging for this logger for a
complete list of JARs that were scanned but no TLDs were found in them.
Skipping unneeded JARs during scanning can improve startup time and JSP
compilation time.
2020-05-15 10:35:33.699  INFO 14684 --- [   main]
o.a.c.c.C.[Tomcat].[localhost].[/]   : Initializing Spring embedded
WebApplicationContext
2020-05-15 10:35:33.699  INFO 14684 --- [   main]
o.s.web.context.ContextLoader: Root WebApplicationContext:
initialization completed in 1645 ms
2020-05-15 10:35:33.955  INFO 14684 --- [   main]
com.zaxxer.hikari.HikariDataSource   : HikariPool-1 - Starting...
2020-05-15 10:35:34.073  INFO 14684 --- [   main]
com.zaxxer.hikari.HikariDataSource   : HikariPool-1 - Start completed.
2020-05-15 10:35:34.120  INFO 14684 --- [   main]
o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing
PersistenceUnitInfo [name: default]
2020-05-15 10:35:34.184  INFO 14684 --- [   main]
org.hibernate.Version: HHH000412: Hibernate ORM core
version 5.4.12.Final
2020-05-15 10:35:34.305  INFO 14684 --- [   main]
o.hibernate.annotations.common.Version   : HCANN01: Hibernate Commons
Annotations {5.1.0.Final}
2020-05-15 10:35:34.406  INFO 14684 --- [   main]
org.hibernate.dialect.Dialect: HHH000400: Using dialect:
org.hibernate.dialect.H2Dialect
2020-05-15 10:35:34.619  INFO 14684 --- [   main]
o.h.e.t.j.p.i.JtaPlatformInitiator   : HHH000490: Using JtaPlatform
implementation:
[org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2020-05-15 10:35:34.625  INFO 14684 --- [   main]
j.LocalContainerEntityManagerFactoryBean : Initialized JPA
EntityManagerFactory for persistence unit 'default'
2020-05-15 10:35:34.687  WARN 14684 --- [   main]   
  
: Failed to resolve default logging config file:
config/java.util.logging.properties
Console logging handler is not configured.
2020-05-15 10:35:34.690  WARN 14684 --- [   main]
o.apache.ignite.internal.util.typedef.G  : Ignite work directory is not
provided, automatically resolved to:
C:\Users\ngovind\eclipse-workspace\springbootapi\ignite\work
2020-05-15 10:35:34.783  INFO 14684 --- [   main]
org.apache.ignite.internal.IgniteKernal  : 

>>>__    
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7// /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.7.6#20190911-sha1:21f7ca41
>>> 2019 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

2020-05-15 10:35:34.783  INFO 14684 --- [   main]
org.apache.ignite.internal.IgniteKernal  : Config URL: n/a
2020-05-15 10:35:34.798  INFO 14684 --- [   main]
org.apache.ignite.internal.IgniteKernal  : IgniteConfiguration
[igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8,
stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=8,
dataStreamerPoolSize=8, utilityCachePoolSize=8,
utilityCacheKeepAliveTime=6, p2pPoolSize=2, qryPoolSize=8,
igniteHome=null,
igniteWorkDir=C:\Users\ngovind\eclipse-workspace\springbootapi\ignite\work,

Re: Scheduling Cache Refresh

[akorensh]

can you share the server logs.
turn on debug for the appropriate package:
org/apache/ignite/cache/store/jdbc
https://apacheignite.readme.io/docs/logging

  What is the behavior you are getting? No logs related to this at all.

you should be getting the following:
   Start loading entries from db using user queries from arguments.

Started load cache using custom statement [cache=" + U.maskName(cacheName) +
  ", keyType=" + keyType + ", stmt=" +
stmt + ']'

"Started load cache using custom query [cache=" + U.maskName(cacheName) +
  ", keyType=" + keyType + ", query=" + qry
+ ']'


The actual loading occurs here: 
https://github.com/apache/ignite/blob/fb2f608005a29e9a558942dd648bac771bc44830/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheAbstractJdbcStore.java#L680


Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Actually the Bean File which contains  the Cache Store Factory Details  and
Cache Configuration Details 
is deployed in Linux Server on which the Ignite Instance is running.I am try
to connect to the ignite instance running on the Linux Server from my local
machine.

PFA for the bean file that is deployed. Ignite-Server.xml
  





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[akorensh]

Hi,  
   loadCache(..) is designed to load the cache from a 3rd party store.

   you are not setting the cache factory in your app. 

  See: https://apacheignite.readme.io/docs/3rd-party-store#rdbms-integration 

  example:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/store/jdbc/CacheJdbcStoreExample.java

  it has:  
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(CacheJdbcPersonStore.class));
   this tell loadCache(..) where to get the data from.

Thanks, Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Attached link is at the end of post.However i am attaching it again here.
srcandpom.zip
  

Please let me know in case of any difficulties while opening the attachment.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[Ilya Kasnacheev]

Hello!

Unfortunately I don't see any attachment to your message. Can you please
share a link?

Regards,
-- 
Ilya Kasnacheev


чт, 14 мая 2020 г. в 10:09, nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com>:

> Hi
>
> Attached the Sample Spring Boot Application that i am running in my local
> machine to connect to ignite cluster deployed in Unix as a client node and
> refreshes the cache when ever a GET Request is made.
>
> Facing the following error.Can you please help me in resolving the errors
>
> *org.apache.ignite.IgniteException: Spring application context resource is
> not injected.*
>
> I am facing this error while trying to refresh the cache using the
> following
> method
>
> ignite.cache("NumberandDateFormatCache").loadCache(null,"java.lang.Long",
> "select a.*,row_number() over(order by COUNTRY_CODE) AS ID
> FROM
> \"vwNumberAndDateFormat\" a where rownum<=10 ");
>
> but the application gets executed fine when i try to get the list of caches
> using the following method
>
> ignite.cacheNames();
>
> Attached the entire project for your reference.Can you please help me in
> resolving this issue.
>
> Also can you please let me know is this the right way to instantiate ignite
> instance for a building
> Rest-API using Spring Boot.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Scheduling Cache Refresh

[nithin91]

Hi 

Attached the Sample Spring Boot Application that i am running in my local
machine to connect to ignite cluster deployed in Unix as a client node and
refreshes the cache when ever a GET Request is made.

Facing the following error.Can you please help me in resolving the errors

*org.apache.ignite.IgniteException: Spring application context resource is
not injected.*

I am facing this error while trying to refresh the cache using the following
method

ignite.cache("NumberandDateFormatCache").loadCache(null,"java.lang.Long",
"select a.*,row_number() over(order by COUNTRY_CODE) AS ID FROM
\"vwNumberAndDateFormat\" a where rownum<=10 ");

but the application gets executed fine when i try to get the list of caches
using the following method

ignite.cacheNames();

Attached the entire project for your reference.Can you please help me in
resolving this issue.

Also can you please let me know is this the right way to instantiate ignite
instance for a building
Rest-API using Spring Boot.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh

[akorensh]

 *  But this apprach isn't working because JDBC POJO method
   ignite.cache("CacheName").loadCache(null,"java.lang.String","CustomSql")
is not available from Spring
   Boot.*


ignite.cache("cacheName").loadCache is located here:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html

It does not conflict w/SpringBoot and is compatible with running SpringBoot
in the same JVM.

See:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/store/spring/CacheSpringStoreExample.java

and:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/store/jdbc/CacheJdbcStoreExample.java


Send over a reproducer project where it does not work and I'll take a look.

Also take examine the possibility of using having the REST API calling a
compute job which would 
execute in a separate
server:https://apacheignite.readme.io/docs/compute-grid

Thanks, Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[Andrei Aleksandrov]

Hi Nithin,

You face current message because your client lost the connection to the 
server. It tries to get the acknowledge message on some operation (I 
guess it should be some cache operation).


You can see that IgniteClientDisconnectedException was thrown. In this 
case, you can get the reconnect future and wait for the client reconnection:


https://apacheignite.readme.io/docs/clients-vs-servers#reconnecting-a-client

Please add try/catch blocks around your cache operation and add next logic:

|catch (IgniteClientDisconnectedException e) { 
e.reconnectFuture().get(); // Wait for reconnect. // Can proceed and use 
the same IgniteCompute instance. }|



I can't say why your client was disconnected. Highly likely it's because 
of some network issues. You can try to take a look at server logs and 
find there *NODE_LEFT *or *NODE_FAILED *messages.


BR,
Andrei

2/14/2020 8:08 AM, nithin91 пишет:

Hi

I am unable to attach any file as a result of which i pasted the code and
bean file in my previous messages.

Following is error i get.

Feb 13, 2020 11:34:40 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage

  


[sndNodeId=null, id=b9bb52d3071-613fd9b8-0c00-4dde-ba8f-8f5341734a3c,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
 at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
 at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Feb 13, 2020 11:34:47 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to reconnect to cluster (consider increasing 'networkTimeout'
configuration property) [networkTimeout=5000]
[11:34:52] Ignite node stopped OK [uptime=00:00:24.772]
Exception in thread "main" javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Failed to execute
dynamic cache change request, client node disconnected.
 at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
 at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3023)
 at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2992)
 at Load.OrdersLoad.main(OrdersLoad.java:82)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Failed
to execute dynamic cache change request, client node disconnected.
 at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:952)
 at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:948)
 ... 4 more
Caused by: class
org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Failed
to execute dynamic cache change request, client node disconnected.
 at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:1180)
 at
org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3949)
 at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:821)
 at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
 at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
 at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
 at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
 at java.lang.Thread.run(Thread.java:748)




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[Andrei Aleksandrov]

Hi,

Can you please attach the full logs with the mentioned exception? BTW I 
don't see any attaches in the previous message (probably user list can't 
do it).


BR,
Andrei

2/13/2020 3:44 PM, nithin91 пишет:

Attached the bean file used



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[nithin91]

Following is the java code that loads the cache.

package Load;

import java.sql.Types;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory;
import org.apache.ignite.cache.store.jdbc.JdbcType;
import org.apache.ignite.cache.store.jdbc.JdbcTypeField;
import org.apache.ignite.cache.store.jdbc.dialect.OracleDialect;
import org.apache.ignite.configuration.CacheConfiguration;
import ignite.example.IgniteUnixImplementation.OrderDetails;
import ignite.example.IgniteUnixImplementation.OrderKey;

public class OrdersLoad {

private static final class CacheJdbcPojoStoreExampleFactory extends
CacheJdbcPojoStoreFactory {
/**
 * 
 */
private static final long serialVersionUID = 1L;

/** {@inheritDoc} */
@Override public CacheJdbcPojoStore create()
{

setDataSourceBean("dataSource");
return super.create();
}
}


private static CacheConfiguration
cacheConfiguration() {
CacheConfiguration cfg = new
CacheConfiguration<>("OrdersCache");

CacheJdbcPojoStoreExampleFactory storefactory =new
CacheJdbcPojoStoreExampleFactory();

storefactory.setDialect(new OracleDialect());

storefactory.setDataSourceBean("dataSource");

JdbcType jdbcType = new JdbcType();

jdbcType.setCacheName("OrdersCache");
jdbcType.setDatabaseSchema("PDS_CACHE");
jdbcType.setDatabaseTable("ORDERS2");

jdbcType.setKeyType("ignite.example.IgniteUnixImplementation.OrderKey");
jdbcType.setKeyFields(new JdbcTypeField(Types.INTEGER, "ORDERID",
Long.class, "OrderID"),
new JdbcTypeField(Types.INTEGER, "CITYID", Long.class, "CityID")


);

   
jdbcType.setValueType("ignite.example.IgniteUnixImplementation.OrderDetails");
jdbcType.setValueFields(
new JdbcTypeField(Types.VARCHAR, "PRODUCTNAME", String.class,
"Productname"),
new JdbcTypeField(Types.VARCHAR, "CUSTOMERNAME", String.class,
"CustomerName"),
new JdbcTypeField(Types.VARCHAR, "STORENAME", String.class,
"StoreName")
);

storefactory.setTypes(jdbcType);

cfg.setCacheStoreFactory(storefactory);

cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);

cfg.setReadThrough(true);
cfg.setWriteThrough(true);
cfg.setSqlSchema("PIE");

return cfg;
}

public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("Ignite-Client.xml")) {

System.out.println(">>> Loading cache OrderDetails");

IgniteCache cache =
ignite.getOrCreateCache(cacheConfiguration());

cache.clear();

ignite.cache("OrdersCache").loadCache(null);

System.out.println(">>> Loaded cache: OrdersCache
Size="+cache.size());

}
}
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[nithin91]

Attached the bean file used



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[nithin91]

Thanks aealexsandrov. This information is very useful.

Also i have one more query.

Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB to Ignite Cache using Cache JDBC Pojo Store.
   
As a part of this process, bean file is custom configured  to start
ignite
node in unix. Attached the bean file.
This bean file consists of both cache configuration details and
ignite
configuration details.
   
Once the node is running, we are trying to do the following
   
1. Connecting to the ignite node running on unix through eclipse by
creating a replica of
the attached bean file from local system and adding an additional property
in Bean file with
Client Mode = true and
   then loading the cache that are defined in the bean file deployed
in
unix using the
   following method from local system using JAVA
   
ignite.cache("CacheName").loadCache(null);
   
   * We are able to do this successfully.*
   
2.  Connecting to the ignite node running on unix by creating a
replica of
the attached bean file
in local system and adding an additional property in Bean
file with Client
Mode = true
and then trying to create a cache and configure the cache
and then finally
loading
the cache using the attached JAVA Code.
   
   
   * When we are trying this approach, we are getting an error
like dynamic
cache change
is not allowed.Not getting this error when Ignite server
node and client node  is running on local machine.Getting this error when
server node is running in unix and trying to connect to this node from local
system.*
   
It would be really helpful if you can help me in resolving
this issue.
   
If this not the right approach, then
Configuring all the caches in the bean file is the only
available
option?If this is case,
What should be the approach for  building some additional
caches in ignite
and load these Caches using Cache JDBC POJO Store when the node is running.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Scheduling Cache Refresh using Ignite

[Andrei Aleksandrov]

Hi,

Please read my comments:

1)Ignite generally doesn't support changing of the cache configuration 
without re-creation of the the cache. But for SQL caches that were 
created via QueryEntity or CREATE TABLE you can add and remove the 
columns using ALTER TABLE commands:


https://apacheignite-sql.readme.io/docs/alter-table
https://apacheignite.readme.io/docs/cache-queries#query-configuration-using-queryentity
https://apacheignite-sql.readme.io/docs/create-table
2)First of all, you can use the following options:

https://apacheignite.readme.io/docs/3rd-party-store#section-read-through-and-write-through

Read through can load the requested keys from DB
Write through will load all the updates to DB.

In case if you require some cache invalidation or refresh then you can 
create some cron job for it.


3)I guess that loadCache is the only to do it. It will filter the values 
that have already existed in the cache.


https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#loadCache-org.apache.ignite.lang.IgniteBiPredicate-java.lang.Object...-

4)You can use a different subset of integrations that can do distributed 
streaming to Ignite like Spark or Kafka:


https://apacheignite-mix.readme.io/docs/getting-started

BR,
Andrei
2/12/2020 9:11 PM, nithin91 пишет:

Hi

We are doing a  a POC on exploring the Ignite in memory capabilities and
building a rest api on
top of it using node express.


Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB
to Ignite Cache using Cache JDBC Pojo Store.

Can someone help me whether the following scenarios can be handled using
Ignite as i couldn't find this in the official documentation.

1. If we want to add/drop/modify a  column to the cache, can we 
update the
bean file directly
   when the node is running or do we need to stop the node and 
then again
restart.
   It would be really helpful if you can  share sample code or
documentation link.

2. How to refresh the ignite cache automatically or schedule 
the cache
refresh.
   It would be really helpful if you can  share sample code or
documentation link.

3. Is incremental refresh allowed? It would be really helpful 
if you can
share sample code or
   documentation link.


4. Is there any other way to load the caches fast other Cache 
JDBC POJO
Store.
   It would be really helpful if you can  share sample code or
documentation link.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/