Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

1) My testing shows that CassandraCacheStoreFactory is read from xml config
once if the cache name does not exist in Ignite Persistence. It appears that
CassandraCacheStoreFactory bean is persisted along with cache name in Ignite
Persistence Store, CassandraCacheStoreFactory will not be re-instantiated. 
So I need to pre-decide dynamicConfigurationReload value. Modifying
dynamicConfigurationReload value later won't take effect at dynamically.  
















  

2) Yes, the similar implementation is applied to data source.  Thanks for
remaindering.   

3) given dynamicConfigurationReload is only loaded once at very first time
of creating cache, I'm thinking we don't use dynamicConfigurationReload,
instead implement the following
3.1) read data source from the xml config file each time an Ignite server
starts, if data source is not null, use the latest one, else use the
existing one.
3.2) read persistenceSettingsBean from xml config file each time an Ignite
server starts, if persistence settings are not null, use the latest one,
else using the existing one.
3.3) obviously, this is aggressive.  But how many times do we start/restart
an Ignite instance.  The performance impacts are minimal.  

if you think it's okay, I start implementing and testing.  
Let me know if you have any other suggestions/recommendations.

Appreciated your help.
Xinmin




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

Looks overall correct.

Regards,
-- 
Ilya Kasnacheev


чт, 14 янв. 2021 г. в 11:43, xmw45688 :

> 1) My testing shows that CassandraCacheStoreFactory is read from xml config
> once if the cache name does not exist in Ignite Persistence. It appears
> that
> CassandraCacheStoreFactory bean is persisted along with cache name in
> Ignite
> Persistence Store, CassandraCacheStoreFactory will not be re-instantiated.
> So I need to pre-decide dynamicConfigurationReload value. Modifying
> dynamicConfigurationReload value later won't take effect at dynamically.
>
> 
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
>  value ="true" />
>  ref="cassandra"/>
>  value="catalog_uom_persistence_settings">
> 
> 
> 
>
> 2) Yes, the similar implementation is applied to data source.  Thanks for
> remaindering.
>
> 3) given dynamicConfigurationReload is only loaded once at very first time
> of creating cache, I'm thinking we don't use dynamicConfigurationReload,
> instead implement the following
> 3.1) read data source from the xml config file each time an Ignite server
> starts, if data source is not null, use the latest one, else use the
> existing one.
> 3.2) read persistenceSettingsBean from xml config file each time an Ignite
> server starts, if persistence settings are not null, use the latest one,
> else using the existing one.
> 3.3) obviously, this is aggressive.  But how many times do we start/restart
> an Ignite instance.  The performance impacts are minimal.
>
> if you think it's okay, I start implementing and testing.
> Let me know if you have any other suggestions/recommendations.
>
> Appreciated your help.
> Xinmin
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

1) Can you please elaborate? If you set this value in a cache store
factory, it will keep this boolean value and deserialize it when node is
started. I'm not entirely sure if Ignite will store cache store factory or
cache store instance in the persistent cache configuration. My expectation
it's the former, since cassandra data source is initialized dynamically.

2)

<-- the value of this reference is a distinct object on each node, it is
never replicated over the network nor taken from persistent cache
configuration.
I'm not sure if it's relevant to your use case, I'm just reminding you to
take care of its life cycle.

Does this change work for you?

Regards,
-- 
Ilya Kasnacheev


вт, 12 янв. 2021 г. в 06:30, xmw45688 :

> Thanks for your recommendations!
> 1) The dynamicConfigurationReload is not able to reset once
> cacheStoreFactory is cached.  It means that cacheStoreFactory does not read
> this value unless cacheStoreFactory is first instantiated.  Do I miss
> something?  Which factory should I put dynamicConfigurationReload in the
> xml
> configuration file?
>
> 2) can you please elaborate more about your comments "the data source may
> come from local Spring context on each node.".
>
>
> 
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
>
>  value ="true" />
>  ref="cassandra"/>
>  value="catalog_uom_persistence_settings">
> 
> 
>
> CassandraCacheStoreFactory changes
>
> private boolean dynamicConfigurationReload;
>
> /**
>  * @return Persistence settings.
>  */
> private KeyValuePersistenceSettings getPersistenceSettings() {
>
>   if (!isDynamicConfigurationReload() && persistenceSettings
> != null)
>   return persistenceSettings;
>
> if (persistenceSettingsBean == null) {
> throw new IllegalStateException("Either persistence
> settings
> bean or persistence settings itself " +
> "should be specified");
> }
>
> if (appCtx == null) {
> throw new IllegalStateException("Failed to get
> Cassandra
> persistence settings cause Spring application " +
> "context wasn't injected into
> CassandraCacheStoreFactory");
> }
>
> Object obj = loadSpringContextBean(appCtx,
> persistenceSettingsBean);
>
> if (!(obj instanceof KeyValuePersistenceSettings)) {
> throw new IllegalStateException("Incorrect persistence
> settings
> bean '" +
> persistenceSettingsBean + "' specified");
> }
>
> return persistenceSettings =
> (KeyValuePersistenceSettings)obj;
>
>
> }
>
>
> public void setDynamicConfigurationReload(boolean
> dynamicConfigurationReload ) {
> this.dynamicConfigurationReload = dynamicConfigurationReload;
> }
>
> public boolean isDynamicConfigurationReload() {
> return dynamicConfigurationReload;
> }
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Thanks for your recommendations!
1) The dynamicConfigurationReload is not able to reset once
cacheStoreFactory is cached.  It means that cacheStoreFactory does not read
this value unless cacheStoreFactory is first instantiated.  Do I miss
something?  Which factory should I put dynamicConfigurationReload in the xml
configuration file?

2) can you please elaborate more about your comments "the data source may
come from local Spring context on each node.".  











 




http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

This seems like a valid change, however, there are issues with it: if you
don't have persistence settings files on some nodes, it will cause failure
on these nodes. It may also re-read these files too eagerly in its current
form.

I can totally see how this may be an option to have in the Cassandra cache
store, e.g. dynamicConfigurationReload={true/false} Please note that you
also need to factor in Spring when updating data source life cycle = i.e.
the data source may come from local Spring context on each node.

Feel free to file a ticket and contribute.

Regards,
-- 
Ilya Kasnacheev


чт, 7 янв. 2021 г. в 05:48, xmw45688 :

> Thanks for your clarifications, and appreciated your suggestions and
> guidance.
>
> My college and I went to ignite-cassandra module,  commented two lines,
> */testing purpose/*,  in
>
> org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory.getPersistenceSettings(),
> see the changes below.  Removing these two lines is to force Ignite to read
> the configuration settings from the xml at runtime.  It looks the change
> works and meets our requirement.
>
> I'm looking for the suggestions and recommendations from Ignite community
> and you, and wondering if we make these changes in Ignite repository.  I
> think that it's very useful for our Ignite community.
>
> Similar changes should be applied to
>
> org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory.getDataSource()
>
>
> private KeyValuePersistenceSettings getPersistenceSettings() {
> // comment the two line below for testing purpose.
> //if (persistenceSettings != null)
> //return persistenceSettings;
>
> if (persistenceSettingsBean == null) {
> throw new IllegalStateException("Either persistence settings
> bean or persistence settings itself " +
> "should be specified");
> }
>
> if (appCtx == null) {
> throw new IllegalStateException("Failed to get Cassandra
> persistence settings cause Spring application " +
> "context wasn't injected into CassandraCacheStoreFactory");
> }
>
> Object obj = loadSpringContextBean(appCtx,
> persistenceSettingsBean);
>
> if (!(obj instanceof KeyValuePersistenceSettings)) {
> throw new IllegalStateException("Incorrect persistence settings
> bean '" +
> persistenceSettingsBean + "' specified");
> }
>
> return persistenceSettings = (KeyValuePersistenceSettings)obj;
> }
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Thanks for your clarifications, and appreciated your suggestions and
guidance.  

My college and I went to ignite-cassandra module,  commented two lines,
*/testing purpose/*,  in
org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory.getPersistenceSettings(),
see the changes below.  Removing these two lines is to force Ignite to read
the configuration settings from the xml at runtime.  It looks the change
works and meets our requirement.  

I'm looking for the suggestions and recommendations from Ignite community
and you, and wondering if we make these changes in Ignite repository.  I
think that it's very useful for our Ignite community.  

Similar changes should be applied to
org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory.getDataSource()


private KeyValuePersistenceSettings getPersistenceSettings() {
// comment the two line below for testing purpose.
//if (persistenceSettings != null)
//return persistenceSettings;

if (persistenceSettingsBean == null) {
throw new IllegalStateException("Either persistence settings
bean or persistence settings itself " +
"should be specified");
}

if (appCtx == null) {
throw new IllegalStateException("Failed to get Cassandra
persistence settings cause Spring application " +
"context wasn't injected into CassandraCacheStoreFactory");
}

Object obj = loadSpringContextBean(appCtx, persistenceSettingsBean);

if (!(obj instanceof KeyValuePersistenceSettings)) {
throw new IllegalStateException("Incorrect persistence settings
bean '" +
persistenceSettingsBean + "' specified");
}

return persistenceSettings = (KeyValuePersistenceSettings)obj;
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

Ignite will use existing cache configuration if cache is already present.
There's nothing you can do here. This includes the cache store.

However, the cache store may determine what to keep in its serialized form,
and what to re-read/infer every time it is started. So as I have said,
you may improve Cassandra cache store to track environment changes.

Regards,

-- 
Ilya Kasnacheev


вт, 5 янв. 2021 г. в 04:38, xmw45688 :

> Hi Ilya,
>
> Thanks for your guidance and happy new year!  Sorry for late catchup.
>
> You are right, Ignite Native Persistence does track the changes in class
> definition for "Ignite Native Persistence".  But the configuration to load
> data from Cache to Cassandra store is stored in xml configuration file.
> This configuration xml file is passed at runtime (see the example
> configuration below). When ignite server is stopped, the cache
> configuration
> is gone, not stored in the any place as far as I understand.  When the
> Ignite is restarted, the new xml is passed, there is no previous
> configuration for the old class definition in the xml configuration.
>
> My question - how/where does the ignite server get/read the old class
> definition if this class definition is provided in the most recent xml
> config file?
>
> So what I said is that the Cassandra Store implementation may not need to
> change.  It's the call that reads the xml configuration and stores the data
> in Cassandra Store via Ignite Cache.   If I remove Ignite Native
> configuration ("persistenceEnabled" value="false"), then Ignite server uses
> the xml configuration file passed at runtime.  The Ignite Cache has all new
> class definiton.
>
> I'd like your help to guide me make the changes to read the xml
> configuration at runtime instead of reading the previously cached
> configuration.
>
>
> 
> class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
> 
> 
> 
> 
> 
>
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
>
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
>  value="Default_Region"/>
> 
>
> 
>
>
>  value="true"/>
> 
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  type="java.lang.Class">java.lang.String
>  type="java.lang.Class">com.procurant.catalog.entity.Uom
> 
> 
> 
>  class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
>  ref="cassandra"/>
>  value="catalog_uom_persistence_settings">
> 
> 
> 
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Hi Ilya,

Thanks for your guidance and happy new year!  Sorry for late catchup. 

You are right, Ignite Native Persistence does track the changes in class
definition for "Ignite Native Persistence".  But the configuration to load
data from Cache to Cassandra store is stored in xml configuration file. 
This configuration xml file is passed at runtime (see the example
configuration below). When ignite server is stopped, the cache configuration
is gone, not stored in the any place as far as I understand.  When the
Ignite is restarted, the new xml is passed, there is no previous
configuration for the old class definition in the xml configuration.  

My question - how/where does the ignite server get/read the old class
definition if this class definition is provided in the most recent xml
config file? 

So what I said is that the Cassandra Store implementation may not need to
change.  It's the call that reads the xml configuration and stores the data
in Cassandra Store via Ignite Cache.   If I remove Ignite Native
configuration ("persistenceEnabled" value="false"), then Ignite server uses
the xml configuration file passed at runtime.  The Ignite Cache has all new
class definiton. 

I'd like your help to guide me make the changes to read the xml
configuration at runtime instead of reading the previously cached
configuration.










   






















java.lang.String
com.procurant.catalog.entity.Uom












--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

Why do you think that change should be on Ignite Persistence side? You said
yourself that Ignite Persistence tracks the changes in class definition,
it's the Cassandra store that does not.

Regards,
-- 
Ilya Kasnacheev


ср, 23 дек. 2020 г. в 00:17, xmw45688 :

> Thanks for your suggestion, Ilya.  Can you give me a reference to override
> default Cassandra Store implementation?  I thought that the changes would
> be
> on Ignite Persistence side because caching data from standalone Cassandra
> store (with Ignite Persistence) did read configuration including class
> definition from the xml file.
>
> Regards,
> Xinmin
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Thanks for your suggestion, Ilya.  Can you give me a reference to override
default Cassandra Store implementation?  I thought that the changes would be
on Ignite Persistence side because caching data from standalone Cassandra
store (with Ignite Persistence) did read configuration including class
definition from the xml file. 

Regards,
Xinmin   



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

I think that maybe you could try coding such a cache store which will
update this information. It's just that the default Cassandra store doesn't
seem to.

Regards,
-- 
Ilya Kasnacheev


сб, 19 дек. 2020 г. в 05:17, xmw45688 :

> Thanks for your confirmation, Ilya.
>
> I do have a follow-up question for you.
> When Ignite Persistence is used together with Cassandra.  The caches for
> Cassandra table mappings are provided via xml file.  The java class for the
> mappings is for BOTH Ignite Persistence and Cassandra cache store.  When a
> table is changed, the mapping class gets changed, and the cluster is
> started. Why is Ignite Persistence able to use this updated java class
> (added/removed variables from the class) to insert/update data into Ignite
> Persistence stores.  But for Cassandra Store, it appears to use the old
> definition of class.  For this case, what's effort if we want to use the
> new
> class of definition?  Do you think that it's reasonable to read the cache
> configuration (i.e. via xml) dynamically for 3rd party Cache store which
> does if the 3rd party store is used along.  I'm not an expert in java, I
> can
> definitely help (or find a resource to do it) to improve this request.
>
> Pleas let me know if I can submit a feature request.
>
> Appreciate your help.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Thanks for your confirmation, Ilya.  

I do have a follow-up question for you. 
When Ignite Persistence is used together with Cassandra.  The caches for
Cassandra table mappings are provided via xml file.  The java class for the
mappings is for BOTH Ignite Persistence and Cassandra cache store.  When a
table is changed, the mapping class gets changed, and the cluster is
started. Why is Ignite Persistence able to use this updated java class
(added/removed variables from the class) to insert/update data into Ignite
Persistence stores.  But for Cassandra Store, it appears to use the old
definition of class.  For this case, what's effort if we want to use the new
class of definition?  Do you think that it's reasonable to read the cache
configuration (i.e. via xml) dynamically for 3rd party Cache store which
does if the 3rd party store is used along.  I'm not an expert in java, I can
definitely help (or find a resource to do it) to improve this request.  

Pleas let me know if I can submit a feature request. 

Appreciate your help.  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

This is currently by design that cache configuration, of which cache store
configuration is a part, cannot be changed after the cache is started.

Regards,
-- 
Ilya Kasnacheev


ср, 16 дек. 2020 г. в 04:29, xmw45688 :

> Thank you for your answer.  I'm able to load the data from Cassandra to
> Ignite Cache stores using Cache.loadfull().  This works well for the
> use
> cases where 3rd party persistence store is used first, then Ignite Native
> Persistence store is used with 3rd party persistence together later.
>
> However, the configuration of 3rd party persistence store along with Ignite
> Native Persistence store appears to be stored inside the Native Persistence
> Store or somewhere, any changes in xml configuration will not propagate to
> the 3rd party data store.  For example, adding a new column to an ignite
> cache, the data for this newly added column is not saved in the 3rd party
> persistence store when the data in is saved into the Ignite Native
> Persistence Store.  In order to save the new added data for this new
> column,
> the existing cache need be destroyed and created.  This works for caching
> small sets of data.  It's probably not practically to destroy a cache with
> huge data set and recreate the cache again.
>
> I don't know what I've missed, or this is by design.  If it's by design,
> I'd
> like to request an enhance so that the configuration can be dynamically
> read
> from xml file when the Ignite Native Persistence store is configured with a
> 3rd party store together.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

Thank you for your answer.  I'm able to load the data from Cassandra to
Ignite Cache stores using Cache.loadfull().  This works well for the use
cases where 3rd party persistence store is used first, then Ignite Native
Persistence store is used with 3rd party persistence together later.  

However, the configuration of 3rd party persistence store along with Ignite
Native Persistence store appears to be stored inside the Native Persistence
Store or somewhere, any changes in xml configuration will not propagate to
the 3rd party data store.  For example, adding a new column to an ignite
cache, the data for this newly added column is not saved in the 3rd party
persistence store when the data in is saved into the Ignite Native
Persistence Store.  In order to save the new added data for this new column,
the existing cache need be destroyed and created.  This works for caching
small sets of data.  It's probably not practically to destroy a cache with
huge data set and recreate the cache again. 

I don't know what I've missed, or this is by design.  If it's by design, I'd
like to request an enhance so that the configuration can be dynamically read
from xml file when the Ignite Native Persistence store is configured with a
3rd party store together. 





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[Ilya Kasnacheev]

Hello!

I'm afraid that you may need to drop and recreate these caches to change
their configuration, including Cache Store config.

Regards,
-- 
Ilya Kasnacheev


пн, 28 сент. 2020 г. в 00:27, xmw45688 :

> I'm able to use Cassandra Together with Ignite Persistence (see the example
> below) usingIgnite 2.8.1.  However I could not change to another keyspace
> which has the same tables as the keyspace initially configurfed in the
> ignite config file.  The data changes are still stored in the old keyspace
> instead of the new keyspace that are in ignite config file.  It's seem to
> me
> that the keyspace are cached in the Ignite native persistence layer.
>
> How do I change the keyspaces of Cassandra?
>
> The use case is that we use Ignite as cache initially with the data stored
> in Cassandra.  With the data increase, caching all data is impractical.  So
> we plan to flush the data into the disk when the memory reaches the
> limits.
> We still want to store the data into Cassandra at the same time.  How do we
> load the the existing data from Cassandra Store to Native Persistence store
> when switching from Ignite Cache + Cassandra Store to Ignite Cache +Ignite
> persistence + Cassandra Store?
> Here is what I did -
> 1) duplicated the existing tables (no data) into the another keyspace in
> Cassandra
> 2) read the data from the original keyspace with Cassndra driver. This
> loaded the data into
>   2.1) ignite cache and native persistence
>   2.2) store the data into the new Cassandra keyspace.  it's undesirable to
> the store the original data back to the original keyspace.
> 3) change the cassandra connection setting to switch to the original
> keyspace because Ignite Cache and Native Persistence layer have all data
> from the Cassandra.
>   3.1) switching to the original cassandra keyspace did not take any
> effect.
> The data changes were still stored into the new keyspace.
>
> Config files
> 1. cassandra-connection-settings.xml
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>
>  class="com.datastax.driver.core.policies.TokenAwarePolicy">
>  type="com.datastax.driver.core.policies.LoadBalancingPolicy">
>  class="com.datastax.driver.core.policies.RoundRobinPolicy"/>
> 
> 
> 
> class="com.datastax.driver.core.policies.ExponentialReconnectionPolicy">
> 
> 
> 
>
>  class="com.datastax.driver.core.policies.DefaultRetryPolicy">
> 
>
>
>  class="org.apache.ignite.cache.store.cassandra.datasource.DataSource"
> name="cassandra">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>
> 2. dim_store_key_persistence.xm
> 
> 
> 
> 
> 
> 
>  >
> 
> 
> 
> 
> 
> 
>
> 3. ignite config
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>
>
>
>
> 
>
> 
>
>
> 
> class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
>  value="classpath:dim_store_key_persistence.xml" />
> 
>
>
> 
>
> 
> 
>
>
>  value="/c:/apache/data/home/work-cassandra"/>
>
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
>
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
> 
>
> 
>
>
> 
> 
> 
>
>  value="c:/apache/data/simple-cassandra/storage"/>
> 
> 
>  value="c:/apache/data/simple-cassandra/wal"/>
>  value="c:/apache/data/simple-cassandra/wal/archive"/>
>
> 
> 
>
> 
>

not able to change keyspace name when Using 3rd Party Persistence (Cassandra) Together with Ignite Native Persistence

[xmw45688]

I'm able to use Cassandra Together with Ignite Persistence (see the example
below) usingIgnite 2.8.1.  However I could not change to another keyspace
which has the same tables as the keyspace initially configurfed in the
ignite config file.  The data changes are still stored in the old keyspace
instead of the new keyspace that are in ignite config file.  It's seem to me
that the keyspace are cached in the Ignite native persistence layer.  

How do I change the keyspaces of Cassandra?  

The use case is that we use Ignite as cache initially with the data stored
in Cassandra.  With the data increase, caching all data is impractical.  So
we plan to flush the data into the disk when the memory reaches the limits. 
We still want to store the data into Cassandra at the same time.  How do we
load the the existing data from Cassandra Store to Native Persistence store
when switching from Ignite Cache + Cassandra Store to Ignite Cache +Ignite
persistence + Cassandra Store?
Here is what I did - 
1) duplicated the existing tables (no data) into the another keyspace in
Cassandra
2) read the data from the original keyspace with Cassndra driver. This
loaded the data into
  2.1) ignite cache and native persistence 
  2.2) store the data into the new Cassandra keyspace.  it's undesirable to
the store the original data back to the original keyspace.
3) change the cassandra connection setting to switch to the original
keyspace because Ignite Cache and Native Persistence layer have all data
from the Cassandra. 
  3.1) switching to the original cassandra keyspace did not take any effect. 
The data changes were still stored into the new keyspace.   

Config files 
1. cassandra-connection-settings.xml
http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>




























2. dim_store_key_persistence.xm






  







3. ignite config 

http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>




   


















   
















   
   










  
 






com.procurant.model.IdKey
com.procurant.model.DimStore















 












--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Native Persistence With write behind additional store

[Evgenii Zhuravlev]

There is a typo in my previous message, I meant "Storages will be
synchronized in case of one of the node failure."

There is no reference for this type of configuration since there is no
guarantee for the consistency. If you still want to use it, you can just
combine configuration for persistence with a configuration for 3rd party
cache store. Both can be found in the documentation and Ignite examples.

Evgenii

ср, 15 июл. 2020 г. в 08:22, Devakumar J :

> Hi,
>
> Thanks for the reply.
>
> is there a reference to configure custom store along with native
> persistence
> enabled.
>
> Thanks & Regards,
> Devakumar J
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite Native Persistence With write behind additional store

[Devakumar J]

Hi,

Thanks for the reply.

is there a reference to configure custom store along with native persistence
enabled.

Thanks & Regards,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Native Persistence With write behind additional store

[Evgenii Zhuravlev]

Hi,

There is no guarantee of data consistency between Ignite persistence and
3rd party DB in this case. Storages will be synchronized in case of one of
the node failure. You can try to run some explicit checking for data
consistency, but I believe it won't be easy with the load.

Evgenii

вт, 14 июл. 2020 г. в 08:23, Devakumar J :

> Hi,
>
> I am exploring ways to do data backup along with native persistence. Is it
> possible to achieve this using cache store implementation. So that data
> will
> be persisted in disk as well as replica copy.
>
> Thanks,
> Devakumar J
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Ignite Native Persistence With write behind additional store

[Devakumar J]

Hi,

I am exploring ways to do data backup along with native persistence. Is it
possible to achieve this using cache store implementation. So that data will
be persisted in disk as well as replica copy.

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: When trying to activate cluster to use ignite native persistence getting topology update failed error .

[Ilya Kasnacheev]

Hello!

By default, information is logged to console (standard out). Please start
your node with -DIGNITE_QUIET=false (or bin/ignite.sh -v) and copy-paste
its output as you try to activate cluster.

Regards,
-- 
Ilya Kasnacheev


чт, 27 февр. 2020 г. в 08:39, Preet :

> I am new to Ignite. I don't know how to check log. When I searched then I
> got
> to know that it is under /work/logs but I think that are static file. I
> don't know why activating cluster node is failing. I want to use
> persistence
> enabled feature for my IGFS application.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: When trying to activate cluster to use ignite native persistence getting topology update failed error .

[Preetiiii]

I am new to Ignite. I don't know how to check log. When I searched then I got
to know that it is under /work/logs but I think that are static file. I
don't know why activating cluster node is failing. I want to use persistence
enabled feature for my IGFS application. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: When trying to activate cluster to use ignite native persistence getting topology update failed error .

[Ilya Kasnacheev]

Hello!

Do you see anything in server nodes' logs?

Regards,
-- 
Ilya Kasnacheev


ср, 26 февр. 2020 г. в 14:31, Preet :

> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2787/Screenshot_2020-02-26_at_4.png
> >
> This is the error I received. Please help me to resolve this error. I want
> to use native persistence feature.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

When trying to activate cluster to use ignite native persistence getting topology update failed error .

[Preetiiii]

This is the error I received. Please help me to resolve this error. I want
to use native persistence feature.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[niamin]

Tried with appending cache name as schema with no success. Attach is a link
to my project:
https://drive.google.com/open?id=1F53um8TeUK45U3SOW0_Vlj04S8DWyjKI

Please take a look. I suspect it is something in my code.

Thanks,
Naushad



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[Denis Magda]

You might need to set the cache name as a schema for the queries. For
instance, the query needs to be transformed to "Select ...
{YOUR_CACHE_NAME}.ARInvoice FROM ..".

Also, do you really need to have both 3rd party and native persistence
enabled for the cluster. If the 3rd party is only used as an initial data
source during the loading phase then you can use can create a simple custom
loader that pulls records from the 3rd party and loads them into Ignite via
IgniteDataStreamer:
https://apacheignite.readme.io/docs/data-loading#ignitedatastreamer


-
Denis


On Tue, Nov 26, 2019 at 11:44 AM niamin  wrote:

> ignite_sql_error.ignite_sql_error
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2637/ignite_sql_error.ignite_sql_error>
>
>
> ARInvoiceRepository.java
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2637/ARInvoiceRepository.java>
>
>
> Attached is the file containing statcktrace and the repository file. The
> test case is trying to executed the
> getInvoicesForOIStatement method.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite native persistence

[niamin]

ignite_sql_error.ignite_sql_error

  

ARInvoiceRepository.java

  

Attached is the file containing statcktrace and the repository file. The
test case is trying to executed the 
getInvoicesForOIStatement method.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[Gaurav Bajaj]

Please send exception/error message you are getting and the query you are
trying to execute.


On Tue, Nov 26, 2019, 8:04 PM niamin  wrote:

> I found what I was doing wrong. I can now load/pre-fetch from 3rd party
> store
> to cache. It looks like the data is present in in-memory cache but also
> stored in the persistent store. However, when I run a native query
> configured on my SpringData Repository I get an error that the SQL table
> mentioned in the query is not found. The same query/api was working when
> persistent store was not enabled.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite native persistence

[niamin]

I found what I was doing wrong. I can now load/pre-fetch from 3rd party store
to cache. It looks like the data is present in in-memory cache but also
stored in the persistent store. However, when I run a native query
configured on my SpringData Repository I get an error that the SQL table
mentioned in the query is not found. The same query/api was working when
persistent store was not enabled.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[niamin]

Thanaks Denis. However, how do or should I preload the data to disk first? Or
as Ignite fetches data from the 3rd part datastore it is expected to
distribute between RAM and disk? Is there an example project that implements
this strategy? On my POC, I have SpringBoot project that includes the Ignite
Server and I was using SpringDataRepoository to configure the Queries which
were executed by Ignite server. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[Denis Magda]

Hi there,

Looks like you are trying to preload data from the persistence to memory
with Cache.loadCache method. Presently, this operation is not supported for
the native persistence and it’s optional - just start issuing your queries
after the cluster restart and Ignite will get all the data needed for you
from disk. Eventually all the data will be loaded in RAM while your app is
running and querying the cluster.

Does it answer your question?

On Monday, November 25, 2019, niamin  wrote:

> What is the right way to enable native persistence? As per Ignite
> documentation, I added the following property to enable native persistence:
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
>   
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
>value="true"/>
> 
>   
> 
>   
>
> I also added ignite.start(true) to my loader. However, I keep getting the
> following WARN level message:
>
> 2019-11-25 12:52:26.709  WARN 16599 --- [   main]
> o.a.i.i.p.c.store.CacheOsStoreManager:  Calling Cache.loadCache()
> method will have no effect, CacheConfiguration.getStore() is not defined
> for
> cache: FOO
>
>
> The solution works as an in-memory cache/solution though.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
-
Denis

Ignite native persistence

[niamin]

What is the right way to enable native persistence? As per Ignite
documentation, I added the following property to enable native persistence:


  

  

  

  

I also added ignite.start(true) to my loader. However, I keep getting the
following WARN level message:

2019-11-25 12:52:26.709  WARN 16599 --- [   main]
o.a.i.i.p.c.store.CacheOsStoreManager:  Calling Cache.loadCache()
method will have no effect, CacheConfiguration.getStore() is not defined for
cache: FOO


The solution works as an in-memory cache/solution though. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Using Ignite Native Persistence as a "temporary durable" cache

[Alexander Korenshteyn]

origonal question:
ferhadcebi...@gmail.com
5:00 AM (5 hours ago)
 to me


 Thanks for answer. But, how Ignite will store that datas to store? Will
append to the end of WAL? If, then it is sure faster than storing cache
operations in some 3rd part database.


Ignite does store data to WAL first, but per my benchmarks, storing 100,000
records took a few seconds, and deleting took a few seconds more.

 You can experiment using :

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java


Here is sample code: (modified section in above example)
[image: image.png]

On Thu, Sep 5, 2019 at 12:17 PM Alexander Korenshteyn <
alexanderko...@gmail.com> wrote:

> Hello,
>Ignite native persistence has a good track record, is fast and
> reliable, you can use it in your application.
>
>Take a look at the following example of how to use a streamer to
> quickly insert data:
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java
>
>Use cache.removeAll() -- to quickly remove all entries once you are
> done.
> Thanks, Alex
>
> On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev 
> wrote:
>
>> Hi all,
>>
>> We have MS SQL database server which contains all data. Our application
>> will fetch some datas from database server and put them to the cache. And
>> then, during 5-10 seconds we will do some updates to that objects and push
>> that changes to ignite in-memory cache. And then, after 5-10 seconds we
>> will take that changes and sync them will database as a bulk operation and
>> then will clear that permanent storage. So, we need some permanent storage
>> to store those updates for 5-10 seconds.
>> And actually, we will work at most with 100.000 entity.
>>
>> The idea behind that flow is that we can't now scale DB and users already
>> putting load to the database.
>>
>> We can't decide whether we have to use Native Persistent or some another
>> 3rd party database like Maria or PostgreSQL for storing that cache
>> operations for 5-10 seconds.
>>
>> Will Ignite works fast if we will use native persistent and clear the
>> cache periodically?
>>
>

Re: Using Ignite Native Persistence as a "temporary durable" cache

[Alexander Korenshteyn]

Hello,
   Ignite native persistence has a good track record, is fast and reliable,
you can use it in your application.

   Take a look at the following example of how to use a streamer to quickly
insert data:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java

   Use cache.removeAll() -- to quickly remove all entries once you are done.
Thanks, Alex

On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev 
wrote:

> Hi all,
>
> We have MS SQL database server which contains all data. Our application
> will fetch some datas from database server and put them to the cache. And
> then, during 5-10 seconds we will do some updates to that objects and push
> that changes to ignite in-memory cache. And then, after 5-10 seconds we
> will take that changes and sync them will database as a bulk operation and
> then will clear that permanent storage. So, we need some permanent storage
> to store those updates for 5-10 seconds.
> And actually, we will work at most with 100.000 entity.
>
> The idea behind that flow is that we can't now scale DB and users already
> putting load to the database.
>
> We can't decide whether we have to use Native Persistent or some another
> 3rd party database like Maria or PostgreSQL for storing that cache
> operations for 5-10 seconds.
>
> Will Ignite works fast if we will use native persistent and clear the
> cache periodically?
>

Using Ignite Native Persistence as a "temporary durable" cache

[Farhad Jabiyev]

Hi all,

We have MS SQL database server which contains all data. Our application
will fetch some datas from database server and put them to the cache. And
then, during 5-10 seconds we will do some updates to that objects and push
that changes to ignite in-memory cache. And then, after 5-10 seconds we
will take that changes and sync them will database as a bulk operation and
then will clear that permanent storage. So, we need some permanent storage
to store those updates for 5-10 seconds.
And actually, we will work at most with 100.000 entity.

The idea behind that flow is that we can't now scale DB and users already
putting load to the database.

We can't decide whether we have to use Native Persistent or some another
3rd party database like Maria or PostgreSQL for storing that cache
operations for 5-10 seconds.

Will Ignite works fast if we will use native persistent and clear the cache
periodically?

Re: SW recommendation: Ignite Native Persistence for traditional relational data warehouse?

[Denis Magda]

Are you available for a verbal conversation? I would invite a solution
architect on a call to figure out if Ignite works or not for your use case.

-
Denis


On Wed, May 15, 2019 at 10:17 PM Denis Magda  wrote:

> Hi,
>
> But after seeing your explanation below I understand that option 2 above is
>> not really the way Ignite is supposed to be used - even if it is on top of
>> hadoop. Did I get that right?
>
>
> In this configuration, Ignite will not be on top of Hadoop, it will be
> close to it - deployed as separate storage for faster performance.
> Synchronization is possible if needed.
>
> As for the other details, let's connect privately, more details are needed
> to give useful suggestions.
>
> --
> Denis Magda
>
>
> On Wed, May 15, 2019 at 4:15 AM m4mmr  wrote:
>
>> Hi Denis,
>>
>> Thank you very much for a good response. Definitely helps. Hope I can ask
>> one follow-up question which I feel I did not make so clear from the
>> beginning:
>>
>> The business has a very strong (non-negotiable) requirement on that the
>> data
>> warehouse should be modeled with high normalisation. These model
>> requirements prevents us from building the data warehouse on hadoop(hive).
>> We won't get rid of hadoop though - it is still used for offloading the
>> operational source systems.
>>
>> Therefore we need to:
>> 1. Either build a data warehouse on a RDBMS on top of the hadoop - to meet
>> the business requirements.
>> 2. Or build the relational data warehouse directly on Ignite with
>> Persistence Store - on top of hadoop.
>>
>> But after seeing your explanation below I understand that option 2 above
>> is
>> not really the way Ignite is supposed to be used - even if it is on top of
>> hadoop. Did I get that right?
>>
>> We can of course use Ignite with the RDBMS from option 1 as persistence
>> store for making the DWH offloading more effective later on. Will look
>> into
>> that.
>>
>> Again - thank you very much for you pedagogical response Denis.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: SW recommendation: Ignite Native Persistence for traditional relational data warehouse?

[Denis Magda]

Hi,

But after seeing your explanation below I understand that option 2 above is
> not really the way Ignite is supposed to be used - even if it is on top of
> hadoop. Did I get that right?


In this configuration, Ignite will not be on top of Hadoop, it will be
close to it - deployed as separate storage for faster performance.
Synchronization is possible if needed.

As for the other details, let's connect privately, more details are needed
to give useful suggestions.

--
Denis Magda


On Wed, May 15, 2019 at 4:15 AM m4mmr  wrote:

> Hi Denis,
>
> Thank you very much for a good response. Definitely helps. Hope I can ask
> one follow-up question which I feel I did not make so clear from the
> beginning:
>
> The business has a very strong (non-negotiable) requirement on that the
> data
> warehouse should be modeled with high normalisation. These model
> requirements prevents us from building the data warehouse on hadoop(hive).
> We won't get rid of hadoop though - it is still used for offloading the
> operational source systems.
>
> Therefore we need to:
> 1. Either build a data warehouse on a RDBMS on top of the hadoop - to meet
> the business requirements.
> 2. Or build the relational data warehouse directly on Ignite with
> Persistence Store - on top of hadoop.
>
> But after seeing your explanation below I understand that option 2 above is
> not really the way Ignite is supposed to be used - even if it is on top of
> hadoop. Did I get that right?
>
> We can of course use Ignite with the RDBMS from option 1 as persistence
> store for making the DWH offloading more effective later on. Will look into
> that.
>
> Again - thank you very much for you pedagogical response Denis.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SW recommendation: Ignite Native Persistence for traditional relational data warehouse?

[m4mmr]

Hi Denis, 

Thank you very much for a good response. Definitely helps. Hope I can ask
one follow-up question which I feel I did not make so clear from the
beginning:

The business has a very strong (non-negotiable) requirement on that the data
warehouse should be modeled with high normalisation. These model
requirements prevents us from building the data warehouse on hadoop(hive).
We won't get rid of hadoop though - it is still used for offloading the
operational source systems. 

Therefore we need to:
1. Either build a data warehouse on a RDBMS on top of the hadoop - to meet
the business requirements.
2. Or build the relational data warehouse directly on Ignite with
Persistence Store - on top of hadoop.

But after seeing your explanation below I understand that option 2 above is
not really the way Ignite is supposed to be used - even if it is on top of
hadoop. Did I get that right?

We can of course use Ignite with the RDBMS from option 1 as persistence
store for making the DWH offloading more effective later on. Will look into
that.

Again - thank you very much for you pedagogical response Denis. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SW recommendation: Ignite Native Persistence for traditional relational data warehouse?

[Denis Magda]

Hello,

In short, Ignite persistence primary advantages are:

   - Enables multi-tiered storage across RAM and disk - 100% of data
   persisted to disk ("warm" and "cold" data sets) while "hot" data always
   stays in RAM. You lay out the data the way you need. Applications just run
   queries and Ignite internally goes either to RAM or disk, transparently.
   This property is crucial for real-time analytics and data warehousing
   offloading.
   - Instantaneous restarts and advanced high-availability - in case of
   full cluster restarts, the cluster becomes fully operational as soon as the
   nodes are interconnected. No need to preload anything from disk to RAM
   (based on the advantage above).

Now, talking about Ignite and data warehousing. Ignite is used for
real-time analytics and Hadoop offloading. But don't treat it as a Hadoop
replacement or a solution for data warehousing. Ignite is used together
with Hadoop but deployed separately. Here are things to consider:

   - Use Ignite for business operations/computations that require
   low-latency response time (seconds or milliseconds) and high-throughput.
   Preload data to Ignite cluster needed for this computations. Enabled Ignite
   persistence for the sake of durability.
   - Keep using Hadoop for high-latency workloads (minutes and hours) and
   batch processing.
   - APIs: modify your applications to ensure that Ignite APIs are used for
   Ignite cluster access (SQL, compute grid, ML). Spark can be used as a
   generic API that can connect to both Hadoop and Ignite and run joins across
   2 storages (use DataFrames).
   - Tooling: data preloading from Hadoop, bi-directional synchronization,
   advanced Spark integration, etc. - reach out GridGain, they have been
   working on a special Hadoop pack.

Hope it helps.

-
Denis


On Tue, May 14, 2019 at 3:20 PM m4mmr  wrote:

> Hi,
>
> I am in a project where we are building a new database with strong
> normalisation requirements - very much like a relational data warehouse.
> We
> source the data from HDFS. And the maintenance team requires the data
> movement to be implemented through SQL APIs.
>
> Main Question: Would it be a viable use case to use Ignite with native
> persistence store as storage when building a relational data warehouse?  So
> that we do not have to manage both a RDBMS and Ignite.
>
> I don’t fully see  a clear picture of what the limitations would be of
> using
> native persistence instead of RDBMS for persistence - but I have not seen a
> single use case where someone use native for relational data warehouses
> either.
>
> Again - I think I am missing some basic understanding of the native
> persistence here - even after reading through the docs I could find. So
> would be happy if someone could shed some light on it.
>
> Would be very thankful for all type of help/assistance!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

SW recommendation: Ignite Native Persistence for traditional relational data warehouse?

[m4mmr]

Hi,

I am in a project where we are building a new database with strong
normalisation requirements - very much like a relational data warehouse.  We
source the data from HDFS. And the maintenance team requires the data
movement to be implemented through SQL APIs.

Main Question: Would it be a viable use case to use Ignite with native
persistence store as storage when building a relational data warehouse?  So
that we do not have to manage both a RDBMS and Ignite.

I don’t fully see  a clear picture of what the limitations would be of using
native persistence instead of RDBMS for persistence - but I have not seen a
single use case where someone use native for relational data warehouses
either. 

Again - I think I am missing some basic understanding of the native
persistence here - even after reading through the docs I could find. So
would be happy if someone could shed some light on it.

Would be very thankful for all type of help/assistance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: pre-load data (Apache ignite native persistence store or Cassandra) into two partitioned cache tables

[Ilya Kasnacheev]

Hello!

If you are using two rows as @AffinityKeyMapped, you can join other tables
which use same two rows, BUT you can't join tables which use only first row
as @AffinityKeyMapped or only second row.

That's why you can join invoice_line to fact_purhcase_line - I guess they
both have invoiceId and factLineId and annotation on both.

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 22:38, xmw45688 :

> Can some one comments on the following questions in my previous post?
>
>
> 4. fact_purhcase_line, invoice and invoice line via factLineId and
> InvoiceID
> do not work, please see annotation below
>
> public class InvoiceLineKey {
> /** Primary key. */
> private long id;
>
> /** Foreign key to fact_purhcase_line */
> @AffinityKeyMapped
> private long factLineId;
>
> /** Foreign key to invoice */
> @AffinityKeyMapped
> private long invoiceId;
>
>
> 5. I don't quite understand that invoiceId affinity key mapped between
> invoice and invoice_line does not require factLineId key mapped between
> fact_purchase_line and invoice_line.  Is this because of having factId key
> affinity between purchase_fact and purchase_fact_line, between
> purchase_fact
> and invoice.
>
> So I just have the following key affinity mapped -
>
> purchase_fact -> factId-> purchase_fact_line
> purchase_fact -> factId -> invoice
> invoice -> invoiceId -> invoice_line
>
> Interestingly, invoice_line join fact_purhcase_line works fine (see
> queries
> below).  Can someone please shed some lights on this?
>
> // expected
> SELECT count(*) from PARTITION.invoice inv, PARTITION.invoiceline il
> WHERE inv.id = il.invoiceid;
>
> // why does this query work? note there is a join between
> li.id=il.factLineId which is not a key affinity mapped.
> SELECT count(*)
>  from PARTITION.factpurchaseline li, PARTITION.invoice inv,
> PARTITION.invoiceline il
> WHERE li.id = il.factlineid
>   AND inv.id = il.invoiceid
> ;
>
> // why does this query work? note there is a join between
> li.id=il.factLineId which is not a key affinity mapped.
> SELECT count(*) from PARTITION.factpurchaseline li, PARTITION.invoiceline
> il
> WHERE li.id = il.factlineid
> ;
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: pre-load data (Apache ignite native persistence store or Cassandra) into two partitioned cache tables

[xmw45688]

Can some one comments on the following questions in my previous post?


4. fact_purhcase_line, invoice and invoice line via factLineId and InvoiceID 
do not work, please see annotation below 

public class InvoiceLineKey { 
/** Primary key. */ 
private long id; 

/** Foreign key to fact_purhcase_line */ 
@AffinityKeyMapped 
private long factLineId; 

/** Foreign key to invoice */ 
@AffinityKeyMapped 
private long invoiceId; 


5. I don't quite understand that invoiceId affinity key mapped between 
invoice and invoice_line does not require factLineId key mapped between 
fact_purchase_line and invoice_line.  Is this because of having factId key 
affinity between purchase_fact and purchase_fact_line, between purchase_fact 
and invoice.   

So I just have the following key affinity mapped - 

purchase_fact -> factId-> purchase_fact_line 
purchase_fact -> factId -> invoice 
invoice -> invoiceId -> invoice_line 

Interestingly, invoice_line join fact_purhcase_line works fine (see queries 
below).  Can someone please shed some lights on this? 

// expected 
SELECT count(*) from PARTITION.invoice inv, PARTITION.invoiceline il 
WHERE inv.id = il.invoiceid; 

// why does this query work? note there is a join between 
li.id=il.factLineId which is not a key affinity mapped. 
SELECT count(*) 
 from PARTITION.factpurchaseline li, PARTITION.invoice inv, 
PARTITION.invoiceline il 
WHERE li.id = il.factlineid 
  AND inv.id = il.invoiceid 
; 

// why does this query work? note there is a join between 
li.id=il.factLineId which is not a key affinity mapped. 
SELECT count(*) from PARTITION.factpurchaseline li, PARTITION.invoiceline il 
WHERE li.id = il.factlineid 
; 






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: pre-load data (Apache ignite native persistence store or Cassandra) into two partitioned cache tables

[xmw45688]

Thanks for reply.  I got the most working thanks to the example
(https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/starschema/CacheStarSchemaExample.java)
provided by Ignite.  

Here is my sql for POC (Cassandra DDL scripts)
create table ignitetest.dim_store (id bigint primary key, name varchar, addr
varchar, zip varchar);
create table ignitetest.dim_product (id bigint primary key, name varchar,
price double, qty int);
create table ignitetest.fact_purchase (id bigint primary key, productId
bigint,  storeId bigint, purchasePrice double);
create table ignitetest.fact_purchase_line(id bigint , factId bigint, line
int, linePrice double, lineQty int,  primary key (id, factId));
create table ignitetest.invoice (id bigint, factId bigint, productId bigint, 
storeId bigint, purchasePrice double, primary key (id));
create table ignitetest.invoice_line (id bigint, invoiceId bigint,
factLineId bigint, line int, price double, qty int, primary key (id,
invoiceId, factLineId));

Have following key affinity mapped - 
purchase_fact -> factId-> purchase_fact_line
purchase_fact -> factId -> invoice
invoice -> invoiceId -> invoice_line

1. fact_purchase and fact_purchase_line via factId affinity works expected.
2. fact_purchase and invoice via factId affinity works expected.
3. invoice and invoice_line via invoiceId affinity works expected. 

However, 
4. fact_purhcase_line, invoice and invoice line via factLineId and InvoiceID
do not work, please see annotation below

public class InvoiceLineKey {
/** Primary key. */
private long id;

/** Foreign key to fact_purhcase_line */
@AffinityKeyMapped
private long factLineId;

/** Foreign key to invoice */
@AffinityKeyMapped
private long invoiceId;


5. I don't quite understand that invoiceId affinity key mapped between
invoice and invoice_line does not require factLineId key mapped between
fact_purchase_line and invoice_line.  Is this because of having factId key
affinity between purchase_fact and purchase_fact_line, between purchase_fact
and invoice.  

So I just have the following key affinity mapped - 

purchase_fact -> factId-> purchase_fact_line
purchase_fact -> factId -> invoice
invoice -> invoiceId -> invoice_line

Interestingly, invoice_line join fact_purhcase_line works fine (see queries
below).  Can someone please shed some lights on this?

// expected
SELECT count(*) from PARTITION.invoice inv, PARTITION.invoiceline il
WHERE inv.id = il.invoiceid;

// why does this query work? note there is a join between
li.id=il.factLineId which is not a key affinity mapped.
SELECT count(*) 
 from PARTITION.factpurchaseline li, PARTITION.invoice inv,
PARTITION.invoiceline il
WHERE li.id = il.factlineid
  AND inv.id = il.invoiceid 
;

// why does this query work? note there is a join between
li.id=il.factLineId which is not a key affinity mapped.
SELECT count(*) from PARTITION.factpurchaseline li, PARTITION.invoiceline il
WHERE li.id = il.factlineid
;




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: pre-load data (Apache ignite native persistence store or Cassandra) into two partitioned cache tables

[Ilya Kasnacheev]

Hello!

1) The best way of preloading data is to add CacheLoader (or CacheStore) to
cache configuration and call loadCache().

2) Data will be collocated as long as both caches use same affinity field
(via annotation, AffinityKey or configuration).
Then if you join tables by that field Ignite will know.
This of course requires that other settings of AffinityFunction are the
same.

Regards,
-- 
Ilya Kasnacheev


пн, 18 февр. 2019 г. в 09:43, xmw45688 :

> Hi All,
>
> Apache Ignite recommends to collocate data with data, which is a very nice
> feature.  This works well if you insert data through APIs provided by
> Ignite
> (see the following code I copy from Ignite Doc).  However, I don't know how
> to handle the following cases
>   1. the data (company and person) exist in the other databases such as
> Cassandra.  How do I pre-cache company and person from pre-existed data?
> Do
> I have to use JDBC to load company, person data row by row, insert into
> person and cache objects, such as
>for org: orgs {
>find persons for org {
>personCache.put(affinityKey(p.id, org.id), org);
>}
>orgCache.put(org.id, org);
>}
>
>If I used Ignite native persistence store, how does Ignite cluster know
> that both company and person are collocated?
>
>   2. How do I know that @AffinityKeyMapped annotated key comes from the
> cache I need to map.  I don't see the logic to define relationship between
> person.companyId and company.id.  If I have two separated two methods to
> load company and person cache separately, will the collocation of persons
> with companies still work?
> preloadCompany();
> preloadPerson(long companyId);
>
> preloadAllPersons() {
> 1. get all companies from the companyCache
> // how personsCache will know this companyId is the same cluster node
> as
> companyCache?
> for (c: companies) {
>preloadPerson(c.companyId);
> }
> }
>
> The following code is excerpted from
> https://apacheignite.readme.io/docs/affinity-collocation.
>
>
> public class PersonKey {
> // Person ID used to identify a person.
> private String personId;
>
> // Company ID which will be used for affinity.
> @AffinityKeyMapped
> private String companyId;
> ...
> }
>
> // Instantiate person keys with the same company ID which is used as
> affinity key.
> Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
> Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
>
> Person p1 = new Person(personKey1, ...);
> Person p2 = new Person(personKey2, ...);
>
> // Both, the company and the person objects will be cached on the same
> node.
> comCache.put("myCompanyId", new Company(...));
> perCache.put(personKey1, p1);
> perCache.put(personKey2, p2);
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

pre-load data (Apache ignite native persistence store or Cassandra) into two partitioned cache tables

[xmw45688]

Hi All,

Apache Ignite recommends to collocate data with data, which is a very nice
feature.  This works well if you insert data through APIs provided by Ignite
(see the following code I copy from Ignite Doc).  However, I don't know how
to handle the following cases
  1. the data (company and person) exist in the other databases such as
Cassandra.  How do I pre-cache company and person from pre-existed data?  Do
I have to use JDBC to load company, person data row by row, insert into
person and cache objects, such as  
   for org: orgs {
   find persons for org {
   personCache.put(affinityKey(p.id, org.id), org);
   }
   orgCache.put(org.id, org);
   }

   If I used Ignite native persistence store, how does Ignite cluster know
that both company and person are collocated?

  2. How do I know that @AffinityKeyMapped annotated key comes from the
cache I need to map.  I don't see the logic to define relationship between
person.companyId and company.id.  If I have two separated two methods to
load company and person cache separately, will the collocation of persons
with companies still work? 
preloadCompany();
preloadPerson(long companyId);

preloadAllPersons() {
1. get all companies from the companyCache
// how personsCache will know this companyId is the same cluster node as
companyCache?
for (c: companies) {
   preloadPerson(c.companyId);
}
}

The following code is excerpted from
https://apacheignite.readme.io/docs/affinity-collocation.  


public class PersonKey {
// Person ID used to identify a person.
private String personId;
 
// Company ID which will be used for affinity.
@AffinityKeyMapped
private String companyId;
...
}

// Instantiate person keys with the same company ID which is used as
affinity key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
 
Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);
 
// Both, the company and the person objects will be cached on the same node.
comCache.put("myCompanyId", new Company(...));
perCache.put(personKey1, p1);
perCache.put(personKey2, p2);




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How to add a new node to an existing cluster which Ignite native persistence is enabled?

[Lukas Polacek]

You can change the baseline topology (
https://apacheignite.readme.io/docs/baseline-topology) either via
control.sh or GridGain web console.

For example, you will see something like this after running "control.sh
--baseline":
Cluster state: active
Current topology version: 5

Baseline nodes:
   ConsistentID=5de27d47-cef3-4a5d-ac1e-6dfe45156e3f, STATE=ONLINE
   ConsistentID=7e610021-052c-48ac-8a93-1d1ec2e10fac, STATE=ONLINE


Number of baseline nodes: 2

Other nodes:
   ConsistentID=e019f021-5988-40c0-84fe-d0bb5226c720
Number of other nodes: 1

You can then add node e019f021-5988-40c0-84fe-d0bb5226c720 to your topology
via "control.sh --baseline add".

On Thu, Dec 20, 2018 at 5:47 AM soonjoin  wrote:

> Hi Team, I am testing Ignite on version 2.7.0. I used Ignite native
> persistence and cache mode is PARTITIONED.
>
> I found there’s no data was stored in the node which joined the cluster
> after the cluster is already active，even though I deactivate the cluster
> and
> activate it again. I have to delete the persistent data on all the existing
> nodes and  restart the cluster, then the new node worked properly.
>
> Is there any way that I can add a new node to an existing cluster which
> Ignite native persistence is enabled without deleting the old persistent
> data?
>
> Thanks a lot.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

How to add a new node to an existing cluster which Ignite native persistence is enabled?

[soonjoin]

Hi Team, I am testing Ignite on version 2.7.0. I used Ignite native
persistence and cache mode is PARTITIONED.

I found there’s no data was stored in the node which joined the cluster
after the cluster is already active，even though I deactivate the cluster and
activate it again. I have to delete the persistent data on all the existing
nodes and  restart the cluster, then the new node worked properly. 

Is there any way that I can add a new node to an existing cluster which
Ignite native persistence is enabled without deleting the old persistent
data?

Thanks a lot.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence and replicated caches - should it even be possible?

[kimec.ethome.sk]

Ilya, thank you very much for your response!

Kamil Mišúth

On 2018-12-17 15:13, Ilya Kasnacheev wrote:

Hello!

I'm not completely sure but persistent REPLICATED cache is the same as
PARTITIONED with MAXINT backups.

It means that every node will have a copy of data, but it has to be in
BLT to be used.

Regards,

--

Ilya Kasnacheev

пн, 17 дек. 2018 г. в 14:46, kimec.ethome.sk [1]
:


Hi all,

Could somebody confirm my conclusion below?

It seems it is possible to declare a REPLICATED cache configuration
for
caches that are mapped to a data region backed by the native
persistence
layer.
Ignite does not complain about this configuration and boots happily.

Yet, after cluster restart, during runtime, the cache behaves as if
it
was PARTITIONED - since the configuration says REPLICATED, Ignite
will
not attempt to reload the data from the node actually owning them (a

node that has the data stored on disk from before cluster restart).

Net effect is that a node that is not a member of a baseline
topology
will report the cache contains no data for a given key even though
persisted data actually do exist in the cluster (but on a separate
node).

The documentation is not very clear on whether REPLICATED caches are

supported by native persistence or not, but reading between the
lines[1], I guess the only supported use case for native persistence
is
PARTITIONED cache.

If that is so, I would expect the node declaring such a cache
configuration to fail fast during startup. Or maybe the
documentation
should state this more clearly. It is not very intuitive, to say the

least.

Anyway, could somebody kindly confirm my suspicion? Thank you!

Kamil Mišúth

[1]


https://apacheignite.readme.io/v2.7/docs/distributed-persistent-store#section-overview


Links:
--
[1] http://kimec.ethome.sk

Ignite native persistence and replicated caches - should it even be possible?

[kimec.ethome.sk]

Hi all,

Could somebody confirm my conclusion below?

It seems it is possible to declare a REPLICATED cache configuration for 
caches that are mapped to a data region backed by the native persistence 
layer.
Ignite does not complain about this configuration and boots happily. 
Yet, after cluster restart, during runtime, the cache behaves as if it 
was PARTITIONED - since the configuration says REPLICATED, Ignite will 
not attempt to reload the data from the node actually owning them (a 
node that has the data stored on disk from before cluster restart).


Net effect is that a node that is not a member of a baseline topology 
will report the cache contains no data for a given key even though 
persisted data actually do exist in the cluster (but on a separate 
node).


The documentation is not very clear on whether REPLICATED caches are 
supported by native persistence or not, but reading between the 
lines[1], I guess the only supported use case for native persistence is 
PARTITIONED cache.


If that is so, I would expect the node declaring such a cache 
configuration to fail fast during startup. Or maybe the documentation 
should state this more clearly. It is not very intuitive, to say the 
least.


Anyway, could somebody kindly confirm my suspicion? Thank you!

Kamil Mišúth

[1] 
https://apacheignite.readme.io/v2.7/docs/distributed-persistent-store#section-overview

Re: Can Ignite native persistence used with 3rd party persistence?

[Dmitriy Setrakyan]

Cross-sending to dev@.

Alexey,

This issue is marked to be fixed for 2.4 which is planned to be released in
a couple of weeks. Do you think you will be able to close this issue before
the release?

D.


On Mon, Dec 18, 2017 at 9:51 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Created the ticket: https://issues.apache.org/jira/browse/IGNITE-7235
>
> 2017-12-15 16:16 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:
>
>> Ray,
>>
>> With the current API it is impossible to get a reliable integration of
>> Ignite native persistence with 3rd party persistence. The reason is that
>> first, CacheStore interface does not have methods for 2-phase commit,
>> second, it would require significant changes to the persistence layer
>> itself to make a consistent crash recovery.
>>
>> We could allow setting the cache store interface with write-through from
>> primary nodes, but in this case, it would be a user's responsibility to
>> verify that the cache store is consistent with the Ignite cluster. We will
>> try to enable and document it in ignite 2.4.
>>
>> --AG
>>
>> 2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <andrey.mashen...@gmail.com>:
>>
>>> Hi Ray,
>>>
>>>
>>>> One more question here, how can a update or new inserts back-propagate
>>>> to
>>>> Ignite when another application(not ignite) writes to
>>>> persistence(hbase)?
>>>
>>>
>>> It is not supported.
>>>
>>>
>>>
>>> On Fri, Dec 1, 2017 at 12:08 PM, Ray <ray...@cisco.com> wrote:
>>>
>>>> http://apache-ignite-users.70518.x6.nabble.com/Two-persisten
>>>> t-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-n
>>>> ative-td18463.html
>>>>
>>>> Found a similar case here, I think I'll try Slava's suggestions first.
>>>>
>>>> One more question here, how can a update or new inserts back-propagate
>>>> to
>>>> Ignite when another application(not ignite) writes to
>>>> persistence(hbase)?
>>>>
>>>> For example, Ignite and hbase both have one entry for now.
>>>> When another application adds an entry to hbase, now hbase has two
>>>> entries.
>>>> Can Ignite be notified and load the newly added entry automatically?
>>>>
>>>> From the document, it looks like the data can only be propagated from
>>>> Ignite
>>>> to persistence, not the other way around.
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>

Re: Can Ignite native persistence used with 3rd party persistence?

[Alexey Goncharuk]

Created the ticket: https://issues.apache.org/jira/browse/IGNITE-7235

2017-12-15 16:16 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:

> Ray,
>
> With the current API it is impossible to get a reliable integration of
> Ignite native persistence with 3rd party persistence. The reason is that
> first, CacheStore interface does not have methods for 2-phase commit,
> second, it would require significant changes to the persistence layer
> itself to make a consistent crash recovery.
>
> We could allow setting the cache store interface with write-through from
> primary nodes, but in this case, it would be a user's responsibility to
> verify that the cache store is consistent with the Ignite cluster. We will
> try to enable and document it in ignite 2.4.
>
> --AG
>
> 2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <andrey.mashen...@gmail.com>:
>
>> Hi Ray,
>>
>>
>>> One more question here, how can a update or new inserts back-propagate to
>>> Ignite when another application(not ignite) writes to persistence(hbase)?
>>
>>
>> It is not supported.
>>
>>
>>
>> On Fri, Dec 1, 2017 at 12:08 PM, Ray <ray...@cisco.com> wrote:
>>
>>> http://apache-ignite-users.70518.x6.nabble.com/Two-persisten
>>> t-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-
>>> native-td18463.html
>>>
>>> Found a similar case here, I think I'll try Slava's suggestions first.
>>>
>>> One more question here, how can a update or new inserts back-propagate to
>>> Ignite when another application(not ignite) writes to persistence(hbase)?
>>>
>>> For example, Ignite and hbase both have one entry for now.
>>> When another application adds an entry to hbase, now hbase has two
>>> entries.
>>> Can Ignite be notified and load the newly added entry automatically?
>>>
>>> From the document, it looks like the data can only be propagated from
>>> Ignite
>>> to persistence, not the other way around.
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>

Re: Can Ignite native persistence used with 3rd party persistence?

[Alexey Goncharuk]

Ray,

With the current API it is impossible to get a reliable integration of
Ignite native persistence with 3rd party persistence. The reason is that
first, CacheStore interface does not have methods for 2-phase commit,
second, it would require significant changes to the persistence layer
itself to make a consistent crash recovery.

We could allow setting the cache store interface with write-through from
primary nodes, but in this case, it would be a user's responsibility to
verify that the cache store is consistent with the Ignite cluster. We will
try to enable and document it in ignite 2.4.

--AG

2017-12-01 14:13 GMT+03:00 Andrey Mashenkov <andrey.mashen...@gmail.com>:

> Hi Ray,
>
>
>> One more question here, how can a update or new inserts back-propagate to
>> Ignite when another application(not ignite) writes to persistence(hbase)?
>
>
> It is not supported.
>
>
>
> On Fri, Dec 1, 2017 at 12:08 PM, Ray <ray...@cisco.com> wrote:
>
>> http://apache-ignite-users.70518.x6.nabble.com/Two-persisten
>> t-data-stores-for-a-single-Ignite-cluster-RDBMS-and-
>> Ignite-native-td18463.html
>>
>> Found a similar case here, I think I'll try Slava's suggestions first.
>>
>> One more question here, how can a update or new inserts back-propagate to
>> Ignite when another application(not ignite) writes to persistence(hbase)?
>>
>> For example, Ignite and hbase both have one entry for now.
>> When another application adds an entry to hbase, now hbase has two
>> entries.
>> Can Ignite be notified and load the newly added entry automatically?
>>
>> From the document, it looks like the data can only be propagated from
>> Ignite
>> to persistence, not the other way around.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Re: Can Ignite native persistence used with 3rd party persistence?

[Andrey Mashenkov]

Hi Ray,


> One more question here, how can a update or new inserts back-propagate to
> Ignite when another application(not ignite) writes to persistence(hbase)?


It is not supported.



On Fri, Dec 1, 2017 at 12:08 PM, Ray  wrote:

> http://apache-ignite-users.70518.x6.nabble.com/Two-
> persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-
> and-Ignite-native-td18463.html
>
> Found a similar case here, I think I'll try Slava's suggestions first.
>
> One more question here, how can a update or new inserts back-propagate to
> Ignite when another application(not ignite) writes to persistence(hbase)?
>
> For example, Ignite and hbase both have one entry for now.
> When another application adds an entry to hbase, now hbase has two entries.
> Can Ignite be notified and load the newly added entry automatically?
>
> From the document, it looks like the data can only be propagated from
> Ignite
> to persistence, not the other way around.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Andrey V. Mashenkov

Re: Can Ignite native persistence used with 3rd party persistence?

[Ray]

http://apache-ignite-users.70518.x6.nabble.com/Two-persistent-data-stores-for-a-single-Ignite-cluster-RDBMS-and-Ignite-native-td18463.html

Found a similar case here, I think I'll try Slava's suggestions first.

One more question here, how can a update or new inserts back-propagate to
Ignite when another application(not ignite) writes to persistence(hbase)?

For example, Ignite and hbase both have one entry for now.
When another application adds an entry to hbase, now hbase has two entries.
Can Ignite be notified and load the newly added entry automatically?

>From the document, it looks like the data can only be propagated from Ignite
to persistence, not the other way around.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Can Ignite native persistence used with 3rd party persistence?

[Ray]

Ignite native persistence provides on-disk sql query and quick cluster
startup without data loading, so we definitely want to use it.
But we have a legacy hbase served as persistence layer and there are some
business rely on it.
So can Ignite native persistence used with 3rd party persistence?

Basically we want data to be both persisted both on native persistence and
hbase when new entries goes into Ignite.
And when user queries data not in memory, Ignite will query its native
persistence.

Is this design supported by Ignite?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Strange problems with Ignite native Persistence when Data exceeds Memory

[dkarachentsev]

Hi Roger,

The reason of slowdown might be the fact, that Ignite by default consumes
80% of available memory, which forces OS to start a lot of swapping, so all
memory operations limited by disk speed. Try to reduce memory size [1].

[1]
https://apacheignite.readme.io/v2.1/docs/memory-configuration#memory-policies

Thanks!
-Dmitry.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Strange-problems-with-Ignite-native-Persistence-when-Data-exceeds-Memory-tp16187p16336.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: Strange problems with Ignite native Persistence when Data exceeds Memory

[Roger Fischer (CW)]

Hi Dmitry,

yes, the data set is bigger than the available memory. I am evaluating native 
persistence, specifically SQL queries when pages have to be loaded from disk. 
Thus I need a data set that is significantly bigger than what fits the memory.

I made another attempt, loading smaller batches and leaving some time between 
batches for things to settle down. All went well.

Thus the problem seems to be related to concurrency and/or write workload. The 
fact that the 3 VMs all use the same physical disk (in the host) does not help 
the situation.

No, there are no duplicate keys. The exact same data set loads fine when 
slowing down the load.

As I mentioned, there were no logs indicating any connection, I/O or other 
issues.

Unfortunately I have to move on, and will not be able to rerun the scenario 
with additional logging at this time. Should I get to it later, I will update 
this thread.

Thanks...

Roger

-Original Message-
From: dkarachentsev [mailto:dkarachent...@gridgain.com] 
Sent: Tuesday, August 15, 2017 12:04 AM
To: user@ignite.apache.org
Subject: Re: Strange problems with Ignite native Persistence when Data exceeds 
Memory

Hi Roger,

It looks like not all data could fit in memory and pages started to swap into 
disk. This means that too many pages marked as dirty that causes more often and 
longer checkpointing, that with WAL writes may significantly slowdown data 
uploading. Please attach Ignite and GC logs and try to set
PersistenceStoreConfiguration.setWalMode() to LOG_ONLY or BACKGROUND (this will 
reduce disk loading).

Data loss looks strange, is it possible that you have duplicated entry keys?
DataStreamer by default does not override existed entries for performance 
reasons.

Thanks!
-Dmitry.



--
View this message in context: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Strange-2Dproblems-2Dwith-2DIgnite-2Dnative-2DPersistence-2Dwhen-2DData-2Dexceeds-2DMemory-2Dtp16187p16194.html=DwICAg=IL_XqQWOjubgfqINi2jTzg=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ=VSN0OFdZMcwOzWGy-IVLaRlPAH6MtsPxxyMYlqibzrE=IROToNMs5CsGrCism3IEjw-DOnrc3y9N9O7tcBzXnUU=
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Strange problems with Ignite native Persistence when Data exceeds Memory

[dkarachentsev]

Hi Roger,

It looks like not all data could fit in memory and pages started to swap
into disk. This means that too many pages marked as dirty that causes more
often and longer checkpointing, that with WAL writes may significantly
slowdown data uploading. Please attach Ignite and GC logs and try to set
PersistenceStoreConfiguration.setWalMode() to LOG_ONLY or BACKGROUND (this
will reduce disk loading).

Data loss looks strange, is it possible that you have duplicated entry keys?
DataStreamer by default does not override existed entries for performance
reasons.

Thanks!
-Dmitry.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Strange-problems-with-Ignite-native-Persistence-when-Data-exceeds-Memory-tp16187p16194.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Strange problems with Ignite native Persistence when Data exceeds Memory

[Roger Fischer (CW)]

Hello,

I am running into strange problems with Ignite Native Persistence when the data 
exceeds the size of the cache (memory).

I have 3 nodes with 8 GB each, one cache, using 1 backup. I am using SQL 
queries, and the cache has 3 indexes.

I am loading 9.6M objects, using IgniteDataStreamer in a client. The work/db 
directory is about 13 GB on each node. Everything is fine.

Then I am trying to load another 4.8M objects. Here is where the problems start:
1) Loading becomes slow, less than 1000 objects per second.
2) Object are missed (not inserted). There is no indication, neither any logs 
nor any feedback to the client that does the loading.
3) The client that does the loading eventually hangs, waiting on a future. It 
never returns to my code.
4) Another client doing SQL queries eventually hangs also.
5) Restarting clients does not help, they will hang again. Thus deactivating 
the cluster to allow a server restart is not possible.

There are no logs to indicate any problems. No errors or warnings. Only 
periodic metrics logs.

The first time I ran into the problem, each server had 16 GB of RAM, and the 
problem happened when the work/db directory was about 35 GB.

This leads me to believe that the problem is related to the number of objects 
becoming bigger than what the cache can hold.

I am using simple native persistence, no special configuration.





ver. 2.1.0#20170720-sha1:a6ca5c8a

OS: Linux 3.10.0-514.el7.x86_64 amd64

VirtualBox VMs (all on same host), with dynamic disk (one drive on the host)

Roger