Data Recovery after cluster restart

[Naveen]

Hi 
I am using Ignite 2.3

I have enables persistence, using Ignite native persistence.

Here is my config file.
Have created a table thru SQL with the below script

CREATE TABLE ASSOCIATED_PARTIES_NEW(PARTY_ID VARCHAR, ASSOCIATED_PARTY_ID
VARCHAR, WALLETID VARCHAR, UPDATEDDATETIME TIMESTAMP, UPDATEDBY VARCHAR,
PRIMARY KEY (PARTY_ID))WITH
"template=partitioned,backups=1,cache_name=ASSOCIATED_PARTIES_NEW,value_type=com.ril.edif.model.ASSOCIATED_PARTIES";
 

Inserted data into thru DataStreamer API and some thru SQL,  I have loaded
1M+ records, you can see the below count

--
0: jdbc:ignite:thin://127.0.0.1> select count(*) from ASSOCIATED_PARTIES;
++
|COUNT(*)|
++
| 1043186|
++
1 row selected (1.227 seconds)
0: jdbc:ignite:thin://127.0.0.1>
--

However, when I restart the cluster and after activation, when I execute
"!tables" thru SQL, I dont see any tables. I could "work" folder having some
huge files. 

How do we get the data back and whats the procedure, do I need to create the
tables again OR do we have any commands to load the data back to memory from
disk.


--




http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xmlns:util="http://www.springframework.org/schema/util;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
  
http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/util
  
http://www.springframework.org/schema/util/spring-util.xsd;>




  

  

  

  

  










  10.144.114.113:47500..47502
 10.144.114.114:47500..47502
 10.144.114.115:47500..47502








-

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: max size of Ignite cache (not memory size, i mean max amount ofrecords)

[Jeff Jiao]

Thank you guys



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Question on ports exposed in kubernetes setup

[vkulichenko]

Vishwas,

I update the doc and the example. Thanks for pointing this out!

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Correct build process for Ignite

[vkulichenko]

Build works for me (and most likely for everyone else as there are no
complaints), so it looks like your local issue. I would try the following:
- Clean up local Maven repo.
- Run without custom setting.xml
- Run with verbose output to see if Maven provides more details on the
issue.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connecting Amazon cluster with client from local

[vkulichenko]

Looks like you provided AddressResolver on TcpDiscoverySpi, so communication
SPI is not aware of it. Try setting it on IgniteConfiguration instead, that
will make both components to pick it up.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Distributed transaction support in Ingnite

[vkulichenko]

Prasad,

1. In write-behind mode database updates are not transactional. If you want
Ignite to maintain underlying Oracle transaction to make sure cache and DB
are always consistent, you need to use synchronous write-through. As for
compute, you can execute transaction within compute, but you can't do
compute inside a transaction like in your example above. Basically, the
whole transaction must be executed in a single thread, while compute can be
executed even on another node.

2. Load cache process is not triggered automatically, you need to start it
manually. Just call IgniteCache#loadCache method and Ignite will make sure
data is loaded on all nodes.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SELECT Statement cancellation & memory sizing

[Dave Harvey]

Just saw 2.4 release notes: Improved COUNT(*) performance 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.4 status

[Paolo Di Tommaso]



On Thu, Mar 8, 2018 at 9:14 PM, Dmitriy Setrakyan 
wrote:

> Paolo, the release is out for a vote:
> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-
> Apache-Ignite-2-4-0-RC1-td27687.html
>
> D.
>
> On Tue, Feb 20, 2018 at 11:55 AM, Paolo Di Tommaso <
> paolo.ditomm...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I was wondering what's the status of Ignite 2.4. Is there any planned
>> release date?
>>
>> The need to support java 9 is becoming a priority.
>>
>>
>> Cheers,
>> Paolo
>>
>>
>

Fwd: Large durable caches

[Dmitriy Setrakyan]

Hi Lawernce,

I believe Alexey Goncharuk was working on improving this scenario. Alexey,
can you provide some of your findings here?

D.

-- Forwarded message --
From: lawrencefinn 
Date: Mon, Mar 5, 2018 at 1:54 PM
Subject: Re: Large durable caches
To: user@ignite.apache.org


BUMP.  Can anyone verify this?  If ignite cannot scale in this manner that
is
fine, i'd just want to know if what i am seeing makes sense.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Business Intelligence connection to Ignite

[Dmitriy Setrakyan]

Hi Steve,

The integration with Tableau was tested and verified:
https://apacheignite-sql.readme.io/docs/tableau

D.

On Mon, Mar 5, 2018 at 12:48 AM, steve.hostettler <
steve.hostett...@gmail.com> wrote:

> Hello,
>
> is there any best practice/recommendation on how to connect 3rd business
> intelligente tools to Ignite.
> For instance, is it possible to connect a BO universe to ignite?
>
> Thansk for your help
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SSD writes get slower and slower

[Dmitriy Setrakyan]

I also think that switching to LOG_ONLY mode should be good enough.

On Mon, Feb 26, 2018 at 6:58 AM, dmitriy.govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Hi,
>
> I guess the problem in "setSwapPath(...)", it is not the path for
> persistence.
>
> Try to do something like this:
>
> storePath="/data/ignite2/swap/"
>
> cfg.setDataStorageConfiguration(
> new DataStorageConfiguration()
> .setWriteThrottlingEnabled(true)
> .setPageSize(4 * 1024)
> .setStoragePath(storePath)
> .setWalPath(storePath + "/wal")
> .setWalArchivePath(storePath + "/archive")
> .setDefaultDataRegionConfiguration(
> new DataRegionConfiguration()
> .setPersistenceEnabled(true)
> .setInitialSize(2L * 1024 * 1024 * 1024)
> .setMaxSize(10L * 1024 * 1024 * 1024)
> )
> );
>
> Are you sure what you need so strong guaranteed? (WALMode.DEFAULT)
> The WALMode.DEFAULT mode provides the strictness of the protection, but it
> is the slowest onе.
> Use LOG_ONLY mode for data loading. More info about wal mode see:
> https://apacheignite.readme.io/docs/write-ahead-log
>
>
>
>
> On 25.02.2018 12:14, VT wrote:
>
>> Hi Stan,
>>
>> The setting is very simple and straightforward, as follows.
>>
>> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
>> dsCfg.setWalMode(WALMode.DEFAULT);
>> dsCfg.setPageSize(4 * 1024);
>> dsCfg.setWriteThrottlingEnabled(true);
>>
>> ..
>>
>> DataRegionConfiguration regionCfg1 = new DataRegionConfiguration();
>> regionCfg1.setName("region_1");
>> regionCfg1.setInitialSize(2L * 1024 * 1024 * 1024);
>> regionCfg1.setMaxSize(10L * 1024 * 1024 * 1024);
>> regionCfg1.setCheckpointPageBufferSize(2L * 1024 * 1024 * 1024);
>> regionCfg1.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
>> regionCfg1.setPersistenceEnabled(true);
>> regionCfg1.setSwapPath("/data/ignite2/swap/"); //SSD
>>
>> ...
>>
>> cacheCfg.setDataRegionName("region_1");
>> cacheCfg.setName(CacheName);
>> cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizati
>> onMode.FULL_ASYNC);
>> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
>> cacheCfg.setCopyOnRead(false);
>>
>> ...
>>
>> I used DataStreammer very simply, like the following.
>>
>> IgniteDataStreamer stmr.addData(key, value);
>>
>> I have tried multiple settings such as perNodeBufferSize,
>> perNodeParallelOperations. Still very slow. Please help. Thanks!
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: Using 3rd party DB together with native persistence (WAS: GettingInvalid state exception when Persistance is enabled.)

[Dmitriy Setrakyan]

To my knowledge, the 2.4 release should have support for both persistence
mechanisms, native and 3rd party, working together. The release is out for
a vote already:
http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html

D.

On Mon, Feb 26, 2018 at 2:43 AM, Humphrey  wrote:

> I think he means when *write-through* and *read-through* modes are enabled
> on
> the 3rd party store, data might be written/read to/from one of those
> persistence storage (not on both).
>
> So if you save data "A" it might be stored in the 3rd party persistence,
> and
> not in the native. When data "A" is not in the cache it might try to look
> it
> up from the native persistence, where it's not available. Same could happen
> with updates, if "A" was updated to "B" it could have changed in the 3rd
> party but when requesting for the data again you might in one case get "A"
> an other case "B" depending on the stores it reads the data from.
>
> At least that is what I understand from his consistency between both
> stores.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite 2.4 status

[Dmitriy Setrakyan]

Paolo, the release is out for a vote:
http://apache-ignite-developers.2346864.n4.nabble.
com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html

D.

On Tue, Feb 20, 2018 at 11:55 AM, Paolo Di Tommaso <
paolo.ditomm...@gmail.com> wrote:

> Hi folks,
>
> I was wondering what's the status of Ignite 2.4. Is there any planned
> release date?
>
> The need to support java 9 is becoming a priority.
>
>
> Cheers,
> Paolo
>
>

Re: And again... Failed to get page IO instance (page content is corrupted)

[Dmitriy Setrakyan]

Hi Sergey,

The 2.4 release is about to be voted for. You can use the RC1 in the mean
time:
http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html

D.

On Mon, Feb 19, 2018 at 6:43 AM, Mikhail 
wrote:

> Hi Sergey,
>
> The release of 2.4 should be soon, in a week or couple, however, there's no
> strong schedule for Apache releases.
>
> Could you please share a reproducer for the issue? Might be you can share a
> storage on which the issue can be reproduced?
>
> Thanks,
> Mike.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

QueryEntity and inheritence

[ralph]

Hi,

I have some objects that I want to put in an ignite cache and also want to
query them using jdbc sql.
I have used QueryEntity to define the mapping between object model and
relational model.

My problem is :

My objects uses inheritence.
"Asset" object is the parent, and "OTC" object and "Security" Object inherit
from Asset.

Asset
 |-OTC
 |-Security

I would like to have an sql model that is represented with 3 tables
ASSET table which will contain all attributs from Asset Object
OTC table which will contain all specialized attributs from OTC Object
SECURITY table which will contain all specialized attributs from Security
object

And of course all data from 3 tables have to be mapped on the same cache (I
do not want to duplicate data in 2 caches)

Here is what I have done:

assetQueryEntity, OTCQueryEntity, SecurityQueryEntity are QueryEntity
definition for each of the 3 tables :

LinkedList qeList = new LinkedList<>();

qeList .add(assetQueryEntity);
qeList .add(OTCQueryEntity);
qeList .add(SecurityQueryEntity);

CacheConfiguration cacheCfg = new CacheConfiguration<>("AssetCache");
cacheCfg.setQueryEntities(qeList);
IgniteCache cache = ignite.getOrCreateCache(cacheCfg);


The result is that I can see 3 tables with the right definition.
I can select data from OTC and SECURITY tables.
but the Asset table is empty

Is there a way to do that ?

Thanks for your help



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: How to use SqlQuery for multi tables

[Stanislav Lukyanov]

You don’t have to explicitly set sqlSchema for a cache – the default value is 
the cache name.

Both cache1.query() and cache2.query() can access the data in both caches,
the only difference is the default schema name that will be used in the query.

If both caches have the same schema name or if you always specify schema 
explicitly in SQL query,
there is no difference in which one to use.

Thanks,
Stan

From: takumi
Sent: 8 марта 2018 г. 18:02
To: user@ignite.apache.org
Subject: RE: How to use SqlQuery for multi tables

I solved my self.
It was the cause that I didn't set to SqlSchema of CacheConfigration.

Please tell me in connection with this.
I don't know which instance query() method should be called.

If it is an instance of the same schema, whichever one I am using will not
change?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: How to use SqlQuery for multi tables

[takumi]

I solved my self.
It was the cause that I didn't set to SqlSchema of CacheConfigration.

Please tell me in connection with this.
I don't know which instance query() method should be called.

If it is an instance of the same schema, whichever one I am using will not
change?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Expiry Inconsistency with Native Persistence

[Subash Chaturanga]

Ideally 3 nodes.

So expiry inconsistency is a known issue ?

But I  reproduce the expiry inconsistency issue even in a single node with
the steps mentioned earlier.

We are evaluating Ignite vs Redis for our use case these days, and wanted
to make sure everything works fine as in docs. Can you please confirm the
cache eviction bahavior also, like can we put a limit to the persistence
later too ?


On Thu, Mar 8, 2018 at 6:51 AM Dmitry Pavlov  wrote:

> Hi,
>
> There was some well known issue with native persistence and rebalancing
> data between nodes, later I can try to find its Id .
>
> How much nodes do you use?
>
> Sincerely,
> Dmitry Pavlov
>
> ср, 7 мар. 2018 г., 22:25 Subash Chaturanga :
>
>> Hi team,
>> With a cache having CreatedExpiryPolicy for 1min duration and having
>> ignite persistence enabled as per docs. And did this.
>>
>> - cache put and then cache get, gives me the value
>> - wait for 1min
>> - cache get, returns null. Perfectly fine up to now.
>> - then recycled JVM.
>> - now only cache.get, no puts, and return == null is my expectation. But
>> it wasn’t the case. It returned me the value.
>>
>> This means cache expiry doesn’t remove the entry from native persistent
>> layer.  But if you do a cache.remove() it will remove it from native
>> persistent later too. Meaning JVM recycle will return null.
>>
>> So compared to cache.remove working as expected, cache expiry behavior is
>> very inconsistent.
>>
>> Can someone please clarify ?
>>
>> --
>> /subash
>>
> --
/subash

RE: How to use SqlQuery for multi tables

[takumi]

Thank you for your reply.

In addition to this, I want to know how to use SqlFieldsQuery instead of
SqlQuery.
I would like you to teach me about that.

My code is here.

--
public class Main {
private static final String CACHE_NAME1 = "binaryCache1";
private static final String CACHE_NAME2 = "binaryCache2";

public static void main(String[] args) {
IgniteConfiguration cfg = new IgniteConfiguration();

BinaryConfiguration bc = new BinaryConfiguration();
bc.setCompactFooter(false);
cfg.setBinaryConfiguration(bc);

Ignition.start(cfg);
Ignite ignite = Ignition.ignite();

CacheConfiguration cc1 = new
CacheConfiguration(CACHE_NAME1).setStoreKeepBinary(true);
CacheConfiguration cc2 = new
CacheConfiguration(CACHE_NAME2).setStoreKeepBinary(true);

List entities = new ArrayList<>();
QueryEntity qe = new QueryEntity(String.class.getName(),
CACHE_NAME1);
qe.addQueryField("prop1", Double.class.getName(), "prop1");
entities.add(qe);
cc1.setQueryEntities(entities);

List entities2 = new ArrayList<>();
QueryEntity qe2 = new QueryEntity(String.class.getName(),
CACHE_NAME2);
qe2.addQueryField("prop1", Double.class.getName(), "prop1");
entities2.add(qe2);
cc2.setQueryEntities(entities2);

IgniteCache cache1 =
ignite.getOrCreateCache(cc1).withKeepBinary();
IgniteCache cache2 =
ignite.getOrCreateCache(cc2).withKeepBinary();
setData(ignite, cache1, cache2);


QueryCursor> results1 = cache1.query(new
SqlQuery(CACHE_NAME1, "select a.* from " + CACHE_NAME1 + " a"));
for(Entry result : results1) {
System.out.println("result :" +result.getKey() + "," +
result.getValue());
}

QueryCursor> results2 = cache2.query(new
SqlQuery(CACHE_NAME2, "select b.* from " + CACHE_NAME2 + " b"));
for(Entry result : results2) {
System.out.println("result :" +result.getKey() + "," +
result.getValue());
}

// this line throws exception
QueryCursor> results3 = cache1.query(new
SqlQuery(CACHE_NAME1, "select a.* from " + CACHE_NAME1 + " a," + CACHE_NAME2
+ " b where a.prop1 = b.prop1"));
for(Entry result : results3) {
System.out.println("result :" +result.getKey() + "," +
result.getValue());
}

}

private static void setData(Ignite ignite, IgniteCache cache1, IgniteCache cache2) {
cache1.put("device1",
ignite.binary().builder(CACHE_NAME1).setField("prop1", 50.0,
Double.class).build());
cache1.put("device2", ignite.binary().builder(CACHE_NAME1)
.setField("prop1", 10.0, Double.class).build());
cache2.put("device1", ignite.binary().builder(CACHE_NAME2)
.setField("prop1",  1.0, Double.class).build());
cache2.put("device2", ignite.binary().builder(CACHE_NAME2)
.setField("prop1", 10.0, Double.class).build());
}
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Segmentation policy configuration

[Stanislav Lukyanov]

Hi,

It’s IgniteConfiguration.segmentationPolicy property.

However, note that Apache Ignite doesn’t provide standard mechanisms for 
network segmentation detection.
By default, a node will only be moved to detected state when it is explicitly 
notified by coordinator that it had failed and was kicked out of the cluster.
If you’re interested in commercial vendors that provide custom Ignite features, 
please see refer to this page 
https://cwiki.apache.org/confluence/display/IGNITE/Third+Party+Support+and+Employment.

Thanks,
Stan

From: Hemasundara Rao
Sent: 8 марта 2018 г. 13:46
To: user@ignite.apache.org
Subject: Segmentation policy configuration

Hi, 
 Could you someone please help me in how to do "Segmentation policy" at 
configuration level?

Regards,
Hemasundar.

RE: How to use SqlQuery for multi tables

[Stanislav Lukyanov]

Hi,

When you create a cache via Java API you provide a value type to the 
QueryEntity.
Then the table name for SQL queries will have a form 
..

Can you share the code that you use to create the caches?
Then it would be easier to tell what will be the exact names of the tables.

Thanks,
Stan


From: takumi
Sent: 7 марта 2018 г. 18:40
To: user@ignite.apache.org
Subject: How to use SqlQuery for multi tables

I want to use sql "select" from 2 caches.

For example.
  SqlQuery sql = new SqlQuery(cache1, "select a.* from cache1 a, cache2 b
where a.id = b.id");
  QueryCursor> results = cache.query(sql);

  ※ cache1, cache2 have BinaryObject Data.
  I call QueryEntity#addQueryField("id", String.class.getName(), "id").

When above-mentioned sql execute, exception occured. It is reason cache2 is
not found.
SqlQuery has the 1st parameter which is target cache' name (=cache1).
So cache2 is not found.
How I use sql from 2 tables (cache1 and cache2)？




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Hortonworks HDP Installation without IGFS

[Evgenii Zhuravlev]

Sorry, wrong link: https://apacheignite-fs.readme.io/docs/map-reduce

Evgenii

2018-03-08 13:17 GMT+03:00 Evgenii Zhuravlev :

> Hi,
>
> I think here is the documentation you're looking for:
> http://apache-ignite-users.70518.x6.nabble.com/
> Hortonworks-HDP-Installation-without-IGFS-td20059.html
>
> Evgenii
>
> 2018-02-15 11:07 GMT+03:00 Sparsh Singhal :
>
>> I have a 10 node cluster with 1 edgenode (for ambari), 1 namenode and 8
>> datanode. I want to integrate Ignite for HDP and for Yarn without IGFS
>> (mainly for In-Memory MapReduce).
>>
>> I set up everything using official docs on namenode, and started the
>> ignite
>> node. It blocks the shell and then I couldn't do anything.
>>
>> Can someone provide me a step by step guide to integrate Ignite with my
>> cluster.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: Ignite Expiry Inconsistency with Native Persistence

[Dmitry Pavlov]

Hi,

There was some well known issue with native persistence and rebalancing
data between nodes, later I can try to find its Id .

How much nodes do you use?

Sincerely,
Dmitry Pavlov

ср, 7 мар. 2018 г., 22:25 Subash Chaturanga :

> Hi team,
> With a cache having CreatedExpiryPolicy for 1min duration and having
> ignite persistence enabled as per docs. And did this.
>
> - cache put and then cache get, gives me the value
> - wait for 1min
> - cache get, returns null. Perfectly fine up to now.
> - then recycled JVM.
> - now only cache.get, no puts, and return == null is my expectation. But
> it wasn’t the case. It returned me the value.
>
> This means cache expiry doesn’t remove the entry from native persistent
> layer.  But if you do a cache.remove() it will remove it from native
> persistent later too. Meaning JVM recycle will return null.
>
> So compared to cache.remove working as expected, cache expiry behavior is
> very inconsistent.
>
> Can someone please clarify ?
>
> --
> /subash
>

Segmentation policy configuration

[Hemasundara Rao]

Hi,
 Could you someone please help me in how to do "Segmentation policy" at
configuration level?

Regards,
Hemasundar.