Ignite TcpDiscoveryS3IpFinder and AWS VPC

[smovva]

I have a cluster running on AWS VPC with TcpDiscoveryS3IpFinder SPI. I
noticed that there are a couple of  entries in the S3 bucket
'127.0.0.1#47500', '0:0:0:0:0:0:0:1%lo#47500' along with entries for the
private IPs of the nodes. If anyone understands what these are, can you
please share the info.

Also, I have another VPC that I want to run some clients from. I was
wondering if there is any documentation around setting up VPC peering and
address translation for this to work.

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite data can't be recovered after node fail

[smovva]

Where you able to resolve this? I'm in a very similar situation.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Uneven partitioning and data not on off-heap memory.

[smovva]

Thanks Denis. That makes sense.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Uneven partitioning and data not on off-heap memory.

[smovva]

> 2. By default all entries are saved off heap (not disk, heap outside the 
Java heap to avoid GC problems), you can configure it to use heap or off 
heap memory as you want. 

This is the output from visor. I'm not completely sure what the difference
between off-heap and off-heap memory is here. So, I was assuming that
"off-heap" is disk and "off-heap memory" is what's in off-heap RAM.

 Total: 419490 
   Heap: 0 
   Off-Heap: 419490 
   Off-Heap Memory: 0 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Uneven partitioning and data not on off-heap memory.

[smovva]

I have started my cluster and pushed some data into a cache. Cluster details
are provided below. The cache has persistence enabled.
I see a couple of issues with how data is cached.
1. Out of 8 nodes only 2 nodes have some entries. I ran all my keys through
Murmur3 hash and they seemed to be evenly distributed. 
2. On the nodes that have data, I only see entires in off-heap (Which I
assume is disk). I have 0 entries in heap and 0 in heap memory. Do entries
only get into off-heap memory when they are read?
This is the output from visor for a node:

Total: 419490  
 Heap: 0
 Off-Heap: 419490
 Off-Heap Memory: 0


My cluster consists of 8 servers running one node each. Each server has 32GB
RAM and 4 Cores. 
All the nodes are started like this 
= 
bin/ignite.sh -J-Xmx3g config/dev-cluster-config.xml 
= 

My cache consists of   
  - key: strings of length 9 chars 
  - Value: POJOs that have some metadata and one large array of integers 
(8800 values) 

I have 4 data streamers and as they start pushing data, I see some server 
nodes throwing OutOfMemoryErrors and the data streamers hangup. 

The streamer code is here. 

Cache configuration:
== 
CacheConfiguration cacheCfg = new
CacheConfiguration<>(DATA_CACHE_NAME); 
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC); 
cacheCfg.setCacheMode(CacheMode.PARTITIONED); 
cacheCfg.setBackups(2); 

   
cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC); 
cacheCfg.setIndexedTypes(String.class, Tile.class); 

== 

Cluster-config.xml :

  
 

   
 
 
 
 
 
 
 
 
 
 

  
 
 
 
 
 

= 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Deadlock during cache loading

[smovva]

I have a fairly similar setup. What type of EC2 instances are you using? Just
for compare my setup.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[smovva]

Thanks again.

I have one more question. After pushing some data, I see only two nodes
having entries out of 8. All the others have 0 entries. 
My keys are 9 char long strings with a common prefix (all have the same
first 3 chars). Wondering why the keys are not evenly distributed. 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[smovva]

breischl, Thanks for the quick response. 
I'm not sure why the node stats show

Non-heap memory maximum | 744mb

Each server node has 32GB and I've assigned 3g for heap. Shouldn't this be
16GB since that's what I set the data region size to? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

OutOfMemoryError while streaming

[smovva]

I'm seeing OutOfMemoryError when I have multiple data streamers pushing data
into the cluster.
My cluster consists of 8 servers running one node each. Each server has 32GB
RAM and 4 Cores.
All the nodes are started like this
=
bin/ignite.sh -J-Xmx3g config/dev-cluster-config.xml
=

My cache consists of  
  - key: strings of length 9 chars 
  - Value: POJOs that have some metadata and one large array of integers
(8800 values)

I have 4 data streamers and as they start pushing data, I see some server
nodes throwing OutOfMemoryErrors and the data streamers hangup. 

The streamer code is here.

==
CacheConfiguration cacheCfg = new
CacheConfiguration<>(DATA_CACHE_NAME);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(2);
   
cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheCfg.setIndexedTypes(String.class, Tile.class);

IgniteCache viewpointCache =
ignite.getOrCreateCache(cacheCfg);

try (IgniteDataStreamer streamer =
ignite.dataStreamer(viewpointCache.getName())) {
Path sourceFile = Paths.get(hashIndexFile);
try(BufferedReader reader = Files.newBufferedReader(sourceFile,
Charset.forName("UTF-8")))
{
String currentLine = null;
while((currentLine = reader.readLine()) != null){
String hash = currentLine.trim();
streamer.addData(hash, getTileFor(hash));
}
}
}
==

Cluster-config.xml

 

   













 










=

Also, I noticed that there is nothing in off-heap memory. Here is the cache
stats for one node.
Total: 513278 
  Heap: 5  
  Off-Heap: 513273
  Off-Heap Memory: 0
and config also shows 
| Non-heap memory initialized | 2mb 
| Non-heap memory used| 86mb  
| Non-heap memory committed   | 88mb  
| Non-heap memory maximum | 744mb
How do I make it use and store more in Non-heap memory?

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/