Re: Affinity calls in stream receiver

[David Harvey]

We are testing whether  removing the nested classes helps things, and if
so, will create a reproducer.

   1. IGNITE-7905 <https://issues.apache.org/jira/browse/IGNITE-7905> is
   the issue where ignite.active(true) fails from a client if userVersion is
   non-zero, which seems to be due to nested classes.


On Tue, Jul 3, 2018, 4:20 AM Denis Mekhanikov  wrote:

> David,
>
> So, the problem is that the same class is loaded multiple times and it
> wastes the metaspace, right?
> Could you share a reproducer?
>
> Denis
>
> вт, 3 июл. 2018 г. в 0:58, David Harvey :
>
>> We have a custom stream receiver that makes affinity calls. This all
>> functions properly, but we see a very large number of the following
>> messages for the same two  classes.   We also just tripped a 2GB limit on
>> Metaspace size, which we came close to in the past.
>>
>> [18:41:50,365][INFO][pub-#6954%GridGainTrial%][GridDeploymentPerVersionStore]
>> Class was deployed in SHARED or CONTINUOUS mode: class
>> com.IgniteCallable
>>
>> So these affinity calls need to load classes that where loaded from
>> client nodes, which may be related to why this happening, but my primary
>> suspect is the fact that both classes are nested.  ( I had previously hit
>> an issue where setting the peer-class-loading "userVersion" would cause
>> ignite to thrown exceptions when the client node attempted to activate the
>> cluster.In that case, the Ignite call into the cluster was also using a
>> nested class. )
>>
>> We will try flattening these classes to see if the problem goes away.
>>
>>
>> *Disclaimer*
>>
>> The information contained in this communication from the sender is
>> confidential. It is intended solely for use by the recipient and others
>> authorized to receive it. If you are not the recipient, you are hereby
>> notified that any disclosure, copying, distribution or taking action in
>> relation of the contents of this information is strictly prohibited and may
>> be unlawful.
>>
>> This email has been scanned for viruses and malware, and may have been
>> automatically archived by *Mimecast Ltd*, an innovator in Software as a
>> Service (SaaS) for business. Providing a *safer* and *more useful* place
>> for your human generated data. Specializing in; Security, archiving and
>> compliance. To find out more Click Here
>> <http://www.mimecast.com/products/>.
>>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Affinity calls in stream receiver

[David Harvey]

We have a custom stream receiver that makes affinity calls. This all
functions properly, but we see a very large number of the following
messages for the same two  classes.   We also just tripped a 2GB limit on
Metaspace size, which we came close to in the past.

[18:41:50,365][INFO][pub-#6954%GridGainTrial%][GridDeploymentPerVersionStore]
Class was deployed in SHARED or CONTINUOUS mode: class
com.IgniteCallable

So these affinity calls need to load classes that where loaded from client
nodes, which may be related to why this happening, but my primary suspect
is the fact that both classes are nested.  ( I had previously hit an issue
where setting the peer-class-loading "userVersion" would cause ignite to
thrown exceptions when the client node attempted to activate the cluster.
  In that case, the Ignite call into the cluster was also using a nested
class. )

We will try flattening these classes to see if the problem goes away.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Deadlock during cache loading

[David Harvey]

Denis does have a point.   When we were trying to run using GP2 storage,
the cluster would simply lock up for an hour.   Once we moved to local SSDs
on i3 instances those issues went away (but we needed 2.5 to have the
streaming rate hold for up as we had a lot of data loaded).   The i3
instances are rated at about 700,000 write IOPS, and we were only getting
about 20-30,000 out of GP2.   You could separate or combine the WAL and
storage, and hardly move the needle.
Will describe cluster snapshots on AWS in more detail when we have
completed that work.



On Mon, Jul 2, 2018 at 11:20 AM, Denis Mekhanikov 
wrote:

> Why did you decide, that cluster is deadlocked in the first place?
>
> > We've had several deployments in a row fail, apparently due to
> deadlocking in the loading process.
> What did you see in logs of the failing nodes?
>
> Denis
>
> пн, 2 июл. 2018 г. в 17:08, breischl :
>
>> Ah, I had not thought of that, thanks.
>>
>> Interestingly, going to a smaller cluster seems to have worked around the
>> problem. We were running a 44-node cluster using 3 backups of the data.
>> Switching to two separate 22-node clusters, each with 1 backup, seems to
>> work just fine. Is there some limit to how large a cluster should be?
>>
>> @smovva - We were using c4.4xl instances, but switched to r4.2xl because
>> we
>> had spare CPU but kept having memory problems. I suspect that there isn't
>> a
>> "right" size to use, it just depends on the use case you have.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Deadlock during cache loading

[David Harvey]

transactions are easy to use: see  examples,  org.apache.ignite.
examples.datagrid.store.auto
We use them in the stream receiver.You simply bracket the get/put in
the transaction, but use a timeout, then bracket that with an "until done"
while loop, perhaps added a sleep to backoff.
We ended up with better performance with PESSIMISTIC transactions, though
we expected OPTIMISTIC to win.

My guess would be the DataStreamer is not a fundamental contributor to the
deadlock you are seeing, and you may have discovered an ignite bug.



On Sun, Jul 1, 2018 at 11:44 AM, breischl  wrote:

> @DaveHarvey, I'll look at that tomorrow. Seems potentially complicated, but
> if that's what has to happen we'll figure it out.
>
> Interestingly, cutting the cluster to half as many nodes (by reducing the
> number of backups) seems to have resolved the issue. Is there a guideline
> for how large a cluster should be?
>
> We were running a single 44-node cluster, with 3 data backups (4 total
> copies) and hitting the issue consistently. I switched to running two
> separate clusters, each with 22 nodes using 1 data backup (2 total copies).
> The smaller clusters seem to work perfectly every time, though I haven't
> tried them as much.
>
>
> @smovva - We're still actively experimenting with instance and cluster
> sizing. We were running on c4.4xl instances. However we were barely using
> the CPUs, but consistently have memory issues (using a 20GB heap, plus a
> bit
> of off-heap). We just switched to r4.2xl instances which is working better
> for us so far, and is a bit cheaper. However I would imagine that the
> optimal size depends on your use case - it's basically a tradeoff between
> the memory, CPU, networking and operational cost requirements of your use
> case.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Deadlock during cache loading

[David Harvey]

You can start a transaction in the stream receiver to make it atomic.

On Fri, Jun 29, 2018, 1:02 PM breischl  wrote:

> StreamTransformer does an invoke() pretty much exactly like what I'm doing,
> so that would not seem to change anything.
>
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/stream/StreamTransformer.java#L50-L53
>
>
>
> I may try using a put(), but since I need to compare the existing cache
> value, I'd need to get(), compare, then put(). I thought that may open up a
> potential race condition, if two different updates happen close to each
> other they could each do get(), then each do put(), but potentially in the
> wrong order. Unless there's some locking I don't understand there?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Re-post: java.io.IOException: Too many open files

[David Harvey]

MYou must increase the Linux NOFILE ulimit when running Ignite.  The
documentation describes how to do this.

On Sun, Jun 24, 2018, 12:47 PM 胡海麟  wrote:

> Hi,
>
> Re-post message 'cause I failed to post my logs pasted.
>
> I have got repeated Too many open files exceptions since sometime.
> 
> [11:26:24,493][SEVERE][grid-nio-worker-tcp-rest-1-#57][GridTcpRestProtocol]
> Failed to process selector key [ses=GridSelectorNioSessionImpl
> [worker=ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null,
> finished=false, hashCode=1611196193, interrupted=false,
> runner=grid-nio-worker-tcp-rest-1-#57]]], writeBuf=null, readBuf=null,
> inRecovery=null, outRecovery=null, super=GridNioSessionImpl
> [locAddr=/10.1.14.11:11211, rmtAddr=/10.1.252.184:40680,
> createTime=1529666783471, closeTime=0, bytesSent=5, bytesRcvd=1074,
> bytesSent0=0, bytesRcvd0=0, sndSchedTime=1529666783481,
> lastSndTime=1529666783481, lastRcvTime=1529666783481,
> readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter
> [parser=GridTcpRestParser [marsh=JdkMarshaller
> [clsFilter=o.a.i.i.IgniteKernal$5@331b0c4a], routerClient=false],
> directMode=false]], accepted=true]]]
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
>
> [11:26:24,493][WARNING][grid-nio-worker-tcp-rest-1-#57][GridTcpRestProtocol]
> Closing NIO session because of unhandled exception [cls=class
> o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
>
> [11:26:24,493][WARNING][grid-nio-worker-tcp-rest-1-#57][GridTcpRestProtocol]
> Closed client session due to exception [ses=GridSelectorNioSessionImpl
> [worker=ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null,
> finished=false, hashCode=1611196193, interrupted=false,
> runner=grid-nio-worker-tcp-rest-1-#57]]], writeBuf=null, readBuf=null,
> inRecovery=null, outRecovery=null, super=GridNioSessionImpl
> [locAddr=/10.1.14.11:11211, rmtAddr=/10.1.252.184:40680,
> createTime=1529666783471, closeTime=1529666784488, bytesSent=5,
> bytesRcvd=1074, bytesSent0=0, bytesRcvd0=0,
> sndSchedTime=1529666783481, lastSndTime=1529666783481,
> lastRcvTime=1529666783481, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter
> [parser=GridTcpRestParser [marsh=JdkMarshaller
> [clsFilter=o.a.i.i.IgniteKernal$5@331b0c4a], routerClient=false],
> directMode=false]], accepted=true]], msg=Connection reset by peer]
> [11:26:24,513][SEVERE][grid-nio-worker-tcp-rest-1-#57][GridTcpRestProtocol]
> Caught unhandled exception in NIO worker thread (restart the node).
> java.lang.NullPointerException
> at
> sun.nio.ch.EPollArrayWrapper.isEventsHighKilled(EPollArrayWrapper.java:174)
> at
> sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:190)
> at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:239)
> at
> sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:178)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:132)
> at
> java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.register(GridNioServer.java:2545)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1934)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at 

Re: Ignite 2.5 uncatched BufferUnderflowException while reading WAL on startup

[David Harvey]

Is https://issues.apache.org/jira/browse/IGNITE-8780 a regression in 2.5 ?

On Thu, Jun 14, 2018 at 7:03 AM, Pavel Kovalenko  wrote:

> DocDVZ,
>
> Most probably you faced with the following issue https://issues.apache.
> org/jira/browse/IGNITE-8780.
> You can try to remove END file marker, in this case node will be recovered
> using WAL.
>
> чт, 14 июн. 2018 г. в 12:00, DocDVZ :
>
>> As i see, last checkpoint-end file, that invoked the problem, was created,
>> but not filled with data:
>>
>> /opt/penguin/ignite/apache-ignite-fabric-2.5.0-bin/work/
>> db/node00-203cc00d-0935-450d-acc9-d59cc3d2163d/cp$
>> ls -lah
>> total 208K
>> drwxr-xr-x 2 penguin penguin 4.0K Jun  9 12:52 .
>> drwxr-xr-x 3 penguin root4.0K Jun  9 14:27 ..
>> <...>
>> -rw-r--r-- 1 penguin penguin   16 Jun  9 12:49
>> 1528537756580-node-started.bin
>> -rw-r--r-- 1 penguin penguin0 Jun  9 12:52
>> 1528537928225-a5e78b9c-26c8-4dc1-b554-2ee35f119f0a-END.bin
>> -rw-r--r-- 1 penguin penguin   16 Jun  9 12:52
>> 1528537928225-a5e78b9c-26c8-4dc1-b554-2ee35f119f0a-START.bin
>>
>> what state does ignite checkpointing have in that moment? Is it save for
>> node persistent data to remove that empty file?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: ignite peer class loading query

[David Harvey]

Certain kinds of requests (e.g., remote calls) carry enough information for
the receiver of the message to do peer class loading.   The main purpose of
peer class loading is to avoid the need to restart the server to install
software, and the peer class loading is therefore done by the server
requesting the class from the client who sent the request.

It is not targeted at what you are trying to do.

On Thu, Jun 7, 2018 at 10:28 AM, vbm  wrote:

> Hi,
>
> I have a Ignite server running with 3rd partyDB (MYSQL). I have the below
> configuration in xml file:
>
>  class="org.springframework.jdbc.datasource.DriverManagerDataSource">
>   
>
>   
>   
> 
>
> 
> * *
>   ...
> 
> 
>
> On the server the jar for mysql jdbc driver is present iand the servers are
> coming up successfully.
> I have placed the mysql jdbc driver in a seperate folder and added it to
> the
> path using USER_LIBS env.
>
>
> Now if I try to start a client, with the same the xml (only extra thing is
> the client mode part), the client ignite node doesn't start and throws
> below
> error.
>
> * org.springframework.beans.MethodInvocationException: Property
> 'driverClassName' threw exception; nested exception is
> java.lang.IllegalStateException: Could not load JDBC driver class
> [com.mysql.jdbc.Driver]*
>
> As peer class loading is enabled, I expect the client to get the class from
> the server. Why is this not happening ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Data Region Concurrency

[David Harvey]

We have a 2.4 ignite peristence cluster on AWS with 8 i3.xlarge nodes,
where the default memory region size is 160GB.When loading a large
amount of data through a custom StreamReceiver, over time we see the
throughput decrease, as well as network and SSD I/O, and one node with
higher CPU usage than the others, but still only 33%.   Looking at that
node, I see (kill -3) that out of 27 sys-stripe* threads, 21 are blocked on
locks, 17 of which are memory segment locks.  The running threads have the
following call stacks:  acquirePage -> get/put ->getKey/Putkey .  The 2.4
documentation says I should have 32 (one per CPU) segments.I wouldn't
expect such high contention if there was random distribution of load across
the segments.I'm only using the default region.Is there a
explaination for such a high contention level.

It looks as if I can set the number of segments higher by configuring the
data region

setConcurrencyLevel().   The 2.0 documentation for memory configuration
said the default was 4 * # CPUS, but the 2.4 Data Region code seems to
default to # CPUs, and we are only using default region.What would be a
reasonable value?


It looks like the current data region settings get communicated to visor,
but I do not see a way to actually display them.   I'm I missing something?


If I change the xml configuration, it is unclear to me whether this will be
applied when I restart the node or is that setting only used when the data
region is originally created.   (As a general documentation thing, it would
be helpful to identify when settings will actually be applied. )

Thanks,
-DH

PS. Reading the code,   my first thought was that locks were configured as
fair, which ensures order but increases lock held time, but the default
seems to be unfair, which is good.   Fair  locks have the characteristic of
allowing high throughput until contended, then locking in low throughput
until the load completely stops.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Distribution of backups

[David Harvey]

We built a cluster with 8 nodes using Ignite persistence and 1 backup, and
had two nodes fail at different times, the first being storage did not get
mounted and ran out of space early, and the second an SSD failed.   There
are some things that we could have done better, but this event brings up
the question of how backups are distributed.

There are two approaches that have substantially different behavior on
double faults, and double faults are more likely at scale.

1) random placement of backup partitions relative to primary
2) backup partitions have similar affinity to the primary partitions, where
in the extreme nodes are paired so that primaries on the node pair have
backups on the other node of the pair

With a 64 node cluster, #2 would have 1/63th of the likelihood of data loss
when 2 nodes fail vs #1.

I'm guessing that ignite ships with #1, but we could provide our own
affinity function which would accomplish #2 if we chose?

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Running Ignite/GridGain in AWS

[David Harvey]

The biggest thing for us using Ignite Persistence on AWS was finding that
we needed to use i3 instances for the SSD write performance.   Ignite
persistences and GP2 was a poor experience.



On Tue, May 15, 2018 at 2:43 AM, John Gardner 
wrote:

> We've been running the GridGain version of Ignite 8.3.4 within AWS for a
> couple of months now, we're still finding little issues here and there, so
> I was wondering if anyone has any advice, tips or hints specifically when
> running this within an AWS environment.  Have you had any problems with
> Network Segmentation?  How are the cluster nodes working within
> Auto-Scaling Groups? How does Ignite cope with dynamic scaling out?  If you
> have anything like that, it would be great to hear from anyone.
>
> Thanks and Regards
>
> John
>
> --
>
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: slow restart of ignite (with persistence enabled)

[David Harvey]

You have deactivated the cluster, e.g., using visor?

On Mon, May 7, 2018 at 9:08 PM, Randall Harmon 
wrote:

>
> What is the recommended way to signal Ignite to shutdown cleanly, so that
> WAL file doesn't have to be replayed (?) when Ignite restarts?  We are
> loading a large set of data from multiple tables, and we are also
> restarting ignite occasionally (new JARs for job running as ignite
> service).
>
> During this restart, even a single node cluster won't serve cmd=metadata
> on its api port until the WAL scanning(?) process is complete.
>
> (we've experience a similar situation with in-memory storage and clustered
> operation, but one thing at a time...)
>
> During the restart, iostat shows mostly lots of small reads:
>
> > sdb 422.00 4140.00 0.00 4140 0
>
> with occasional tiny writes:
>
> > sdb 462.00 4512.00 128.00 4512 128
>
>
> I'm looking for a way to have Ignite shut down cleanly, or another
> configuration option so that normal restart (SIGINT/SIGTERM) does not cause
> a long restart scenario.   Here's one case that shows 19 minutes of pause
> during a restart.  I've seen it take 40 to 60 minutes too (when 2 or 3
> times as much data has been loaded through it over the course of weeks).
>
> @40005aeca93c09d3481c 14:40:50.160 [ker-#42] WARN
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
> - Logging at INFO level without checking if INFO level is enabled: Found
> last checkpoint marker [cpId=bcffa2cb-803c-4378-bf52-74b7dedf6285,
> pos=FileWALPointer [idx=658, fileOff=45873219, len=281]]
>
> @40005aecadb123dbc964 14:59:51.600 [rker-#3] WARN
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor -
> Logging at INFO level without checking if INFO level is enabled: Received
> state change finish message: true
>
> Any help suggested so I can restart Ignite when I need to, and have it
> start responding in a reasonable amount of time?
>
> 
> 
> 
> 
>  value="/srv/var/ignite-data/wal/archive" />
>
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
> 
> 
> 
> 
> 
> 
>
>
>
> Thanks,
> R
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: IgniteDataStreamer.flush() returns before all futures are completed

[David Harvey]

"Could you please refine what kind of flaw do you suspect? "
I think the basic flaw is that it is unclear how to write a simple task to
stream in some data, and then confirm that all of that data was
successfully  stored before taking an action to record that.   This may be
simply a documentation issue for flush(), where I can't tell what the
intended design would be for such code.

We ran into this issue because we assumed that we needed to test the status
of all of the futures returned by addData, and we found that the listener
was not always called before flush() returned.

As I dig deeper into the code, I see an attempt to cause flush() to throw
an exception if any exception was thrown on the server for *any* prior
record.   If that is the intent (which is not stated but would be
convenient),  then I think there is a race:

   - DataStreamerImpl.flush() calls EnterBusy while activeFuts is non
   empty.  This seems to be the last test of "cancelled".   If there were
   failed requests before this, flush() would throw.
   - Before doFlush() looks at activeFuts, activeFuts becomes empty,
   because the requests failed.
   - flush() returns without throwing an exception.


-DH


On Thu, Apr 26, 2018 at 10:23 AM, Andrey Kuznetsov <stku...@gmail.com>
wrote:

> Hi, David,
>
> As far as a can see from streamer implementation, flush() call
> 1. won't return until all futures returned by addData() are (successfully)
> finished
> 2. will throw an exception if at least one future has been failed.
>
> Could you please refine what kind of flaw do you suspect?
>
> As for callback added by listen(), it is called right _after_ the future
> gets to 'done' state, thus flush() completion does not guarantee callback's
> apply() completion.
>
>
> 2018-04-26 16:37 GMT+03:00 David Harvey <dhar...@jobcase.com>:
>
>> Thanks Yakov.   I think this is more subtle.
>>
>> Our loading via IgniteDatastreamer is idempotent, but this depends on
>> being certain that a batch of work has successfully completed.   It is
>> *not* sufficient for us to listen to the futures returned by addData,
>> then to call flush(), and then to record success if there have been no
>> exceptions.  We must wait until get() is called on every future before
>> recording that the operation was successful.The fact that the future is
>> done is not sufficient, we need to know that it is done and there is no
>> exception.   We can call flush and then do a future.get() on the incomplete
>> futures, but not as an assert.   (It is valid to assert that fut.isDone(),
>> but that is not sufficient.)
>>
>> Based on by current understanding, I think this is a flaw in Ignite, even
>> if the fix might only be to clarify the comments for flush() to make this
>> behavior clear.
>>
>>
>>
> --
> Best regards,
>   Andrey Kuznetsov.
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: IgniteDataStreamer.flush() returns before all futures are completed

[David Harvey]

Thanks Yakov.   I think this is more subtle.

Our loading via IgniteDatastreamer is idempotent, but this depends on being
certain that a batch of work has successfully completed.   It is *not*
sufficient for us to listen to the futures returned by addData, then to
call flush(), and then to record success if there have been no exceptions.
We must wait until get() is called on every future before recording that
the operation was successful.The fact that the future is done is not
sufficient, we need to know that it is done and there is no exception.   We
can call flush and then do a future.get() on the incomplete futures, but
not as an assert.   (It is valid to assert that fut.isDone(), but that is
not sufficient.)

Based on by current understanding, I think this is a flaw in Ignite, even
if the fix might only be to clarify the comments for flush() to make this
behavior clear.

-DH

On Wed, Apr 25, 2018 at 5:43 PM, Yakov Zhdanov <yzhda...@apache.org> wrote:

> David, can you try adding all the futures to a collection and change
> assert condition to check that for all futures in that collection
> fut.isDone() is true?
>
> This should be a proper assertion. Please let me know if it works.
>
> Yakov Zhdanov
>
> чт, 26 апр. 2018 г., 0:30 David Harvey <dhar...@jobcase.com>:
>
>> We had assumed that if we did, in one thread:
>>   IgniteFuture f = streamer.addData(...);
>>   f.listen(...)
>>
>>   streamer.flush();
>>
>>   assert( all the prior futures have completed);   << this triggered.
>>
>>
>>
>> I can't determine if this a bug, or just that the description could use
>> improvement.
>>
>> Inspecting the code it looks like the thread issuing the flush can be
>> awoken prior to the apply function for the addData future is called, since
>> GridFutureAdapter.unblockAll() will unpark or notify in a arbitrary
>> order.
>>
>>
>> *Disclaimer*
>>
>> The information contained in this communication from the sender is
>> confidential. It is intended solely for use by the recipient and others
>> authorized to receive it. If you are not the recipient, you are hereby
>> notified that any disclosure, copying, distribution or taking action in
>> relation of the contents of this information is strictly prohibited and may
>> be unlawful.
>>
>> This email has been scanned for viruses and malware, and may have been
>> automatically archived by *Mimecast Ltd*, an innovator in Software as a
>> Service (SaaS) for business. Providing a *safer* and *more useful* place
>> for your human generated data. Specializing in; Security, archiving and
>> compliance. To find out more Click Here
>> <http://www.mimecast.com/products/>.
>>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

IgniteDataStreamer.flush() returns before all futures are completed

[David Harvey]

We had assumed that if we did, in one thread:
  IgniteFuture f = streamer.addData(...);
  f.listen(...)

  streamer.flush();

  assert( all the prior futures have completed);   << this triggered.



I can't determine if this a bug, or just that the description could use
improvement.

Inspecting the code it looks like the thread issuing the flush can be
awoken prior to the apply function for the addData future is called,
since GridFutureAdapter.unblockAll()
will unpark or notify in a arbitrary order.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Ignite visor command issue on K8s cluster

[David Harvey]

I've had problems with the visor cache command failing when a client is
connected, which was related to TCP port firewalls.   Visor was trying to
make a connection that was blocked.  I don't remember whether top failed.


On Mon, Apr 9, 2018 at 7:10 AM, vbm  wrote:

> Hi Dave
>
> In the ignite pod logs of the server, I see that the cluster is formed.
> I see the below trace when I run the command,
> [02:53:18] Topology snapshot [ver=4, servers=4, clients=0, CPUs=8,
> heap=6.0GB]
>
> This indicates the cluster of 4 nodes has formed.
>
> When I login to the container and try to run the ignite visor, it says
> connected. But when I tried some commands like top, cache. They were not
> returning any result.
>
>
> I got a link in stack overflow on the usage of visor in K8s
> https://stackoverflow.com/questions/45197520/running-the-
> ignite-visor-in-kubernetes
>
> So as per this, I created a new pod and as part of docker run command I
> gave
> the config file for ignite visor. With this it was able to connect to the
> cluster.
>
> Is this the right approach in K8s environment ? If this is correct, how do
> I
> take care of different input from user to the ignite visor to this pod ?
>
>
> Regards,
> Vishwas
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Ignite visor command issue on K8s cluster

[David Harvey]

You have 4 nodes running, but are they part of a single cluster?   If you
look at the log file for one of the server nodes, has it connected to 4
servers or only one?

PS.  I've found this avoids the need to look up the URL "bin/ignitevisor.sh
-cfg=$CONFIG_URI".

On Sat, Apr 7, 2018 at 8:40 AM, vbm  wrote:

> Hi,
>
> I have deployed 4 pods (ignite server) on a  K8s cluster. I am using ignite
> 2.3 docker image.
> I have followed the steps from the  link
>   .
>
> Inside the container, I can see the ignite process running with the config
> file:
>
> root5412  0 11:20 ?00:00:34 /docker-java-home/bin/java
> -Xms512m -Xmx1536m -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -DIGNITE_QUIET=true
> -DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-fabric-2.
> 3.0-bin/work/ignite_success_22e88558-2de7-4b54-b0c4-e958859baa57
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49112
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -DIGNITE_HOME=/opt/ignite/apache-ignite-fabric-2.3.0-bin
> -DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-fabric-2.
> 3.0-bin/bin/ignite.sh
> -cp
> /opt/ignite/apache-ignite-fabric-2.3.0-bin/libs/*:/opt/
> ignite/apache-ignite-fabric-2.3.0-bin/libs/ignite-indexing/*
> :/opt/ignite/apache-ignite-fabric-2.3.0-bin/libs/ignite-
> spring/*:/opt/ignite/apache-ignite-fabric-2.3.0-bin/libs/licenses/*
> org.apache.ignite.startup.cmdline.CommandLineStartup
> *file:tmp/ignite/ignite-config.xml*
> root   866   271  0 12:32 ?00:00:00 grep ignite
>
>
> From inside the container, when I try to connect using same config file in
> the ignite visor console.
>
> Command to connect to the cluster:
> visor> open -cpath=/tmp/ignite/ignite-config.xml
>
> +--+
> | Status   | Connected |
> | Ignite instance name |  |
> | Config path  | /tmp/ignite/ignite-config.xml |
> | Uptime   | 00:00:00  |
> +--+
> visor>
> *visor> node
> (wrn) : Topology is empty.*
>
> When I do a curl request using below command, I get the list of topology as
> expected.
> curl 'http://10.254.61.219:8080/ignite?cmd=top' | jq
>
> I think it is not able to discover the other pods. How to use the ignite
> visor script in a K8s environment ?
>
> Regards,
> Vishwas
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: POC with DataStreamer (Kafka or Ignite - security question)

[David Harvey]

Assuming Ignite Persistence,  you can create a cache in a specific Data
Regions, but I'm unclear whether this properties can be set per region.  We
are setting them in

org.apache.ignite.configuration.DataStorageConfiguration.   What you seem
to be asking for is to set these per Data Region.


   



   




On Fri, Apr 6, 2018 at 1:58 PM, Wilhelm  wrote:

> Hello,
>
> I'm building a POC, right now I have Kafka feeding ignite with the
> constraint of having 1 topic per tenant for security reason (historical
> data
> is persisted to files per topic by Kafka and each "file" container is
> encrypted differently per customer)
>
> If I decide to use only ignite with the DataStream (instead of Kafka), how
> can I make sure the customer data will be separated to disk (like separate
> file or db per customer)? and how can I encrypt this cache file per
> customer?
>
> I hope that make sense. So I guess it come down to: can I have the ignite
> cache persisted in different file/db per some rules (like a customer id)?
> or
> does it needs to be in different ignite memory caches? or it is not
> possible?
>
> Thanks for your help
>
> w
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Broadcast method fails with DataStorage peristenceEnabled

[David Harvey]

I found what I posted on our internal slack at the time, suggesting that I
thought this was a library version mismatch.
"I’m getting some pretty squirrelly behavior.   I’m using maven to
build/run the test on the client node.   I was trying to fix the logging
issue below (which was not blocking).   I added a dependency on
ignite-slf4j on the client, which did not improve the errors, but caused
the server to get  “Caused by: java.lang.ClassNotFoundException: Unknown
pair [platformId=0, typeId=555206796]”in the binary deserialization. "

On Tue, Apr 3, 2018 at 4:50 PM, Supun Nakandala <supun.nakand...@gmail.com>
wrote:

> I verified the java version is the same.
>
> The interesting thing here is the code runs correctly if I disable
> persistence.
>
> 
>
> I wonder whether there is an issue in enabling persistence or I am doing
> something wrong here.
>
> On Tue, Apr 3, 2018 at 12:12 PM, David Harvey <dhar...@jobcase.com> wrote:
>
>> PS.  I'm actually remembering it was due to a mismatch of Java7 vs 8,
>> which does not make that much sense, because we have been running Java7
>> clients recently with no issues.
>>
>>
>>
>>
>> On Tue, Apr 3, 2018 at 3:05 PM, David Harvey <dhar...@jobcase.com> wrote:
>>
>>> I had those "UNKNOWN PAIR" issues early on, and it seemed to be some
>>> kind of version problem, like the client was not running at the same rev.
>>>
>>> On Tue, Apr 3, 2018 at 2:15 PM, Supun Nakandala <
>>> supun.nakand...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am trying to setup Apache Ignite with persistence enabled and facing
>>>> a deserialization failure when executing broadcast method with data storage
>>>> persistence enabled.
>>>>
>>>> The following code will recreate this problem:
>>>>
>>>> public static void main(String[] args) throws IOException {
>>>> final String CONFIG_PATH = "./config/example-ignite.xml";
>>>>
>>>> Ignition.setClientMode(true);
>>>> try (Ignite ignite = Ignition.start(CONFIG_PATH)) {
>>>> ignite.active(true);
>>>> IgniteCompute compute = ignite.compute(ignite.cluster(
>>>> ).forRemotes());
>>>> compute.broadcast(() -> System.out.println("Hello Node: " +
>>>> ignite.cluster().localNode().id()));
>>>> }
>>>> }
>>>>
>>>> Persistence configuration used in example-default.xml (everything else
>>>> is set to default)
>>>>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> >>> value="true"/>
>>>>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>
>>>> I am using Apache Ignite 2.4.0.
>>>>
>>>> Your help for fixing this issue is greatly appreciated.
>>>>
>>>> Thank You
>>>> -Supun
>>>>
>>>
>>>
>>
>>
>> *Disclaimer*
>>
>> The information contained in this communication from the sender is
>> confidential. It is intended solely for use by the recipient and others
>> authorized to receive it. If you are not the recipient, you are hereby
>> notified that any disclosure, copying, distribution or taking action in
>> relation of the contents of this information is strictly prohibited and may
>> be unlawful.
>>
>> This email has been scanned for viruses and malware, and may have been
>> automatically archived by *Mimecast Ltd*, an innovator in Software as a
>> Service (SaaS) for business. Providing a *safer* and *more useful* place
>> for your human generated data. Specializing in; Security, archiving and
>> compliance. To find out more Click Here
>> <http://www.mimecast.com/products/>.
>>
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Broadcast method fails with DataStorage peristenceEnabled

[David Harvey]

PS.  I'm actually remembering it was due to a mismatch of Java7 vs 8, which
does not make that much sense, because we have been running Java7 clients
recently with no issues.




On Tue, Apr 3, 2018 at 3:05 PM, David Harvey <dhar...@jobcase.com> wrote:

> I had those "UNKNOWN PAIR" issues early on, and it seemed to be some kind
> of version problem, like the client was not running at the same rev.
>
> On Tue, Apr 3, 2018 at 2:15 PM, Supun Nakandala <supun.nakand...@gmail.com
> > wrote:
>
>> Hi all,
>>
>> I am trying to setup Apache Ignite with persistence enabled and facing a
>> deserialization failure when executing broadcast method with data storage
>> persistence enabled.
>>
>> The following code will recreate this problem:
>>
>> public static void main(String[] args) throws IOException {
>> final String CONFIG_PATH = "./config/example-ignite.xml";
>>
>> Ignition.setClientMode(true);
>> try (Ignite ignite = Ignition.start(CONFIG_PATH)) {
>> ignite.active(true);
>> IgniteCompute compute = ignite.compute(ignite.cluster(
>> ).forRemotes());
>> compute.broadcast(() -> System.out.println("Hello Node: " +
>> ignite.cluster().localNode().id()));
>> }
>> }
>>
>> Persistence configuration used in example-default.xml (everything else is
>> set to default)
>>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> I am using Apache Ignite 2.4.0.
>>
>> Your help for fixing this issue is greatly appreciated.
>>
>> Thank You
>> -Supun
>>
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Broadcast method fails with DataStorage peristenceEnabled

[David Harvey]

I had those "UNKNOWN PAIR" issues early on, and it seemed to be some kind
of version problem, like the client was not running at the same rev.

On Tue, Apr 3, 2018 at 2:15 PM, Supun Nakandala 
wrote:

> Hi all,
>
> I am trying to setup Apache Ignite with persistence enabled and facing a
> deserialization failure when executing broadcast method with data storage
> persistence enabled.
>
> The following code will recreate this problem:
>
> public static void main(String[] args) throws IOException {
> final String CONFIG_PATH = "./config/example-ignite.xml";
>
> Ignition.setClientMode(true);
> try (Ignite ignite = Ignition.start(CONFIG_PATH)) {
> ignite.active(true);
> IgniteCompute compute = ignite.compute(ignite.cluster(
> ).forRemotes());
> compute.broadcast(() -> System.out.println("Hello Node: " +
> ignite.cluster().localNode().id()));
> }
> }
>
> Persistence configuration used in example-default.xml (everything else is
> set to default)
>
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
> 
>
> I am using Apache Ignite 2.4.0.
>
> Your help for fixing this issue is greatly appreciated.
>
> Thank You
> -Supun
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: One time replicating of the cluster data for setting up a new cluster

[David Harvey]

When using Ignite Persistence, I have found that you can simply replicate
the folders, if you deactivate the cluster first.

You need the wal and store, as well as two folders under $IGNITE_HOME/work
which are named something like marshaller* and binary*

On Tue, Apr 3, 2018 at 2:52 AM, Naveen  wrote:

> Hi
>
> Am using 2.3
> I do have one cluster running with 3 servers, each with 2 nodes.
>
> Now we need to replicate this data on a new cluster which is going to have
> the same configuration.
>
> Just copying node folders enough to get the data ?
>
> Whats the best practice for doing this
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Slow data load in ignite from S3

[David Harvey]

When I did this, I found that with Ignite Persistence,  there is a lot of
write amplification (many times more bytes written to SSD than data bytes
written) the during the checkpoints, which makes sense because ignite
writes whole pages, and each record written dirties pieces of many pages.

The SSD write latency and throughput become critical.On slower devices
(e.g., EBS GP2) separating the WAL  can help a bit, but the key is device
write speed.On AWS, I found I needed to use local storage.

On Mon, Apr 2, 2018 at 6:22 AM, Andrey Mashenkov  wrote:

> Hi Rahul,
>
> Possibly, mostly a new data is loaded to Ignite.
> I meant, Ignite allocate new pages, rather than update ones.
>
> In that case, you may not get benefit from increasing checkpoint region
> size. It will just deffer a checkpoint.
>
> Also, you can try to move WAL and ignite store to different disks and to
> set region initial size to reduce or avoid region extents allocation .
>
> On Mon, Apr 2, 2018 at 9:59 AM, rahul aneja 
> wrote:
>
>> Hi Andrey,
>>
>> Yes we are using SSD. Earlier we were using default checkpoint buffer 256
>> MB , in order to reduce the frequency, we increased the buffer size , but
>> it didn’t have any impact on performance
>>
>> On Fri, 30 Mar 2018 at 10:49 PM, Andrey Mashenkov <
>> andrey.mashen...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Possibly, storage is a bottleneck or checkpoint buffer is too large.
>>> Do you use Provissioned IOPS SSD?
>>>
>>>
>>> On Fri, Mar 30, 2018 at 3:32 PM, rahul aneja 
>>> wrote:
>>>
 Hi ,

 We are trying to load orc data (around 50 GB) on s3  from spark using
 dataframe API. It starts fast with good write throughput  and then after
 sometime throughput just drops and it gets stuck.

 We also tried changing multiple configurations , but no luck
 1. enabling checkpoint write throttling
 2. disabling throttling and increasing checkpoint buffer


 Please find below configuration and properties of the cluster


1. 10 node cluster r4.4xl (EMR aws) and shared with spark
2.  ignite is started with -Xms20g -Xmx30g
3.  Cache mode is partitioned

4. persistence is enabled
5. DirectIO is enabled
6. No backup

 
>>> guration.DataStorageConfiguration”>



>>> guration.DataRegionConfiguration”>
>>> value=“true”/>
>>>value=“#{20L * 1024 * 1024 * 1024}“/>

>>> 1024 * 1024}“/>







 Thanks in advance,

 Rahul Aneja



>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Data Streamer not flushing data to cache

[David Harvey]

By default, DataStreamer will only send a full buffer, unless you
explicitly flush or close it, or as suggested, implicitly close it.   You
can set a flush timer also.

The last time I looked (2.3), flush timer is implemented to flush
periodically, and this is unaffected by when data was last added to the
stream, i.e., if you want to ensure that it has stored all of the records
older than 5 seconds, setting the flush timer to 5 seconds will cause it to
flush every 5 seconds, even when all of the outstanding data was added in
the last 10ms.  So the flush timer creates an (unnecessary) performance
anomaly under heavy load.

Note that while the DataStreamer is thread safe, if you have a
multi-threaded producer, works much better to have a DataStreamer per
producer thread, especially if there are explicit or timer driven flushes.


-DH


On Sat, Mar 31, 2018 at 5:38 AM, begineer  wrote:

> Hi, This must be something very simple. I am adding 100 items to data
> streamer. But it is not flushing items to cache. Is there a settings much
> enables it. Cache size is zero. Am I doing something wrong ?
>
> public class DataStreamerExample {
> public static void main(String[] args) throws InterruptedException {
> Ignite ignite = Ignition.start("examples/
> config/example-ignite.xml");
> CacheConfiguration config = new
> CacheConfiguration<>("mycache");
> IgniteCache cache = ignite.getOrCreateCache(config);
> IgniteDataStreamer streamer =
> ignite.dataStreamer("mycache");
> LongStream.range(1, 100).forEach( l->{
> System.out.println("Adding to streamer "+ l);
> streamer.addData(l, l);
> });
> System.out.println(streamer.perNodeBufferSize());
> System.out.println("Cache size : "+ cache.size(CachePeekMode.ALL))
> ;
> cache.query(new ScanQuery<>()).getAll().stream().forEach(entry->{
> System.out.println("cache Entry: " + entry.getKey()+" "+
> entry.getValue());
> });
> }
> }
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Determining BinaryObject field type

[David Harvey]

 I had stopped at BinaryObject, and didn't follow the indirection though
type() to BinaryType. I think I assumed that this had only information at
the higher level, and wouldn't drill down into the fields.
This also answers a question about how to enumerate the fields.

Thanks.
-DH

On Wed, Mar 28, 2018 at 3:10 AM, Pavel Vinokurov 
wrote:

> Dave,
>
> I suppose there isn't way to delete the schema.
> You could get the meta information about binary objects using Ignite#binary()
> method.
> For example ignite.binary().type("package.Employeee").
> fieldTypeName("name").
>
>
>
> Thanks,
> Pavel
>
> 2018-03-24 1:10 GMT+03:00 Dave Harvey :
>
>> Once a BinaryObjectSchema is created, it is not possible to change the
>> type
>> of a field for a known field name.
>>
>> My question is whether there is any way to determine the type of that
>> field
>> in the Schema.
>>
>> We are hitting a case were the way we get the data out of a different
>> database returns a TIMESTAMP, but our binary object wants a DATE. In
>> this test case, I could figure out that, but in the general case,  I have
>> a
>> BinaryObject type name, and a field name, and an exception if I try to put
>> the wrong type in that field.
>>
>> The hokey general solutions I have come up with are:
>> 1) Parse the exception message to see what type it wants
>> 2) Have a list of conversions to try for the source type, and step through
>> them on each exception.
>> 3) Get the field from an existing binary object of that type, and use the
>> class of the result.   But there is the chicken/egg problem.
>>
>> I have found that I can create a cache on a cluster with persistence, with
>> some type definition, then delete that cache, the cluster will remember
>> the
>> BinaryObjectSchema for that type, and refuse to allow me to change the
>> field's type.  If I  don't remember the field's type, how can I
>> build the binary object?
>>
>> Is there any way to delete the schema without nuking some of the
>> binary-meta/marshaller files when the cluster is down?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
>
> Regards
>
> Pavel Vinokurov
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Ingite deploy in AWS

[David Harvey]

As you already know, you cannot have the config file in the discovery
bucket, because discovery enumerates the contents of the discovery bucket.


But the credentials in the config bucket are not yet visible, so they
cannot be used.   You can make the config bucket public, but not with the
key stored in it.   But doing this briefly may help make sure that you are
on the right track.

We created a VPC endpoint, and gave that endpoint access to the config
bucket. The bucket policy we used to do this:
{
"Version": "2012-10-17",
"Id": "Policy1442576558687",
"Statement": [
{
"Sid": "Stmt1442576554133",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::-config",
"arn:aws:s3:::-config/*"
],
"Condition": {
"StringEquals": {
"aws:sourceVpc": "vpc-b77954d2"
}
}
},
{
"Sid": "Access from JC Non-prod",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111:root"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::-config"
}
]
}


Test using the instance credentials
aws s3 ls s3://...-ignite-Config/

Uses no AWS credentials except that of the VPC endpoint
curl -s http://s3.amazonaws.com/...Config


We also use this in the config file


which requires the the instance role be able to access the discovery
bucket.  But it gets rid of the exposed keys.





On Mon, Mar 26, 2018 at 12:28 PM, theizer  wrote:

> I am trying to deploy an Apache Ignite cluster in our AWS VPC.  I have done
> the following:
>
> •   Created an S3 Bucket to store the ignite-config.xml file
> o   Uploaded the config file to the bucket.
>    Config file contents (just based on example in github):
> http://www.springframework.org/schema/beans;
>xmlns:util="http://www.springframework.org/schema/util;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd
>http://www.springframework.org/schema/util
>http://www.springframework.org/schema/util/spring-util.xsd;>
>
>class="org.apache.ignite.configuration.IgniteConfiguration">
> 
>   
> 
>   
>   
>   
>   
> 
>  class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
> 
> 
>   
>   
> 
> 
> 
>  
> 
> 
> 
>   
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.
> TcpDiscoveryS3IpFinder">
>   
>value="S3BucketNAME-ignite-discovery"/>
>  
>   
> 
> 
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
>   
> 
>   
>   
>
>   
> 
> 
>   
> 
> •   Created an S3 bucket for S3 Discovery
> •   Created an IAM role for Ignite-S3-Access with the
> AmazoneS3FullAccess
> policy attached and ec2.amazonaws.com as a trusted entity
> •   Launched an EC2 instance from the Apache Ignite AMI for the
> us-east-1
> region
> o   In my VPC private subnet
> o   With IAM role created above for S3 Access
> o   User Data:
> 
> CONFIG_URI=https://s3.amazonaws.com/S3BucketName-
> ignite-config/ignite-config.xml
>
>
> When I log into the instance, the docker container is NOT running.
>
> I can verify that the IAM role for S3 is working by typing aws s3 ls and
> getting a list of all s3 buckets.
>
> If I try to manually start the docker container with the command:  sudo
> docker run -it --net=host -e
> "CONFIG_URI=https://s3.amazonaws.com/S3BucketName-
> ignite-config/test-cluster.xml"
> apacheignite/ignite I get a 403 which makes since as the S3Bucket is not
> public:
> o   class org.apache.ignite.IgniteException: Failed to instantiate
> Spring XML
> application context
> [springUrl=https://s3.amazonaws.com/S3BucketName-
> ignite-config/test-cluster.xml,
> err=IOException parsing XML document from URL
> [https://s3.amazonaws.com/S3BucketName-ignite-config/test-cluster.xml];
> nested exception is java.io.IOException: Server returned HTTP response
> code:
> 403 for URL:
> https://s3.amazonaws.com/S3BucketName-ignite-config/test-cluster.xml]
>
>
> If I try running it like this:  sudo docker run -it --net=host -e
> "CONFIG_URI=$CONFIG_URI" apacheignite/ignite  it seems to start but does
> not
> register it’s ip with the S3 Discovery bucket.
>
>
> I'm a newbie and would appreciate this groups assistance! Please help me
> see
> what I'm missing.
>
> Thanks!
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Basic terms in Ignite

[David Harvey]

1.   You can call ignite.start() twice in the same JVM, and you get two
Ignite nodes (there are some restrictions around the same cluster I don't
understand).  An ignite node may be a client or a server.  I believe
can have two client nodes in the same JVM connected to two different
clusters.( For peer class loading, the terms master and workers
introduced, just to be confusing, where in typical use cases the master is
the client and the workers the servers.)

2. A cache is broken, by default, into 1000 partitions, and the primary for
a partition exists on exactly one node, as does each backup for the
partition.   A partitioned cache can be created on any "cluster group",
which is not, as you might think, a "group of clusters" but instead a
subset of the nodes in a cluster.Of course, it would not make sense to
have less nodes than # backups + 1.

3. In 2.4, you are using Ignite Persistence,  the cluster will remain
inactive on startup until the set of nodes you have defined as necessary
have started, and then it will activate automatically.  Prior versions
needed to be manually activated.   I'm not aware of any controls w/o Ignite
Persistence for the case your describe.

On Sun, Mar 25, 2018 at 7:53 AM, begineer  wrote:

> Hi,
> I am confused about few terms used often in ignite docs. Could someone
> please clarify them.
>
> 1. What is difference between a Server, JVM and ignite node. Can one have
> multiple instances of other 2, like a jvm has two nodes or a node has 2
> jvms. Please explain in detail, I am very confused about them
>
> 2. In partitioned mode, can a partition span across multiple nodes. Again
> relating to first point. If I have 10 servers of 10GB each so total memory
> is 100 GB. Can a partition be created on 2-3 servers of 30 GB size. This is
> hypothetical example though.
>
> 3. Lets say I have 90 GB data to be loaded on 10 servers of 10GB each. Now
> if first ignite node joins the gird and starts loading data. There might be
> a chance that it loads so much data that it runs out of memory before new
> ones join the grid. How do we handle this scenario. Is there a way to wait
> for some condition before kicking of cache loading.
>
> Thanks
> Surinder
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Issues trying to force redeployment in shared mode

[David Harvey]

   1. I had created IGNITE-7905 for this interaction with userVersion and
   perhaps only ignite.Activate() issued from the client.  I think it deals
   with the code for Activate() using nested classes.  The server decided that
   it already had a version of the ignite code loaded, and wasn't going to
   accept a different version from the client.   My intent was to shoot down
   only my code.   When I stopped trying to activate the cluster from the
   client, this symptom disappeared.And in 2.4 the need to try to activate
   the cluster from the client went away.
   2. -DH
   3.
  


On Thu, Mar 22, 2018 at 12:00 PM, ilya.kasnacheev  wrote:

> Hello Dave!
>
> I suggest filling an issue agains Apache Ignite JIRA with this scenario and
> suggestions about how it should work after fix. Test case would be awesome.
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Any references - Syncing Ignite and Oracle DB with Oracle GoldenGate - updates from DB to ignite

[David Harvey]

For MySql we are reading the binlog in row,format and sending the updates,
with the LSN appended on a,data streamer with a custom Stream Reciever that
uses the lsn to detect out of order cases.   We get eventually consistent
results only.

On Fri, Mar 23, 2018, 3:48 AM Naveen  wrote:

> Hi
>
> We are using ignite 2.3/2.4
>
> We are looking for a solution which requires syncing ignite with DB updates
>
> We have legacy systems directly updating the DB and these updates should
> get
> propagated to ignite, so that end users are served from Ignite cluster
> (RAM)
>
> Has any one implemented this solution ?
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Affinity Key column to be always part of the Primary Key

[David Harvey]

Based this  latest description, simply not specifying an affinity key here
would be sufficient.   But presumably you were specifying the affinity key
to cause co-location.   The reason a lookup on the K of the KV pair is fast
is because it can hash to a node.   If  the affinity key was not included
in the K then you would not get colocation.  To do a lookup on only part of
K means you cannot hash to node.Assuming that you can't know the
affinity key a priori, i.e., you know the person and not the company, there
are other solutions.   Depending on the data, you may be able to use a
replicated cache for other (company) data rather than trying to co-locate
this record with that other data.  That increases the storage and
insert/update cost for company records.
There are also a number of solutions using Java to send closures to the
server nodes.

On Wed, Mar 21, 2018 at 1:55 AM, Naveen  wrote:

> My whole and sole requirement is to make lookup work fast, this can only be
> achieved with lookup based on primary key, cluster knows which node is
> holding this record based on the key, no other look-up can give best
> results
> close to primary key lookup.
>
> We have done some tests on lookup based on the indexed column, which is
> giving (1/5)th of the TPS of lookup based on the primary key
>
> Lookup with primary key was around 30K TPS
> Lookup on indexed column was around 6 K TPS
>
> Though its indexed, request still goes to all the nodes, instead of
> querying
> the source table it will query index table which is small in size and
> sorted, so retrieval process is faster, is this understanding correct ??
>
> Results are justifiable, considering the architecture, where indexes are
> distributed across the nodes.
>
> In my view, affinity should not influence the data model, at the most it
> can
> add  to the memory footprint by adding extra column to the table just for
> the same of using affinity (collocating the data).
>
> If the whole idea of having affinity key as part of the primary key is to
> just identify the order of fields, then we should look for other ways not
> by
> enforcing the affinity key as part of composite primary key.
>
> Regards
> Naveen
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Affinity Key column to be always part of the Primary Key

[David Harvey]

What is your goal?   If you have a unique key that does not contain the
affinity key, and the primary key contains both fields, you can create an
index on that unique key, so that you could have fast node local lookups
using SqlQuery().If you want to find an object only by that original
unique key from a random node, then you must either search all nodes, or
not append an affinity key.So if Ignite had done what you suggested,
the cost of a single lookup by primary key would increase with node count,
which would be at odds with it being declared a primary key.


On Tue, Mar 20, 2018 at 12:57 PM, Dmitriy Setrakyan 
wrote:

>
>
> On Tue, Mar 20, 2018 at 2:09 PM, Vladimir Ozerov 
> wrote:
>
>> Internally Ignite is key-value storage. It use key to derive partition it
>> belongs to. By default the whole key is used. Alternatively you can use
>> @AffinityKey annotation in cache API or "affinityKey" option in CREATE
>> TABLE to specify *part of the key* to be used for affinity calculation.
>> Affinity column cannot belong to value because in this case single
>> key-value pair could migrate between nodes during updates and
>> IgniteCache.get(K) will not be able to locate the key in cluster.
>>
>
> Vladimir, while it makes sense that the key must be composed of the ID and
> Affinity Key, I still do not understand why we require that user declares
> them both as PRIMARY KEY. Why do you need to enforce that explicitly? In my
> view you can do it automatically, if you see that the table has both,
> PRIMARY KEY and AFFINITY KEY declared.
>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: New feature with 2.4 - Using 3rd Party Persistence With Native Persistence

[David Harvey]

IIgnite persistence is per data region
https://apacheignite.readme.io/docs/memory-configuration#section-data-regions

On Mon, Mar 19, 2018 at 10:11 AM, Naveen  wrote:

> Hi
>
> Looks like dual persistence - 3rd party (Oracle) and native persistence -
> both are supported
>
> "*Starting with Apache Ignite 2.4, 3rd party persistence can be used
> together with native persistence*"
>
> Can someone explain how this is going to work, how do we config this.
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Affinity Key column to be always part of the Primary Key

[David Harvey]

Yes, the affinity key must be part of the primary key.   Welcome to my
world

On Fri, Mar 16, 2018 at 3:23 AM, Naveen  wrote:

> Hi Mike
>
> I have created a table called CITY
>
> : jdbc:ignite:thin://127.0.0.1> CREATE TABLE City (  city_id LONG PRIMARY
> KEY, name VARCHAR)  WITH "template=replicated";
> No rows affected (0.224 seconds)
>
> Creating a table called Person with affinity key as city_id
>
> 0: jdbc:ignite:thin://127.0.0.1> CREATE TABLE IF NOT EXISTS Person ( age
> int, id int, city_id LONG , name varchar, company varchar,  PRIMARY KEY
> (name, id)) WITH "template=partitioned,backups=1,affinitykey=city_id,
> key_type=PersonKey, value_type=MyPerson";
>
> This is the exception I get
>
> Error: Affinity key column must be one of key columns: CITY_ID
> (state=42000,code=0)
> java.sql.SQLException: Affinity key column must be one of key columns:
> CITY_ID
> at
> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(
> JdbcThinConnection.java:671)
> at
> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.
> execute0(JdbcThinStatement.java:130)
> at
> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.
> execute(JdbcThinStatement.java:299)
> at sqlline.Commands.execute(Commands.java:823)
> at sqlline.Commands.sql(Commands.java:733)
> at sqlline.SqlLine.dispatch(SqlLine.java:795)
> at sqlline.SqlLine.begin(SqlLine.java:668)
> at sqlline.SqlLine.start(SqlLine.java:373)
> at sqlline.SqlLine.main(SqlLine.java:265)
> 0: jdbc:ignite:thin://127.0.0.1>
>
> And, when I change the primary key to include affinity id, below DDL is
> working fine.
> 0: jdbc:ignite:thin://127.0.0.1> CREATE TABLE IF NOT EXISTS Person ( age
> int, id int, city_id LONG , name varchar, company varchar,  PRIMARY KEY
> (name, id,city_id)) WITH
> "template=partitioned,backups=1,affinitykey=city_id, key_type=PersonKey,
> value_type=MyPerson";
>
> This is what I was trying to explain, is affinity key to be part of the
> primary key ??
>
> If this is the case, whole my data model will change drastically.
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

SELECT Statement cancellation & memory sizing

[David Harvey]

I have a question about SQL query via JDBC cancellation, as well as a
performance question.

I've connected Ignite from SQL/Workbench using the Thin driver, however via
ssh -L proxy and I issued
"select count(*) from table1 where indexedField = x;" and that took 1:40
(m:ss)

"select count(*) from table1;"   seemed to take 1:20:00, based on when the
cluster went idle again.

This is using Ignite Persistence on 2.3.   The cluster is 8 x i3.8xlarge,
with the 4 local SSDs configured as a RAID 0.  These have 244MB memory
each, and 32 CPUs. The table perhaps has 500M rows at an average of 1-2KB
each.QueryParallelism is at the default of 1.  Each node has about
220GB of SSB space allocated.

Repeating this when everything was idle,I issue the select statement,
and then cancelled it using the SQL/Workbench button (which works for
MySQL).   Nothing happened, and I can see that the cluster is reading about
50MB/s from each node.   I then killed SQL/Workbench  to break the
connection, and then the ssh proxy, but the connection loss did not
terminate the query either.   What is the expected behavior when a Query is
killed via JDBC.

On the performance side, it appears that there are no optimizations around
"count(*)", and it is reading all of the data on the SSDs (50MB/s gets
close to 220GB in 1:20:00).   I could believe this is just a standard scan
problem when the data size is a bit larger than the cache, and we could
improve this by increasing query parallelism, or adding nodes, if this use
case was important.

However, node stats from visor, if I'm reading them correctly,  indicate it
is not using the memory, but I believe I enabled 160GB of off-heap space.
 How do I reconcile the visor stats with the memory settings?   All of the
nodes have been up since the data was loaded.

 Current CPU load %  | 0.03%
|

| Average CPU load %  | 0.16%
  |

| Heap memory initialized | 30gb
  |

| Heap memory used| 15gb
  |

| Heap memory committed   | 30gb
  |

| Heap memory maximum | 30gb
  |

| Non-heap memory initialized | 2mb
  |

| Non-heap memory used| 125mb
  |

| Non-heap memory committed   | 132mb
  |

| Non-heap memory maximum | 1gb




101 

102 

103

104 

105 

106 

107 

108

109 

110 

111

112 

113

114

115

116

117

118

119

120

121

122

123  

124  

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: NODE_SEGMENTED AND NODE_FAILED errors

[David Harvey]

When I hit something like this, the logs suggested increasing
socketWriteTimeout, and that helped for me.

On Fri, Mar 2, 2018 at 7:58 AM, KR Kumar  wrote:

> Hi - I have a three node cluster with c4.2xlarge machines with ignite
> persistence enabled. While doing a load testing of the cluster, after
> writing about 50 million events to the cache, I am getting NODE_SEGMENTED
> and NODE_FAILED notifications and after the cache server sort of hangs. But
> when I see all the JVMs are perfectly up and running.
>
> For NODE_FAILED and NODE_SEGMENTED errors, can you give me some pointers as
> to what I should be looking at??
>
> Thanx and Regards,
> KR Kumar
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Large durable caches

[David Harvey]

In our case, we use 1+ backups, so Ignite replicates the data to other
nodes.   My  understanding:  Ignite replicates changes at the K/V or record
level and sends them to the backup node's WAL.  The backup node may differ
per partition.   The backup node applies the changes to its storage, and
also updates the local SQL indices (which only reference records on the
local node).   Ignite persistence writes data to the SSDs in pages, so if
you change 80 bytes, it must write multiple 4K pages on the SSDs, but only
send around those 80 bytes over the network.  So you need a lot more SSD
bandwidth than network bandwidth when using Ignite persistence with backups.

-DH

On Thu, Feb 22, 2018 at 6:52 PM, Raymond Wilson <raymond_wil...@trimble.com>
wrote:

> If you’re using local SSDs, these are not durable by definition. How are
> you ensuring your data survives termination of the instance, or a failure
> of the underlying physical host?
>
>
>
> *From:* David Harvey [mailto:dhar...@jobcase.com]
> *Sent:* Friday, February 23, 2018 10:03 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: Large durable caches
>
>
>
> Using the local SSDs, which are radically faster than EBS.
>
>
>
> On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn <lar...@gmail.com> wrote:
>
> I could try a different AWS instance.  Im running these tests on r4.8xlarge
> boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
> using IO1 disks at 20,000 IOPS but still had issues.
>
> Dave with the i3 instances, were you using the local ssds?  Or still using
> EBS?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> <http://www.mimecast.com/products/>.
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Large durable caches

[David Harvey]

Using the local SSDs, which are radically faster than EBS.

On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn  wrote:

> I could try a different AWS instance.  Im running these tests on r4.8xlarge
> boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
> using IO1 disks at 20,000 IOPS but still had issues.
>
> Dave with the i3 instances, were you using the local ssds?  Or still using
> EBS?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Ignite Persistence performance issue

[David Harvey]

I have had more success with i3 instances

On Feb 21, 2018 10:01 PM, "KR Kumar"  wrote:

Hi All - I am using Ignite Persistence ( 2.3) for storing events that we
receive from different sources. Currently the write and read throughput is
2k to 4k per second on a t2.xlarge and c4.2xlarge machines on a 3 node
cluster. The disks attached are provisioned SSDs with 1 IOPS.  These
numbers are not very encouraging and I am wondering if there is any way to
improve or I am missing something fundamental..

The event size that i have is around 1kb.

The datastorageconfiguration config is as follows :





















Not sure the details i have provided are good enough to give a concrete
suggestion, so let me know if you need more info.

Thanx and Regards,
KRKumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: 20 minute 12x throughput drop using data streamer and Ignite persistence

[David Harvey]

We are pulling in a large number of records from an RDB, and reorganizing
the data so that our analytics will be much faster.

I'm running Sumo, and have looked at all of the log files from all the
nodes, and the only things are checkpoints and GC logs.  The checkpoints
are fast, and occur at a lower rate during the slowdowns.   GC is not a
problem at all.
(I see a project in the future where the number of messages/bytes per TOPIC
are counted.)

The average packet size goes to 6KB from a normal ~ 400 bytes.   I'm going
to add  a rebalance throttle.


On Tue, Feb 13, 2018 at 5:41 AM, slava.koptilin 
wrote:

> Hi Dave,
>
> Could you please provide more details about that use-case.
> Is it possible to reproduce the issue and gather JFR and log files from all
> participated nodes?
> It would be very helpful in order to understand the cause of that behavior.
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.