compute ignite data with spark

[shawn.du]

Hi,Spark is a compute engine.  Ignite also provide compute feature. Also Ignite can integrate with spark.We are using ignite compute map-reduce feature now.  It is very fast.I am just curious how spark compares with ignite on computing.it is possible using spark API computing ignite cache data? 






ThanksShawn

Re: Ignite with mariadb

[Denis Magda]

Sumanth,

There should be no problems with MariadDB in Ignite. You just need to add
the JDBC driver to your classpath. I'll make sure it's properly documented.

A series of "Ignite Web Console - Automatic RDBMS Integration" screencasts
shows how to do this with MySQL.
https://ignite.apache.org/screencasts.html

--
Denis

On Mon, Feb 19, 2018 at 12:48 AM, Sumanth Sumanth <4112...@gmail.com> wrote:

> Hi,
>
> Can Ignite be used with Mariadb?
> When attempted (via jdbc url:  jdbc:mariadb://localhost:3306/
> db1?user=root=password) Ignite throws exception that it is not
> able to find suitable jdbc driver for the Mariadb.
> What is the recommended way of integrating Mariadb with Ignite?
>
> Thanks,
> Sumanth
>

RE: Does object should implement Serializable?

[Ariel Tubaltsev]

Hi Stan

Makes sense, thank you for clarification.

Ariel



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Large durable caches

[David Harvey]

In our case, we use 1+ backups, so Ignite replicates the data to other
nodes.   My  understanding:  Ignite replicates changes at the K/V or record
level and sends them to the backup node's WAL.  The backup node may differ
per partition.   The backup node applies the changes to its storage, and
also updates the local SQL indices (which only reference records on the
local node).   Ignite persistence writes data to the SSDs in pages, so if
you change 80 bytes, it must write multiple 4K pages on the SSDs, but only
send around those 80 bytes over the network.  So you need a lot more SSD
bandwidth than network bandwidth when using Ignite persistence with backups.

-DH

On Thu, Feb 22, 2018 at 6:52 PM, Raymond Wilson 
wrote:

> If you’re using local SSDs, these are not durable by definition. How are
> you ensuring your data survives termination of the instance, or a failure
> of the underlying physical host?
>
>
>
> *From:* David Harvey [mailto:dhar...@jobcase.com]
> *Sent:* Friday, February 23, 2018 10:03 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: Large durable caches
>
>
>
> Using the local SSDs, which are radically faster than EBS.
>
>
>
> On Thu, Feb 22, 2018 at 10:34 AM, lawrencefinn  wrote:
>
> I could try a different AWS instance.  Im running these tests on r4.8xlarge
> boxes, which are pretty beefy and "ebs optimized".  I tried the same tests
> using IO1 disks at 20,000 IOPS but still had issues.
>
> Dave with the i3 instances, were you using the local ssds?  Or still using
> EBS?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> .
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

getting error "Queue has been removed from cache: GridCacheQueueAdapter" with ExecutorService

[nheron]

Hi, 
I am using ExecutorService from Ignite  instance and I sometimes get this
error :

"Queue has been removed from cache: GridCacheQueueAdapter "

Where can I configure that queue ?

Using apacheIgnite 2.3.0 in a spring boot application 

Thanks 
Nicolas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to dismiss a ignite server node.

[shawn.du]

Hi,I know it is hard.  But it is a common requirement. I wish ignite can have this feature in future.Think below scenarios:I initialize a 2-nodes server and all works fine.  later  I find my memory will be used up soon.so i decide to add a new node.  I know ignite can do this. All works fine. later then I find thatuse 3 node is too expensive(we run ignite on AWS) we don't need that much memory, I want to shrink the cluster, but I don't wantto stop it. It means even we are shrinking the cluster, all our business logic  like cache, computing, SQL queryshould works as usual. For we really need this feature, I have a careful thought about it. My thought is simple:1) use admin tools to mark a node in a DISMISSING status. when node in this status, it is only readable, computing jobs also OK, but can't write.2) copy the data in above node to other node. 3) when copy is finished, set the node status in DISMISSED status. node in DISMISSED status will not involved any operations.4) then user can safely kill the process. I know talking is always simple, coding is hard.  Leveraging ignite's atomic, distributed locks, it may be possible.welcome any comments.






ThanksShawn





On 2/23/2018 17:47，Вячеслав Коптилин wrote： 


Hello,If you don't have configured 3rd party cache store or backups,then there is only one possible way, I think.You need to  - store all the data from the node you want to shut down, - shut down the node  - upload stored data back to the cluster.It can be done via an external database, for example.If the data set is not huge, you can try to use a new partitioned cache (with node filter/backups) or replicated cache.Thanks.2018-02-23 10:00 GMT+03:00 shawn.du :







Hi,Suppose I have several ignite server nodes which compose a cluster.all data in PARTITIONED mode, and with no backups.It is possible to dismiss a node without restarting and data lose?if possible, what are the steps?






ThanksShawn

Re: Using 3rd party DB together with native persistence (WAS: GettingInvalid state exception when Persistance is enabled.)

[Prasad Bhalerao]

Hi Stan,

Could you please elaborate more on this point "*but if cache configured
with write-through or read-through modes enabled then data consistency
between Ignite persistence and CacheStore is not guaranteed at all times*" .

If I have enabled ignite persistence and my cache is also read-through and
write-through then why does it not guarantee 100% data consistency?


Thanks,
Prasad

On Wed, Feb 21, 2018 at 7:29 PM, Stanislav Lukyanov 
wrote:

> Sorry, I wasn’t completely correct. Persistence and CacheStore can work
> together, but if cache configured with write-through or read-through modes
> enabled then data consistency between Ignite persistence and CacheStore is
> not guaranteed at all times, so such configurations are usually avoided.
> However, from what you’re saying it seems that you don’t need write-through
> or read-through, so it should work fine.
>
> Your solution seems OK. If you only need a one-time setup I guess you
> could also do it without a CacheStore and just get the data from Oracle and
> put it into Ignite via putAll or DataStreamer.
>
>
>
> Thanks,
>
> Stan
>
>
>
> *From: *Prasad Bhalerao 
> *Sent: *21 февраля 2018 г. 14:33
> *To: *user@ignite.apache.org
> *Subject: *Re: Using 3rd party DB together with native persistence (WAS:
> GettingInvalid state exception when Persistance is enabled.)
>
>
>
> Hi Stan,
>
>
>
> Thank you for the reply. I will send different questions separately now
> onwards.
>
>
>
> I do not understand what are you trying to say ( "*Ignite doesn’t support
> using 3rd party DBs and native persistence with the same cache*" ) .
>
>
>
> If the persistence is enabled and if you create or load the cache from 3rd
> party DB or any other source, data is always stored/backed in ignite native
> persistence store.
>
>
>
> I have enable the persistence using IgniteConfiguration. I believe that
> this configuration is global and so it is applicable to all the caches I
> have created. As per my understanding, if persistence is enabled any cache
> created will be stored/backed in ignite persistence store.
>
> So now if I load the data in cache from oracle or any other DB table, it
> will be persisted in ignite native persistence file system. The point is I
> want to load the cache from oracle tables only first time or when the cache
> is empty. To load the cache from 3rd party DB I am using cache.loadCache
> method.
>
>
>
> I am checking cache size, if the size is zero I call loadCache method. If
> the cache is not empty it means that data is already loaded in previous
> attempt so no need to call loadCache method.
>
> I just wanted to know if there is any better solution to achieve this.
>
>
>
>
>
> Thanks,
>
> Prasad
>
>
>
> On Wed, Feb 21, 2018 at 1:33 PM, Stanislav Lukyanov <
> stanlukya...@gmail.com> wrote:
>
> Hi Prasad,
>
>
>
> // Please send different questions separately – this way it’s easier to
> answer and to search for existing answers
>
>
>
> > Also, I am loading the cache from oracle table using loadCache method.
> If the persistence is enabled and if the data is already persisted, I want
> to make sure that the cache is loaded from persisted data instead of
> loading it from oracle table using loadCache. Can someone please advise how
> this can be achieved?
>
>
>
>
>
> Ignite doesn’t support using 3rd party DBs and native persistence with the
> same cache.
>
> If you need to use both, I’d suggest to create two caches, one backed by
> Oracle and one with enabled Ignite persistence, and alternate between them
> in your application code.
>
>
>
> Thanks,
>
> Stan
>
>
>
> *From: *Prasad Bhalerao 
> *Sent: *20 февраля 2018 г. 15:24
> *To: *user@ignite.apache.org
> *Subject: *Getting Invalid state exception when Persistance is enabled.
>
>
>
> Hi,
>
>
>
> I am starting ignite node in server mode in intellij. I am starting only
> one instance of it. I am using IgniteSpringBean to set configuration and
> start the node as shown below. But when I enable persistence, I get
> following exception.
>
>
>
> Caused by: java.lang.IllegalStateException: Ignite is in invalid state to
> perform this operation. It either not started yet or has already being or
> have stopped [ignite=null, cfg=null]
>
>
>
> As per the doc, IgniteSpringBean is responsible for starting the ignite.
> So how to set node to active state in case this case?
>
>
>
> Also, I am loading the cache from oracle table using loadCache method. If
> the persistence is enabled and if the data is already persisted, I want to
> make sure that the cache is loaded from persisted data instead of loading
> it from oracle table using loadCache. Can someone please advise how this
> can be achieved?
>
>
>
> Code to config ignite and cache:
>
>
>
> @Bean
> *public *IgniteSpringBean igniteInstance() {
> IgniteSpringBean ignite = *new *IgniteSpringBean();
> *ignite.active(**true**);*
> 

RE: SSD writes get slower and slower

[Stanislav Lukyanov]

Hi,

Just to confirm – you’re talking about Ignite’s native persistence, right?

Can you share the code and configs you use in your testing?

Thanks,
Stan

From: VT
Sent: 23 февраля 2018 г. 12:02
To: user@ignite.apache.org
Subject: SSD writes get slower and slower

Hello, 

I am trying to test use SSD. But as more data are written, the write
performance drops significantly. Is there something I can do to solve this
problem? Or are there any known issues to this problem that might be fixed
in the upcoming versions? The test was done on a single node, locally. The
total data size is small, not enough to fill half of SSD.

Thanks!

For example, 

Loaded 1 keys 239ms.
Loaded 2 keys 122ms.
Loaded 3 keys 141ms.
Loaded 4 keys 110ms.

...

Loaded 842 keys 8068ms.
Loaded 843 keys 201ms.
Loaded 844 keys 187ms.
Loaded 845 keys 162ms.
Loaded 846 keys 182ms.
Loaded 847 keys 164ms.
Loaded 848 keys 171ms.
Loaded 849 keys 8475ms.

...

Loaded 8437 keys 5684ms.
Loaded 8438 keys 4329ms.
Loaded 8439 keys 5191ms.
Loaded 8440 keys 7352ms.
Loaded 8441 keys 11171ms.

...

Loaded 9994 keys 13352ms.
Loaded 9995 keys 28771ms.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Production ready on Windows? Free vs Commercial difference?

[Olexandr K]

Thank you!

I'm taking Ignite for our project then

Will come with more concrete/technical questions soon )

On Fri, Feb 23, 2018 at 1:23 AM, vkulichenko 
wrote:

> Hi Oleksandr,
>
> 1. Yes, Ignite is production ready.
> 2. I doubt there is a fully pledged example for this, but I don't see any
> reason why it wouldn't be possible. You can run anything you want within a
> service.
> 3. Ignite is always and fully free. If you're asking about commercial
> products build on top of Ignite, please contact vendor companies that
> provide them.
> 4. I doubt that would be possible as in that case you have to operate with
> objects, and therefore write code. But you can emulate such workload with
> simple SQL queries that select/update by primary key.
> 5. Basically it's a tool to route requests of a legacy thin client. Not
> sure
> if it would be useful for the new client protocol, but for now I think you
> can ignore it.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite on Kubernetes: Question on external jars

[Вячеслав Коптилин]

Hello,

It seems to me, that you have to restart your cluster.
Java classpath is not something that can be dynamically changed at runtime.

Thanks!

2018-02-23 12:14 GMT+03:00 vbm :

> Hi,
>
> I have a java program which is used to write to ignite cache and then into
> the persistant db.
> I have  created a client jar. I am trying to use this to interact with
> ignite servers on kubernetes.
> I have followed  this
>  web=1=rja=8=0ahUKEwiptbHM2LvZAhXHe7wKHaeNA
> eMQFggmMAA=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2Fkubernetes-
> deployment=AOvVaw0c802vvylyi4aFPEyagE9O>
> document to deploy ignite on kubernetes.
>
> As per my understanding, the client program jar needs to be present on the
> libs folder of all servers.
> I think this can be set using EXTERNAL_LIBS option. But in my case, ignite
> servers are already started.
>
> Do i need to restart now, for the changes to take affect ?
> Can I just place it in libs folder and servers will pick up the latest jars
> ?
>
>
> Regards,
> Vishwas
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: how to dismiss a ignite server node.

[Вячеслав Коптилин]

Hello,

If you don't have configured 3rd party cache store or backups,
then there is only one possible way, I think.
You need to
 - store all the data from the node you want to shut down,
 - shut down the node
 - upload stored data back to the cluster.

It can be done via an external database, for example.
If the data set is not huge, you can try to use a new partitioned cache
(with node filter/backups) or replicated cache.

Thanks.

2018-02-23 10:00 GMT+03:00 shawn.du :

> Hi,
> Suppose I have several ignite server nodes which compose a cluster.
> all data in PARTITIONED mode, and with no backups.
> It is possible to dismiss a node without restarting and data lose?
> if possible, what are the steps?
>
> Thanks
> Shawn
>
>

RE: Does object should implement Serializable?

[Stanislav Lukyanov]

Hi Ariel,

BinaryMarshaller is used to store data by default. All the data will be stored 
and sent across the cluster in the Ignite’s binary format.
You don’t need to use Serializable since BInaryMarshaller doesn’t rely on the 
standard Java serialization.
Also, there are cases when Ignite will not use BinaryMarshaller and fallback to 
other mechanisms. These cases are:
- If your class implements Externalizable
- If your class implements Serializable and defines custom 
writeObject/readObject methods
In these cases to continue using BinaryMarshaller with that class (and you 
generally want to use BinaryMarshaller everywhere) you need to implement 
Binarylizable interface.

Stan

From: Ariel Tubaltsev
Sent: 23 февраля 2018 г. 8:04
To: user@ignite.apache.org
Subject: Does object should implement Serializable?

Hi

I'm working with regular K,V cache:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html

I'm a bit puzzled whether objects that I want to put into cache should
implement Serializable.

Documentation(https://apacheignite.readme.io/docs/binary-marshaller) says
that by default values are stored deserialized. I'm wondering, how then they
are passed on the wire, between for example client and server? Does it
always use binary marshaller?

Thank you
Ariel



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite on Kubernetes: Question on external jars

[vbm]

Hi,

I have a java program which is used to write to ignite cache and then into
the persistant db.
I have  created a client jar. I am trying to use this to interact with
ignite servers on kubernetes.
I have followed  this

  
document to deploy ignite on kubernetes. 

As per my understanding, the client program jar needs to be present on the
libs folder of all servers. 
I think this can be set using EXTERNAL_LIBS option. But in my case, ignite
servers are already started. 

Do i need to restart now, for the changes to take affect ? 
Can I just place it in libs folder and servers will pick up the latest jars
?

 
Regards,
Vishwas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

SSD writes get slower and slower

[VT]

Hello, 

I am trying to test use SSD. But as more data are written, the write
performance drops significantly. Is there something I can do to solve this
problem? Or are there any known issues to this problem that might be fixed
in the upcoming versions? The test was done on a single node, locally. The
total data size is small, not enough to fill half of SSD.

Thanks!

For example, 

Loaded 1 keys 239ms.
Loaded 2 keys 122ms.
Loaded 3 keys 141ms.
Loaded 4 keys 110ms.

...

Loaded 842 keys 8068ms.
Loaded 843 keys 201ms.
Loaded 844 keys 187ms.
Loaded 845 keys 162ms.
Loaded 846 keys 182ms.
Loaded 847 keys 164ms.
Loaded 848 keys 171ms.
Loaded 849 keys 8475ms.

...

Loaded 8437 keys 5684ms.
Loaded 8438 keys 4329ms.
Loaded 8439 keys 5191ms.
Loaded 8440 keys 7352ms.
Loaded 8441 keys 11171ms.

...

Loaded 9994 keys 13352ms.
Loaded 9995 keys 28771ms.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/