External data sources

[Dmitri Bronnikov]

What's the best way to implement SQL that joins an Ignite cache with an
external data source, e.g. a Cassandra column family or a MySql table? A
very big external table, not one that can be cached in memory prior to
running the query.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/External-data-sources-tp11766.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Presenting Apache Ignite in Miami, May 2017

[Denis Magda]

Igniters,

I’m presenting two talks related to Apache Ignite in May at ASF Apache Con and 
Apache BigData conferences. Latest news: https://ignite.apache.org/news.html 


Direct links:
https://apachecon2017.sched.com/event/9zot?iframe=no 

https://apachebigdata2017.sched.com/event/A01a?iframe=no

Please help to promote the talks using your social channels. 

BTW, does anyone go to the conferences? It’s a good chance to meet with each 
other in person.

—
Denis

Re: Disable WriteBehind

[waterg]

Thank you for the reply.
Yes, I disabled both write through and write behind.
We're trying evaluate the application's performance on ignite and by taking
the persistent store out of equation, we thought the performance shall
improve, but on the contrary we saw performance dropped over 30%. What would
explain this kind of behavior?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Disable-WriteBehind-tp11729p11763.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Stoppin listen for topics in Ignite

[daniels]

Thank you very much.You helped me a lot. 
p.s. Ignite is very powerfull. :-)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Stoppin-listen-for-topics-in-Ignite-tp11604p11762.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Client near cache with Apache Flink

[nragon]

I think this kind of what i'm doing, in a abstract way as simple as possible

IgniteTest.java
  

Here I get this when executing IgniteTest:
gets: 3462000 - seconds past: 15 - rate: 230800 gets/s

If i run the same logic within flink:
gets: 6000 - seconds past: 26 - rate: 230 gets/s

Thanks



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11761.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Error in executing hadoop job using ignite

[Andrey Mashenkov]

Hi,

Please properly subscribe to the mailing list so that the community can
receive email notifications for your messages. To subscribe, send empty
email to user-subscr...@ignite.apache.org and follow simple instructions in
the reply.


It looks like you have outdated "objectweb-asm" jar library in classpath.


You wrote:

Hi,
I am using ignite hadoop accelerator and hdfs as secondry file system. But
when I submit job using ignite configuration it show following error.
Please tell if you feel anything wrong.
]$ hadoop --config ~/ignite_conf jar
/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
wordcount /PrashantSingh/1184-0.txt /output4tyy
Apr 04, 2017 3:33:30 PM
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection

INFO: Client TCP connection established: hmaster/10.202.17.60:11211
Apr 04, 2017 3:33:30 PM
org.apache.ignite.internal.client.impl.GridClientImpl 
INFO: Client started [id=1ce64156-1137-4a77-bed3-d32b962ce3c4, protocol=TCP]
2017-04-04 15:33:31,688 INFO [main] input.FileInputFormat
(FileInputFormat.java:listStatus(283)) - Total input paths to process : 1
2017-04-04 15:33:32,202 INFO [main] mapreduce.JobSubmitter
(JobSubmitter.java:submitJobInternal(198)) - number of splits:1
2017-04-04 15:33:33,222 INFO [main] mapreduce.JobSubmitter
(JobSubmitter.java:printTokens(287)) - Submitting tokens for job:
job_6f75490d-9038-43af-93ba-3d06081f65d2_0002
2017-04-04 15:33:33,445 INFO [main] mapreduce.Job (Job.java:submit(1294)) -
The url to track the job: N/A
2017-04-04 15:33:33,447 INFO [main] mapreduce.Job
(Job.java:monitorAndPrintJob(1339)) - Running job:
job_6f75490d-9038-43af-93ba-3d06081f65d2_0002
java.io.IOException: Job tracker doesn't have any information about the
job: job_6f75490d-9038-43af-93ba-3d06081f65d2_0002
at
org.apache.ignite.internal.processors.hadoop.impl.proto.HadoopClientProtocol.getJobStatus(HadoopClientProtocol.java:192)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:604)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1349)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1311)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


and at ignite node it shows following error
[15:33:33,408][ERROR][pub-#117%null%][HadoopJobTracker] Failed to submit
job: 6f75490d-9038-43af-93ba-3d06081f65d2_2
class org.apache.ignite.IgniteCheckedException: class
org.apache.ignite.IgniteException: null
at
org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2JobResourceManager.prepareJobEnvironment(HadoopV2JobResourceManager.java:169)
at
org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.initialize(HadoopV2Job.java:319)
at
org.apache.ignite.internal.processors.hadoop.jobtracker.HadoopJobTracker.job(HadoopJobTracker.java:1123)
at
org.apache.ignite.internal.processors.hadoop.jobtracker.HadoopJobTracker.submit(HadoopJobTracker.java:313)
at
org.apache.ignite.internal.processors.hadoop.HadoopProcessor.submit(HadoopProcessor.java:173)
at
org.apache.ignite.internal.processors.hadoop.HadoopImpl.submit(HadoopImpl.java:69)
at
org.apache.ignite.internal.processors.hadoop.proto.HadoopProtocolSubmitJobTask.run(HadoopProtocolSubmitJobTask.java:50)
at
org.apache.ignite.internal.processors.hadoop.proto.HadoopProtocolSubmitJobTask.run(HadoopProtocolSubmitJobTask.java:33)
at
org.apache.ignite.internal.processors.hadoop.proto.HadoopProtocolTaskAdapter$Job.execute(HadoopProtocolTaskAdapter.java:101)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:560)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6618)
at

Re: Distributed Closures VS Executor Service

[kmandalas]

Hello Christo,

Thank you for your prompt response. I will check your idea and get back to
you and the community.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11759.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed Closures VS Executor Service

[christos]

If I understand correctly, the SQL query is executed within every task on
each of the nodes and it is not set to be a local query. Correct?

If so then what you are really doing is executing the SQL queries from all
the nodes to all the nodes. This is bound to be inefficient.

In essence what you want to do is make your tasks work at a local level
only. Why not just switch to simple distributed closures and broadcast the
same task to all the nodes but configure the SQL query within the task to
execute as local one - setLocal(true). I mean you are already using
ComputeTaskSplitAdapter which abstracts the only difference between
closures: the capability to automatically assign jobs to nodes. Broadcasting
the same task with a local query means the same task would execute on all
the nodes in parallel and perform only a local query on the node it is
running on. Then your tasks can proceed to do the required calculations and
write the results etc. 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11758.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Disable WriteBehind

[Nikolai Tikhonov]

Seems I misunderstood your question. Did you disable both writeThrough and
writeBehind properties and got worse perfomance?

On Wed, Apr 5, 2017 at 4:39 PM, Nikolai Tikhonov 
wrote:

> Hi,
>
> Yes, it's expected behavior. If you disable writeBehind then each update
> will be propagated in underlying a store immediately and it's impact to
> performance. You can find more details in docs [1].
>
> 1. https://apacheignite.readme.io/docs/persistent-
> store#section-write-behind-caching
>
> On Wed, Apr 5, 2017 at 4:09 AM, waterg 
> wrote:
>
>> Hello all,
>>
>> I'm testing the effect of write behind and how it impacts the performance.
>> I found that if I simply disable write behind by setting writeThrough and
>> writeBehindEnabled property to false, my application performance is WORSE
>> than before. Is it something expected?
>>
>> Thank you!
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Disable-WriteBehind-tp11729.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: NameNode sync using DUAL_ASYNC mode on IGFS

[dkarachentsev]

Hi,

You may use it in cases when you use temporary files that should be removed
after node stop, or any other data that don't require persistence.

-Dmitry.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NameNode-sync-using-DUAL-ASYNC-mode-on-IGFS-tp11644p11756.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery With BinaryObject

[Andrey Mashenkov]

Hi David,

Thanks for reproducer. I've create a ticket [1] for this bug.

[1] https://issues.apache.org/jira/browse/IGNITE-4918

On Wed, Apr 5, 2017 at 5:03 PM, Andrey Mashenkov  wrote:

> I've creted a ticket [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4918
>
> On Wed, Apr 5, 2017 at 4:42 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Looks like a bug. Can you create a ticket?
>>
>> -Val
>>
>> On Wed, Apr 5, 2017 at 2:01 AM, Andrey Mashenkov <
>> andrey.mashen...@gmail.com
>> > wrote:
>>
>> > To reproduce
>> > - start standalone server
>> > - run test, it should work fine.
>> > - run test with changed filter code, it will return same results.
>> >
>> >
>> > On Wed, Apr 5, 2017 at 11:40 AM, Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com> wrote:
>> >
>> >> Can you provide the test?
>> >>
>> >> -Val
>> >>
>> >> On Wed, Apr 5, 2017 at 1:37 AM, Andrey Mashenkov <
>> >> andrey.mashen...@gmail.com
>> >> > wrote:
>> >>
>> >> > Hi Val,
>> >> >
>> >> > I run test with no filter class in server classpath. I've got an
>> error
>> >> > wiith peerClassLoading disabled, which is ok as server can't
>> unmarshal
>> >> > filter object.
>> >> > But with peerClassLoading enabled, query works fine but looks like
>> >> filter
>> >> > class won't be updated on server.
>> >> >
>> >> > On Wed, Apr 5, 2017 at 11:14 AM, Valentin Kulichenko <
>> >> > valentin.kuliche...@gmail.com> wrote:
>> >> >
>> >> > > Andrey,
>> >> > >
>> >> > > Marshaller cache does not store classes, only class names. So I'm
>> not
>> >> > sure
>> >> > > what you mean by this.
>> >> > >
>> >> > > I looked at the code and it seems I was wrong - peer class loading
>> >> does
>> >> > > work for scan query filter, which is good. But as I mentioned
>> before,
>> >> if
>> >> > > the class is available on local classpath, it will never be
>> >> dynamically
>> >> > > loaded, even if peer class loading is enabled. Therefore server
>> will
>> >> not
>> >> > > know about any changes happening to the class definition on the
>> >> client.
>> >> > > This is correct behavior.
>> >> > >
>> >> > > -Val
>> >> > >
>> >> > > On Wed, Apr 5, 2017 at 12:01 AM, Andrey Mashenkov <
>> >> > > andrey.mashen...@gmail.com> wrote:
>> >> > >
>> >> > > > Hi Val,
>> >> > > >
>> >> > > > Filter object serialized and send inside GridCacheQueryRequest,
>> and
>> >> > then
>> >> > > it
>> >> > > > is deserialized on server side and loaded by cache class loader.
>> >> > > > Looks like filter class is cached in marshaller cache.
>> >> > > >
>> >> > > >
>> >> > > > On Tue, Apr 4, 2017 at 4:43 PM, Valentin Kulichenko <
>> >> > > > valentin.kuliche...@gmail.com> wrote:
>> >> > > >
>> >> > > > > Andrey,
>> >> > > > >
>> >> > > > > To my knowledge, peer class loading is not supported for scan
>> >> query
>> >> > > > > filters, but I'm not sure though. Can you please check?
>> >> > > > >
>> >> > > > > Note that this actually doesn't matter if the class is
>> available
>> >> on
>> >> > > > > server's local classpath. In this case it will be always used
>> >> > > regardless
>> >> > > > of
>> >> > > > > any changes done on a client (i.e. will never be dynamically
>> >> loaded).
>> >> > > > This
>> >> > > > > is true for any functionality, including Compute Grid.
>> >> > > > >
>> >> > > > > -Val
>> >> > > > >
>> >> > > > > On Mon, Apr 3, 2017 at 10:19 AM, Andrey Mashenkov <
>> >> > > > > andrey.mashen...@gmail.com> wrote:
>> >> > > > >
>> >> > > > > > Crossposted to dev:
>> >> > > > > >
>> >> > > > > > Guys,
>> >> > > > > >
>> >> > > > > > ScanQuery filter code (see IgniteBiPredicate implementation
>> >> below)
>> >> > > can
>> >> > > > be
>> >> > > > > > cached on server side
>> >> > > > > > that can cause unexpected results.
>> >> > > > > > The main point here is server node never restarts while
>> client
>> >> does
>> >> > > it
>> >> > > > > with
>> >> > > > > > filter code changed.
>> >> > > > > >
>> >> > > > > > Is it ok?
>> >> > > > > >
>> >> > > > > > I try to add *serialVersionUID* and it was useless. The only
>> >> class
>> >> > > > > renaming
>> >> > > > > > was helpful.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > -- Forwarded message --
>> >> > > > > > From: David Li 
>> >> > > > > > Date: Mon, Apr 3, 2017 at 9:24 AM
>> >> > > > > > Subject: Re: ScanQuery With BinaryObject
>> >> > > > > > To: user@ignite.apache.org
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Sorry, please ignore the previous email, it was sent by
>> mistake.
>> >> > > > > >
>> >> > > > > > 1. I download apache-ignite-fabric-1.9.0-bin.zip, and unzip
>> it.
>> >> > > > > > 2. In terminal, I start an ignite instance by *bin/ignite.sh
>> >> > > > > > examples/config/example-ignite.xml*
>> >> > > > > > 3. I create a Java application, source code as below:
>> >> > > > > >
>> >> > > > > > public static void main(String[] args) {
>> >> > > > > > String ORG_CACHE = "org_cache_remote";
>> >> > > > > >* 

Re: Client near cache with Apache Flink

[nragon]

So, when flink as to process the first record it creates a latch telling the
other task managers(jvm) to wait until the cache is loaded.

//


  Ignite ignite = IgniteClient.ignite();
  IgniteCountDownLatch latch =
ignite.countDownLatch(this.cacheConfig.getName() + "-loadlatch", 1, false,
true);
  if (!cacheExists(ignite)) {
ignite.getOrCreateCache(this.cacheConfig, this.nearCacheConfig);
load(ignite);
latch.countDown();
  }
  latch.await();
  this.nearCache =
ignite.getOrCreateNearCache(this.cacheConfig.getName(),
this.nearCacheConfig);
} catch (Exception var3) {
  System.out.println(var3);
}

//


As testing purposes I'm calling the function for lookup but with fixed
Object[] when the key is TEST123

//


if(key.contains("TEST123")){
  this.matchRow = new Object[]{1, 1, 1};
}
else{
  this.matchRow = this.nearCache.get(key);
}

//


If I access ignite with this.nearCache.get(key) it drops my throughtput like
i mentioned before.
Let me know if more information is needed.

Thanks




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11754.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Asp.net client reconnect problem

[ozgurnevres]

log.txt   

Hi Pavel,
The log file (after ClientReconnected is fired while server is still down)
is attached.

thanks



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735p11752.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Disable WriteBehind

[Nikolai Tikhonov]

Hi,

Yes, it's expected behavior. If you disable writeBehind then each update
will be propagated in underlying a store immediately and it's impact to
performance. You can find more details in docs [1].

1.
https://apacheignite.readme.io/docs/persistent-store#section-write-behind-caching

On Wed, Apr 5, 2017 at 4:09 AM, waterg  wrote:

> Hello all,
>
> I'm testing the effect of write behind and how it impacts the performance.
> I found that if I simply disable write behind by setting writeThrough and
> writeBehindEnabled property to false, my application performance is WORSE
> than before. Is it something expected?
>
> Thank you!
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Disable-WriteBehind-tp11729.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Asp.net client reconnect problem

[ozgurnevres]

And yes, I am sure there's no other server node. (Topology shows 1 server ven
I restart the server)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735p11751.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Asp.net client reconnect problem

[ozgurnevres]

Hi, Pavel
The steps are below:

1- I am starting server node.
2- Then starting asp.net client node.
3- Querying database. Works perfect.
4- Stopping server node. Ignite fires ClientDisconnected as one expected.
5- A few seconds later, while server is still down, Ignite fires
ClientReconnected event (this is confusing, I don't know why).
6- After ClientReconnected fired, I restart the server node.
7- Nothing happens. It never fires ClientReconnected again.
8- When I query database in this situation, I get "Grid is in invalid state
to perform this operation. It either not started yet or has already being or
have stopped" exception.

I didn't implemented the logging yet, I'll read how to do it and then I'll
send you the logs.
Thanks



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735p11750.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache ignite persistant store implementation

[Andrey Mashenkov]

NearCache is designed to store small portion of frequently used data [1] of
non-local partitions.

[1] https://apacheignite.readme.io/docs/near-caches

On Wed, Apr 5, 2017 at 2:19 PM, ALEKSEY KUZNETSOV 
wrote:

> What meaning did u put into "is frontend to" ?
>
> вт, 4 апр. 2017 г. в 13:52, Andrey Mashenkov :
>
>> Client can have only LOCAL cache and NearCache that is frontend to
>> partitioned, see [1]. Moreover, client node requires live server node in
>> topology.
>>
>> However, you can try to force "server" mode on client node (via
>> *TcpDiscoverySpi.setForceServerMode(true)*) to start it without any
>> servers and use LOCAL cache with read-through.
>>
>> [1] https://apacheignite.readme.io/docs/clients-vs-
>> servers#creating-distributed-caches
>>
>> On Tue, Apr 4, 2017 at 10:56 AM, ALEKSEY KUZNETSOV <
>> alkuznetsov...@gmail.com> wrote:
>>
>> client node doesnt store cache entries, at all ?
>>
>> вт, 4 апр. 2017 г. в 9:42, Andrey Mashenkov :
>>
>> Hi Sweta,
>>
>> You should to start server node to Ignite could keep cache data in
>> memory.
>>
>> On Tue, Apr 4, 2017 at 8:43 AM, sweta Das  wrote:
>>
>> Hi
>> I have few clarifications on the read and write through implementation of
>> ignite cache.
>> I have a mysql database with a simple table (1 int key and 2 columns of
>> String).
>> I know that I have to start a ignite client node with cache store
>> configuration. But do I also need to start the server node passing the jdbc
>> driver connection details in the spring xml file?
>>
>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>> --
>>
>> *Best Regards,*
>>
>> *Kuznetsov Aleksey*
>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>



-- 
Best regards,
Andrey V. Mashenkov

Re: Non-collocated distributed SQL Joins across caches over separate cluster groups

[christos]

Thanks Sergi,

I understand that non-collocated distributed joins is a last resort very
well.

I still don't understand why cluster groups would make this any worse since
in distributed non-collocated joins the data is NOT expected to be on the
same node. Sounds to me that the cross-node calls would be almost the
same...

Christos



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Non-collocated-distributed-SQL-Joins-across-caches-over-separate-cluster-groups-tp11734p11748.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Stoppin listen for topics in Ignite

[vkulichenko]

This doesn't make a lot of sense, because "during stop" is not defined. From
your code standpoint, stopListen() is an atomic "immediate" operation, and
any message can be received before of after this operation. You should call
stopListen() only if you don't intend to receive messages anymore.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Stoppin-listen-for-topics-in-Ignite-tp11604p11747.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: data consistency in atomic mode

[dkarachentsev]

Yes, you're correct, in this case atomic cache could be inconsistent and
transactional cache will provide consistency guarantees.

-Dmitry.




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/data-consistency-in-atomic-mode-tp11719p11746.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache ignite persistant store implementation

[ALEKSEY KUZNETSOV]

What meaning did u put into "is frontend to" ?

вт, 4 апр. 2017 г. в 13:52, Andrey Mashenkov :

> Client can have only LOCAL cache and NearCache that is frontend to
> partitioned, see [1]. Moreover, client node requires live server node in
> topology.
>
> However, you can try to force "server" mode on client node (via
> *TcpDiscoverySpi.setForceServerMode(true)*) to start it without any
> servers and use LOCAL cache with read-through.
>
> [1]
> https://apacheignite.readme.io/docs/clients-vs-servers#creating-distributed-caches
>
> On Tue, Apr 4, 2017 at 10:56 AM, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com> wrote:
>
> client node doesnt store cache entries, at all ?
>
> вт, 4 апр. 2017 г. в 9:42, Andrey Mashenkov :
>
> Hi Sweta,
>
> You should to start server node to Ignite could keep cache data in memory.
>
> On Tue, Apr 4, 2017 at 8:43 AM, sweta Das  wrote:
>
> Hi
> I have few clarifications on the read and write through implementation of
> ignite cache.
> I have a mysql database with a simple table (1 int key and 2 columns of
> String).
> I know that I have to start a ignite client node with cache store
> configuration. But do I also need to start the server node passing the jdbc
> driver connection details in the spring xml file?
>
>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

Re: Asp.net client reconnect problem

[Pavel Tupitsyn]

> Grid is in invalid state to perform this operation

When does this happen? Can you provide more details or a project to
reproduce?
Are you sure you don't call Ignition.Stop somewhere?

On Wed, Apr 5, 2017 at 2:08 PM, ozgurnevres  wrote:

> While we send a query to our cluster, We get the following error message
> from
> asp.net client node
>
> "Grid is in invalid state to perform this operation. It either not started
> yet or has already being or have stopped [gridName=myGrid1, state=STOPPED]"
>
> Then we restart every thing on client node(app pool etc) and everything
> works fine.
> After i got this error, the client never connects again.
> Should I handle this error and how?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735p11743.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Asp.net client reconnect problem

[ozgurnevres]

While we send a query to our cluster, We get the following error message from
asp.net client node

"Grid is in invalid state to perform this operation. It either not started
yet or has already being or have stopped [gridName=myGrid1, state=STOPPED]"

Then we restart every thing on client node(app pool etc) and everything
works fine. 
After i got this error, the client never connects again. 
Should I handle this error and how?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735p11743.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed Closures VS Executor Service

[kmandalas]

Hi Christo,

I have tried the following:

- When app starts, I stream into PARTITIONED CACHE one (1) table needed for
the distributed jobs
- Now when a Task is submitted, each Job takes over some CATEGORYIDs.
Performs an Ignite cache-query (I paste it below), gathers the results, runs
calculations on them, transforms them to other objects and saves them to
some other smaller table in the Database. Finally it returns a success flag
true/false.

[13:29:09,520][WARN ][pub-#50%ppsp-cluster-IV%][IgniteH2Indexing] Query
execution is too long [time=5007 ms, sql='SELECT
"PPSP-IMDG-CACHE".SimulationInitialValues._KEY,
"PPSP-IMDG-CACHE".SimulationInitialValues._VAL FROM
"PPSP-IMDG-CACHE".SimulationInitialValues WHERE categoryId in ( 1, 2, 3, 4,
5, 6, 8 ) and geoChannelId in ( 3, 4, 5, 6 ) and type= 3 and week between
1888 and 1939', plan=
SELECT
"PPSP-IMDG-CACHE".SIMULATIONINITIALVALUES._KEY,
"PPSP-IMDG-CACHE".SIMULATIONINITIALVALUES._VAL
FROM "PPSP-IMDG-CACHE".SIMULATIONINITIALVALUES
/* "PPSP-IMDG-CACHE"."type_idx": TYPE = 3 */
WHERE ((WEEK >= 1888)
AND (WEEK <= 1939))
AND ((TYPE = 3)
AND ((CATEGORYID IN(1, 2, 3, 4, 5, 6, 8))
AND (GEOCHANNELID IN(3, 4, 5, 6
, parameters=[]]

Now, if I can align the CATEGORYIDs that each node has on its local cache
with the CATEGORYIDs that it receives along with a Job execution, then the
SQL query could be local-only. I have seen examples of affinityRun and
affinityCompute buf these are for distributed closures only. I would like to
do similar with ComputeTask approach because I need it in order to control
other things as well.

My *ComputeTask*is:
public class SimulationComputeTask extends
ComputeTaskSplitAdapter {
...
}

*SimulationMessage *is:

@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public class SimulationMessage implements Serializable {
private long projectId;
private long scenarioId;
private long[] categoryIds;
private long[] geoChannels;
private int channelType;
private int weekStart;
private int weekEnd;
}

 






--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11742.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Non-collocated distributed SQL Joins across caches over separate cluster groups

[Sergi Vladykin]

Hi,

Moreover distributed joins can be executed only between caches with the
same affinity (same partitions on the same nodes).

Keep in mind that distributed join is already a "last resort" thing and you
have to prefer collocated joins as much as possible, if you want to achieve
good performance. Distributed join between different cluster groups will
make things even worse.

Sergi

2017-04-05 12:02 GMT+03:00 Christos Erotocritou :

> Igniters,
>
> Is it correct to assume the following:
>
>- We have an Ignite cluster comprised of 2 cluster groups A & B that
>have different caches deployed.
>- We use an Ignite client to obtain API access to the whole cluster
>and execute a join query that joins data across the 2 caches
>
> My understanding is that this is *not possible*, correct?
>
> Reading this article [1
> ]
> it seems that such cross-cluster-group behaviour is supported with the
> transactions API and also advised.
>
> Any thoughts why the SQL API would not allow this and requires caches to
> be located on all nodes when the JOIN query is executed?
>
> Cheers,
> Christos
>

Re: Distributed Closures VS Executor Service

[Christos Erotocritou]

Hi Kyriako,

Agree with Nikolai on both points.

Regarding point number 2, you want to use affinity data / data colocation and 
data / compute colocation. You basically need to ensure that when you are 
inserting data into the cache you use the same affinity for all entries that 
need to be collocated to ensure they end up on the same server. Then you can 
use an affinity run or broadcast with local query enabled to only execute the 
query locally. Now in your case you said the data is in an external database so 
I’m not sure how you would ensure the data for each query is local.

Christos

> On 5 Apr 2017, at 11:00, kmandalas  wrote:
> 
> Hello Nikolai, all
> 
> About 1: yes, this was my perception as well, thanks for the confirmation
> 
> About 2: Even if all the nodes provide result sets of local execution to the
> query initiator, if we are talking about results being Lists containing a
> couple of thousands POJOs, then wouldn't be a big overhead these objects to
> be transferred over the Network? At least from some tests I perform, the
> response time is worse than querying the DB directly. Is there a way I make
> sure each node has all the data that will query locally? In combination with
> Compute Task Adapter always.
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11739.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed Closures VS Executor Service

[kmandalas]

Hello Nikolai, all

About 1: yes, this was my perception as well, thanks for the confirmation

About 2: Even if all the nodes provide result sets of local execution to the
query initiator, if we are talking about results being Lists containing a
couple of thousands POJOs, then wouldn't be a big overhead these objects to
be transferred over the Network? At least from some tests I perform, the
response time is worse than querying the DB directly. Is there a way I make
sure each node has all the data that will query locally? In combination with
Compute Task Adapter always.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11739.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Stoppin listen for topics in Ignite

[daniels]

I mean all  concurrently sending messages . During stop I want to know
wheather there are messages being done in process so let them be done.And i
will execute by hand the ones that just came.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Stoppin-listen-for-topics-in-Ignite-tp11604p11738.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Asp.net client reconnect problem

[Pavel Tupitsyn]

> a few seconds later fires "ClientReconnected" event
This is suspicious. Are you sure no other nodes are running on your machine
or in the local network?
Can you attach the log?

I have tested this exact scenario today (ASP.NET runs Ignite in client
mode, server is a standalone executable) and it works as expected.


On Wed, Apr 5, 2017 at 12:54 PM, ozgurnevres  wrote:

> Hi,
> I am running Ignite on asp.net as a client node. I have also a server node
> as console application.
> M problem is:
> 1- When I stop the server node when asp.net client node is connected, it
> fires first "ClientDisconnected" event. Then, a few seconds later fires
> "ClientReconnected" event. Why? The server is still down.
> 2- Then I restarting the server node. The client never reconnects again. In
> the server's console screen, the number of clients remains 0.
>
> Am I missing something? How can I correctly handle the "server temporarily
> down" or "server is restarting" case?
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Runtime error caught during grid runnable execution: GridWorker

[vkulichenko]

Different application instances are usually deployed with different class
loaders, so that they are completely independent from each other, including
any static content. For example, if there are two instances, they will not
know about each other's Ignite instances. Each of them should start Ignite
on its own.

Nature of your exceptions is not clear to me though. It looks like
classpaths interfere with each other somehow which is weird. I would check
JBOSS deployment model and settings.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Runtime-error-caught-during-grid-runnable-execution-GridWorker-tp11016p11736.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Asp.net client reconnect problem

[ozgurnevres]

Hi,
I am running Ignite on asp.net as a client node. I have also a server node
as console application.
M problem is:
1- When I stop the server node when asp.net client node is connected, it
fires first "ClientDisconnected" event. Then, a few seconds later fires
"ClientReconnected" event. Why? The server is still down.
2- Then I restarting the server node. The client never reconnects again. In
the server's console screen, the number of clients remains 0.

Am I missing something? How can I correctly handle the "server temporarily
down" or "server is restarting" case?







--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Asp-net-client-reconnect-problem-tp11735.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Non-collocated distributed SQL Joins across caches over separate cluster groups

[Christos Erotocritou]

Igniters,

Is it correct to assume the following:
We have an Ignite cluster comprised of 2 cluster groups A & B that have 
different caches deployed. 
We use an Ignite client to obtain API access to the whole cluster and execute a 
join query that joins data across the 2 caches 
My understanding is that this is not possible, correct? 

Reading this article [1 
]
 it seems that such cross-cluster-group behaviour is supported with the 
transactions API and also advised.

Any thoughts why the SQL API would not allow this and requires caches to be 
located on all nodes when the JOIN query is executed?

Cheers,
Christos

Re: Apache Ignite WriteBehind Guarantee during a Graceful Shutdown

[Andrey Mashenkov]

Hi Jaipal,

All data should be flushed to back store automatically on graceful shutdown.

On Wed, Apr 5, 2017 at 10:44 AM, Jaipal Reddy 
wrote:

> Hii,
>
> We are using Apache Ignite for caching and using its writeBehind feature
> for pushing updated data asynchronoulsy to Oracle DB. The following are
> relevant properties from our ignite cache configuration.
>
>
>
>
> * value="0"/>*
> With these properties, what I understand is IgniteCacheStore is invoked
> either after 512 ache entries are updated or at the end of 60 seconds(
> which ever is earlier). Say, I  initiate a graceful shutdown (not a server
> crash) of the Ignite Instance and I have 10 records which are updated in
> cache but are NOT YET updated to DB. During the graceful shutdown, will the
> IgniteCacheStore be automatically invoked and flush the 10 updated records
> to DB or will I lose the update on these 10 records ?
>
> If IgniteCacheStore is not automatically invoked during Graceful Shutdown,
> is there a way I can ensure that all my records are updated to the DB
> during the shutdown(say Cache Event Listeners or shutdown hooks etc..).
>
> Regards,
> Jaipal
>



-- 
Best regards,
Andrey V. Mashenkov

Apache Ignite WriteBehind Guarantee during a Graceful Shutdown

[Jaipal Reddy]

Hii,

We are using Apache Ignite for caching and using its writeBehind feature
for pushing updated data asynchronoulsy to Oracle DB. The following are
relevant properties from our ignite cache configuration.




**
With these properties, what I understand is IgniteCacheStore is invoked
either after 512 ache entries are updated or at the end of 60 seconds(
which ever is earlier). Say, I  initiate a graceful shutdown (not a server
crash) of the Ignite Instance and I have 10 records which are updated in
cache but are NOT YET updated to DB. During the graceful shutdown, will the
IgniteCacheStore be automatically invoked and flush the 10 updated records
to DB or will I lose the update on these 10 records ?

If IgniteCacheStore is not automatically invoked during Graceful Shutdown,
is there a way I can ensure that all my records are updated to the DB
during the shutdown(say Cache Event Listeners or shutdown hooks etc..).

Regards,
Jaipal