Task flow implementation question

[Pascoe Scholle]

Hello everyone,

So I am trying to put together a task flow chain. A task can have any
number of inputs and outputs, what I would call ports and each port has a
value stored in the cache.

A task can have numerous predecessors.

For example say I have two nodes which can be executed in parallel: one
generates a large 2d sparse array and saves this to the cache and the
second generates a vector which is also saved to cache. A successor is
linked to these two tasks, and has an input of type array and a second
input of type vector, looking like follows:

GEN_MAT (e.g. 15 seconds)  - >
  MAT_VEC_MUL - >
GEN_VEC(e.g. 1 second)   - >

As I have tried to show, the GEN_MAT takes a lot longer, MAT_MUL can only
execute once all input ports are set.

My question is how to implement this functionality effectively.
The input and output ports use a KEY:VALUE scheme, so with cache events,
all successor nodes can have their input ports listen for their KEY values
to be set, and it does work. But it feels very clunky. I am playing around
with setting attributes in ComputeTaskSession, but have not managed to get
it working. In a way cache events seem like the best option.

Any recommendations or ideas would be really helpful, I am very new to
apache ignite and just programming in general.

Thanks and kind regards,
Pascoe

Re: FW: class loading, peer class loading, jars, fun times in ignite

[Dave Harvey]

With these symptoms when the programmer understands the rules, there is
still a somewhat frequent bug where  "cache.withKeepBinary() "is what is
required, rather than simply "cache".

On Wed, May 29, 2019 at 12:29 PM Dmitriy Pavlov  wrote:

> Hi Scott,
>
> actually, users are encouraged to suggest edits to the documentation.
>
> Let me share
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document#HowtoDocument-Basics
>  page.
> You may sign up into readme.io and suggest edits.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 29 мая 2019 г. в 15:54, Scott Cote :
>
>> Ivan,
>>
>> I think that you gave me the right answer and confirmed my suspicion -
>> that peer class loading is only for just an executable and not data.
>>
>> Can I assist by helping edit the documentation on the Apache Ignite site
>> to add clarity on when a jar is needed in the lib folder?
>>
>> Also - I'll have to come up with a plan to flush out of date java data
>> classes, or maybe you guys have some techniques that allow for online
>> migration of v1 of a java data class to v2 of a java data class without
>> having to shutdown the whole set of vm's.
>>
>> Our system is pretty simple.
>>
>>
>> We use the cache's.
>>
>> Not the streamers.
>>
>> Not the callables ...
>>
>> It’s a big fantastic cache 
>>
>> (With some stuff that I built on top of it - like a priority queue
>> framework)
>>
>> SCott
>>
>> -Original Message-
>> From: Павлухин Иван 
>> Sent: Wednesday, May 29, 2019 1:16 AM
>> To: user@ignite.apache.org
>> Subject: Re: FW: class loading, peer class loading, jars, fun times in
>> ignite
>>
>> Hi Scott,
>>
>> As far as I know, peer class loading does not work for data classes
>> (which are stored in a cache). It works for tasks sended for execution
>> using IgniteCompute.
>>
>> It is only a partial answer. Could you describe your use case in more
>> details?
>>
>> вт, 28 мая 2019 г. в 23:35, Scott Cote :
>> >
>> > Whoops – sent to the wrong list …
>> >
>> >
>> >
>> > From: Scott Cote
>> > Sent: Tuesday, May 28, 2019 1:04 PM
>> > To: d...@ignite.apache.org
>> > Subject: class loading, peer class loading, jars, fun times in ignite
>> >
>> >
>> >
>> > I am fairly certain that I don’t know how to use peer class loading
>> properly.
>> >
>> >
>> >
>> > Am using Apache Ignite 2.7.  If I have a node running on 192.168.1.2
>> with a peer class loading enabled, and I start up a second node –
>> 192.168.1.3, client mode enabled and peer class loading enabled, then I
>> expected the following:
>> >
>> >
>> >
>> > Running the snippet (based on
>> https://apacheignite.readme.io/docs/getting-started#section-first-ignite-data-grid-application
>> ) on the client (192.168.1.3):
>> >
>> >
>> >
>> > try (Ignite ignite =
>> > Ignition.start("examples/config/example-ignite.xml")) {
>> >
>> > IgniteCache cache =
>> > ignite.getOrCreateCache("myCacheName");
>> >
>> >
>> >
>> > // Store keys in cache (values will end up on different cache
>> nodes).
>> >
>> > for (int i = 0; i < 10; i++)
>> >
>> > cache.put(i,new MyWrapperOfString( Integer.toString(i)));
>> >
>> >
>> >
>> > for (int i = 0; i < 10; i++)
>> >
>> > System.out.println("Got [key=" + i + ", val=" + cache.get(i) +
>> > ']');
>> >
>> > }
>> >
>> >
>> >
>> >
>> >
>> > Would cause the cache of “MyWrapperOfString” instances to be available
>> on 192.168.1.2 and on 192.168.1.3 .   Also be able to observe the cache
>> using visor, etc ….
>> >
>> >
>> >
>> > However – I instead get an error that the class “MyWrapperOfString” is
>> not available on 192.168.1.2.   Now if I take the jar that the class is
>> packed, and place it in the lib folder, all is happy.
>> >
>> >
>> >
>> > Should I have to do this?
>> >
>> > If yes – how do I update the jar if I have a cluster of nodes doing
>> this?   Do I have to shutdown the entire cluster in order to not have class
>> loader problems?
>> >
>> > I thought the peer class loading is supposed to solve this problem.
>> >
>> >
>> >
>> > I think it would be VERY INSTRUCTIVE for the snippet that I anchored to
>> not use a standard java library cache object, but to demonstrate the need
>> to package value object into a jar and stuff it into the lib folder (If
>> this is what is expected). Running lambdas that use basic java
>> primitives is cool, but is this the normal?
>> >
>> >
>> >
>> > Switching up …. Is there interest in me creating class loader that
>> would load java classes into the vm that could be incorporated into
>> ignite?   So instead of reading a jar, you load the class bytes into a
>> cache .  You want to hot load a new class?  Fine ! pump into the
>> DISTRIBUTED_CLASS_PATH_CACHE .
>> >
>> >
>> >
>> > Cheers.
>> >
>> >
>> >
>> > SCott
>> >
>> >
>>
>>
>>
>> --
>> Best regards,
>> Ivan Pavlukhin
>>
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the 

Re: FW: class loading, peer class loading, jars, fun times in ignite

[Ilya Kasnacheev]

Hello!

What kind of migrations are you expecting? I guess you could add or remove
fields just by doing rolling upgrade (restart nodes to V2 one by one until
no nodes with V1 are left).

Trying to rename fields is not recommended and changing fields' type is not
supported.

Regards,
-- 
Ilya Kasnacheev


ср, 29 мая 2019 г. в 15:54, Scott Cote :

> Ivan,
>
> I think that you gave me the right answer and confirmed my suspicion -
> that peer class loading is only for just an executable and not data.
>
> Can I assist by helping edit the documentation on the Apache Ignite site
> to add clarity on when a jar is needed in the lib folder?
>
> Also - I'll have to come up with a plan to flush out of date java data
> classes, or maybe you guys have some techniques that allow for online
> migration of v1 of a java data class to v2 of a java data class without
> having to shutdown the whole set of vm's.
>
> Our system is pretty simple.
>
>
> We use the cache's.
>
> Not the streamers.
>
> Not the callables ...
>
> It’s a big fantastic cache 
>
> (With some stuff that I built on top of it - like a priority queue
> framework)
>
> SCott
>
> -Original Message-
> From: Павлухин Иван 
> Sent: Wednesday, May 29, 2019 1:16 AM
> To: user@ignite.apache.org
> Subject: Re: FW: class loading, peer class loading, jars, fun times in
> ignite
>
> Hi Scott,
>
> As far as I know, peer class loading does not work for data classes (which
> are stored in a cache). It works for tasks sended for execution using
> IgniteCompute.
>
> It is only a partial answer. Could you describe your use case in more
> details?
>
> вт, 28 мая 2019 г. в 23:35, Scott Cote :
> >
> > Whoops – sent to the wrong list …
> >
> >
> >
> > From: Scott Cote
> > Sent: Tuesday, May 28, 2019 1:04 PM
> > To: d...@ignite.apache.org
> > Subject: class loading, peer class loading, jars, fun times in ignite
> >
> >
> >
> > I am fairly certain that I don’t know how to use peer class loading
> properly.
> >
> >
> >
> > Am using Apache Ignite 2.7.  If I have a node running on 192.168.1.2
> with a peer class loading enabled, and I start up a second node –
> 192.168.1.3, client mode enabled and peer class loading enabled, then I
> expected the following:
> >
> >
> >
> > Running the snippet (based on
> https://apacheignite.readme.io/docs/getting-started#section-first-ignite-data-grid-application
> ) on the client (192.168.1.3):
> >
> >
> >
> > try (Ignite ignite =
> > Ignition.start("examples/config/example-ignite.xml")) {
> >
> > IgniteCache cache =
> > ignite.getOrCreateCache("myCacheName");
> >
> >
> >
> > // Store keys in cache (values will end up on different cache nodes).
> >
> > for (int i = 0; i < 10; i++)
> >
> > cache.put(i,new MyWrapperOfString( Integer.toString(i)));
> >
> >
> >
> > for (int i = 0; i < 10; i++)
> >
> > System.out.println("Got [key=" + i + ", val=" + cache.get(i) +
> > ']');
> >
> > }
> >
> >
> >
> >
> >
> > Would cause the cache of “MyWrapperOfString” instances to be available
> on 192.168.1.2 and on 192.168.1.3 .   Also be able to observe the cache
> using visor, etc ….
> >
> >
> >
> > However – I instead get an error that the class “MyWrapperOfString” is
> not available on 192.168.1.2.   Now if I take the jar that the class is
> packed, and place it in the lib folder, all is happy.
> >
> >
> >
> > Should I have to do this?
> >
> > If yes – how do I update the jar if I have a cluster of nodes doing
> this?   Do I have to shutdown the entire cluster in order to not have class
> loader problems?
> >
> > I thought the peer class loading is supposed to solve this problem.
> >
> >
> >
> > I think it would be VERY INSTRUCTIVE for the snippet that I anchored to
> not use a standard java library cache object, but to demonstrate the need
> to package value object into a jar and stuff it into the lib folder (If
> this is what is expected). Running lambdas that use basic java
> primitives is cool, but is this the normal?
> >
> >
> >
> > Switching up …. Is there interest in me creating class loader that would
> load java classes into the vm that could be incorporated into ignite?   So
> instead of reading a jar, you load the class bytes into a cache .  You want
> to hot load a new class?  Fine ! pump into the DISTRIBUTED_CLASS_PATH_CACHE
> .
> >
> >
> >
> > Cheers.
> >
> >
> >
> > SCott
> >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Cache Preload from database using IgniteDatastreamer

[akurbanov]

There is no so called 'silver bullet' for data streamer parameters that will
fit in all or most cases, you should try tuning individual streamer
parameters to find out how it affects performance. Also it would be useful
to understand is there anything that could be done against GC pressure
(tuning GC on server nodes, tuning heap parameters etc.). There are just too
many unknown values to say what could be done exactly.

>And using spring data JPA gives me gc overhead and jvm pause issues. 
Not entirely sure how to understand this, do you have GC issues on client
node?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Compress binary object to save storage spac

[nadav.hoze]

Hello, 

 

Is there a way in ignite to compress big binary objects ? something like
cacheConfiguration.setCompressed(true)?

I'm using the latest ignite 2.7

 

 

Thanks,


Nadav

System Architect



+972-544821606
  DocAuthority.com

 

Re: Error Running Gridgain's LoadCaches java application

[Jay Fernandez]

This did stop the error from being logged.   However, when I start the
loadCaches program, nothing is logged and it seems to just hang.  The
ignite logs show that a client connected but nothing after that.  In
addition, the web console heap size monitoring jumps up right away and then
stops monitoring immediately after.

On Tue, May 28, 2019 at 9:42 AM Jay Fernandez 
wrote:

> Thanks for the reply Denis.  Is the correct way to disable the checker?
>
> 
>
> On Fri, May 24, 2019 at 5:59 PM Denis Magda  wrote:
>
>> Hi Jay,
>>
>> Could you please try to disable the "crtical workers checker"?
>>
>> https://apacheignite.readme.io/docs/critical-failures-handling#section-critical-workers-health-check
>>
>> It will be disabled by default in Ignite 2.7.5 since requires more
>> automation and tuning.
>>
>> Let us know if it doesn't work.
>>
>> -
>> Denis
>>
>>
>> On Fri, May 24, 2019 at 9:57 AM jay.fernandez 
>> wrote:
>>
>>> Hello, very new to Ignite and excited about using the application.  I
>>> have
>>> installed one Apache Ignite 2.7 node on a GCP VM.  I have the web agent
>>> running locally and I am using Gridgain's Web Console.  I am getting an
>>> error trying to run the LoadCaches java application that the Gridgain Web
>>> Console generated based on my MySQL database.
>>>
>>> Logs from Ignite Server:
>>>
>>> May 24 16:54:50 gdw-mysql57 service.sh[26542]: [16:54:50] Ignite node
>>> started OK (id=1b7f4add)
>>> May 24 16:54:50 gdw-mysql57 service.sh[26542]: [16:54:50] Topology
>>> snapshot
>>> [ver=1, locNode=1b7f4add, servers=1, clients=0, state=ACTIVE, CPUs=2,
>>> offheap=1.5GB, heap=1.0GB]
>>> May 24 16:55:03 gdw-mysql57 service.sh[26542]: [16:55:03] Topology
>>> snapshot
>>> [ver=2, locNode=1b7f4add, servers=1, clients=1, state=ACTIVE, CPUs=10,
>>> offheap=1.5GB, heap=8.1GB]
>>>
>>>
>>> Error from the Java project below, any help would be appreciated.
>>>
>>> May 24, 2019 12:53:02 PM java.util.logging.LogManager$RootLogger log
>>> WARNING: Failed to resolve default logging config file:
>>> config/java.util.logging.properties
>>> [12:53:02]__  
>>> [12:53:02]   /  _/ ___/ |/ /  _/_  __/ __/
>>> [12:53:02]  _/ // (7 7// /  / / / _/
>>> [12:53:02] /___/\___/_/|_/___/ /_/ /___/
>>> [12:53:02]
>>> [12:53:02] ver. 2.7.0#20181130-sha1:256ae401
>>> [12:53:02] 2018 Copyright(C) Apache Software Foundation
>>> [12:53:02]
>>> [12:53:02] Ignite documentation: http://ignite.apache.org
>>> [12:53:02]
>>> [12:53:02] Quiet mode.
>>> [12:53:02]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
>>> [12:53:02]   ^-- To see **FULL** console log here add
>>> -DIGNITE_QUIET=false
>>> or "-v" to ignite.{sh|bat}
>>> [12:53:02]
>>> [12:53:02] OS: Windows 10 10.0 amd64
>>> [12:53:02] VM information: Java(TM) SE Runtime Environment 1.8.0_201-b09
>>> Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.201-b09
>>> [12:53:02] Please set system property '-Djava.net.preferIPv4Stack=true'
>>> to
>>> avoid possible problems in mixed environments.
>>> [12:53:02] Initial heap size is 510MB (should be no less than 512MB, use
>>> -Xms512m -Xmx512m).
>>> [12:53:02] Configured plugins:
>>> [12:53:02]   ^-- None
>>> [12:53:02]
>>> [12:53:02] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
>>> [tryStop=false, timeout=0, super=AbstractFailureHandler
>>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED
>>> [12:53:03] Message queue limit is set to 0 which may lead to potential
>>> OOMEs
>>> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
>>> message queues growth on sender and receiver sides.
>>> [12:53:03] Security status [authentication=off, tls/ssl=off]
>>> [12:53:03] REST protocols do not start on client node. To start the
>>> protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
>>> property.
>>> log4j:WARN No appenders could be found for logger
>>> (org.springframework.beans.factory.support.DefaultListableBeanFactory).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>>> more info.
>>> May 24, 2019 12:53:18 PM org.apache.ignite.logger.java.JavaLogger error
>>> SEVERE: Blocked system-critical thread has been detected. This can lead
>>> to
>>> cluster-wide undefined behaviour [threadName=partition-exchanger,
>>> blockedFor=12s]
>>> May 24, 2019 12:53:18 PM java.util.logging.LogManager$RootLogger log
>>> SEVERE: Critical system error detected. Will be handled accordingly to
>>> configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
>>> timeout=0, super=AbstractFailureHandler
>>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
>>> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
>>> [name=partition-exchanger, igniteInstanceName=ImportedCluster,
>>> finished=false, heartbeatTs=1558716785615]]]
>>> class org.apache.ignite.IgniteException: GridWorker
>>> [name=partition-exchanger, 

Re: ignite work directory: binary_meta and marshaller

[akurbanov]

Yes, these directories should be available for the node, so you should mount
the storage with binary_meta and marshaller.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: FW: class loading, peer class loading, jars, fun times in ignite

[Dmitriy Pavlov]

Hi Scott,

actually, users are encouraged to suggest edits to the documentation.

Let me share
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document#HowtoDocument-Basics
page.
You may sign up into readme.io and suggest edits.

Sincerely,
Dmitriy Pavlov

ср, 29 мая 2019 г. в 15:54, Scott Cote :

> Ivan,
>
> I think that you gave me the right answer and confirmed my suspicion -
> that peer class loading is only for just an executable and not data.
>
> Can I assist by helping edit the documentation on the Apache Ignite site
> to add clarity on when a jar is needed in the lib folder?
>
> Also - I'll have to come up with a plan to flush out of date java data
> classes, or maybe you guys have some techniques that allow for online
> migration of v1 of a java data class to v2 of a java data class without
> having to shutdown the whole set of vm's.
>
> Our system is pretty simple.
>
>
> We use the cache's.
>
> Not the streamers.
>
> Not the callables ...
>
> It’s a big fantastic cache 
>
> (With some stuff that I built on top of it - like a priority queue
> framework)
>
> SCott
>
> -Original Message-
> From: Павлухин Иван 
> Sent: Wednesday, May 29, 2019 1:16 AM
> To: user@ignite.apache.org
> Subject: Re: FW: class loading, peer class loading, jars, fun times in
> ignite
>
> Hi Scott,
>
> As far as I know, peer class loading does not work for data classes (which
> are stored in a cache). It works for tasks sended for execution using
> IgniteCompute.
>
> It is only a partial answer. Could you describe your use case in more
> details?
>
> вт, 28 мая 2019 г. в 23:35, Scott Cote :
> >
> > Whoops – sent to the wrong list …
> >
> >
> >
> > From: Scott Cote
> > Sent: Tuesday, May 28, 2019 1:04 PM
> > To: d...@ignite.apache.org
> > Subject: class loading, peer class loading, jars, fun times in ignite
> >
> >
> >
> > I am fairly certain that I don’t know how to use peer class loading
> properly.
> >
> >
> >
> > Am using Apache Ignite 2.7.  If I have a node running on 192.168.1.2
> with a peer class loading enabled, and I start up a second node –
> 192.168.1.3, client mode enabled and peer class loading enabled, then I
> expected the following:
> >
> >
> >
> > Running the snippet (based on
> https://apacheignite.readme.io/docs/getting-started#section-first-ignite-data-grid-application
> ) on the client (192.168.1.3):
> >
> >
> >
> > try (Ignite ignite =
> > Ignition.start("examples/config/example-ignite.xml")) {
> >
> > IgniteCache cache =
> > ignite.getOrCreateCache("myCacheName");
> >
> >
> >
> > // Store keys in cache (values will end up on different cache nodes).
> >
> > for (int i = 0; i < 10; i++)
> >
> > cache.put(i,new MyWrapperOfString( Integer.toString(i)));
> >
> >
> >
> > for (int i = 0; i < 10; i++)
> >
> > System.out.println("Got [key=" + i + ", val=" + cache.get(i) +
> > ']');
> >
> > }
> >
> >
> >
> >
> >
> > Would cause the cache of “MyWrapperOfString” instances to be available
> on 192.168.1.2 and on 192.168.1.3 .   Also be able to observe the cache
> using visor, etc ….
> >
> >
> >
> > However – I instead get an error that the class “MyWrapperOfString” is
> not available on 192.168.1.2.   Now if I take the jar that the class is
> packed, and place it in the lib folder, all is happy.
> >
> >
> >
> > Should I have to do this?
> >
> > If yes – how do I update the jar if I have a cluster of nodes doing
> this?   Do I have to shutdown the entire cluster in order to not have class
> loader problems?
> >
> > I thought the peer class loading is supposed to solve this problem.
> >
> >
> >
> > I think it would be VERY INSTRUCTIVE for the snippet that I anchored to
> not use a standard java library cache object, but to demonstrate the need
> to package value object into a jar and stuff it into the lib folder (If
> this is what is expected). Running lambdas that use basic java
> primitives is cool, but is this the normal?
> >
> >
> >
> > Switching up …. Is there interest in me creating class loader that would
> load java classes into the vm that could be incorporated into ignite?   So
> instead of reading a jar, you load the class bytes into a cache .  You want
> to hot load a new class?  Fine ! pump into the DISTRIBUTED_CLASS_PATH_CACHE
> .
> >
> >
> >
> > Cheers.
> >
> >
> >
> > SCott
> >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: ignite work directory: binary_meta and marshaller

[akurbanov]

Yes, these directories should be on persistent directories that are mounted
in k8s.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Trouble with continuous queries

[Mike Needham]

I have tried various things, but it never fires for a change in the cache.
that is why I do not think it is set-up correctly


On Tue, May 28, 2019 at 9:30 AM Alexandr Shapkin  wrote:

> Hi,
>
> You can just save an instance of continuous query somewhere and dispose it
> when required
>
>
>   var listener = new SomeListener();
>   var someFilter= new
> CacheEntryEventFilter(_ignite.GetCluster().GetLocalNode(),
> Const.CacheName);
>   var query = new ContinuousQuery(listener, someFilter);
>
>//save the reference in a private field
>   _continuousQuery = cache.QueryContinuous(query);
>
>
> Regardless of a cache value.
> As I understand correctly you have a mixed platform solution (Java + .NET)
> This may lead to additional marshalling configuration.
>
> I will try the posted solution a bit later and reply
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
*Don't be afraid to be wrong. Don't be afraid to admit you don't have all
the answers. Don't be afraid to say "I think" instead of "I know."*

RE: FW: class loading, peer class loading, jars, fun times in ignite

[Scott Cote]

Ivan,

I think that you gave me the right answer and confirmed my suspicion - that 
peer class loading is only for just an executable and not data.

Can I assist by helping edit the documentation on the Apache Ignite site to add 
clarity on when a jar is needed in the lib folder?

Also - I'll have to come up with a plan to flush out of date java data classes, 
or maybe you guys have some techniques that allow for online migration of v1 of 
a java data class to v2 of a java data class without having to shutdown the 
whole set of vm's.

Our system is pretty simple. 


We use the cache's.

Not the streamers.

Not the callables ...

It’s a big fantastic cache 

(With some stuff that I built on top of it - like a priority queue framework)

SCott

-Original Message-
From: Павлухин Иван  
Sent: Wednesday, May 29, 2019 1:16 AM
To: user@ignite.apache.org
Subject: Re: FW: class loading, peer class loading, jars, fun times in ignite

Hi Scott,

As far as I know, peer class loading does not work for data classes (which are 
stored in a cache). It works for tasks sended for execution using IgniteCompute.

It is only a partial answer. Could you describe your use case in more details?

вт, 28 мая 2019 г. в 23:35, Scott Cote :
>
> Whoops – sent to the wrong list …
>
>
>
> From: Scott Cote
> Sent: Tuesday, May 28, 2019 1:04 PM
> To: d...@ignite.apache.org
> Subject: class loading, peer class loading, jars, fun times in ignite
>
>
>
> I am fairly certain that I don’t know how to use peer class loading properly.
>
>
>
> Am using Apache Ignite 2.7.  If I have a node running on 192.168.1.2 with a 
> peer class loading enabled, and I start up a second node – 192.168.1.3, 
> client mode enabled and peer class loading enabled, then I expected the 
> following:
>
>
>
> Running the snippet (based on 
> https://apacheignite.readme.io/docs/getting-started#section-first-ignite-data-grid-application
>  ) on the client (192.168.1.3):
>
>
>
> try (Ignite ignite = 
> Ignition.start("examples/config/example-ignite.xml")) {
>
> IgniteCache cache = 
> ignite.getOrCreateCache("myCacheName");
>
>
>
> // Store keys in cache (values will end up on different cache nodes).
>
> for (int i = 0; i < 10; i++)
>
> cache.put(i,new MyWrapperOfString( Integer.toString(i)));
>
>
>
> for (int i = 0; i < 10; i++)
>
> System.out.println("Got [key=" + i + ", val=" + cache.get(i) + 
> ']');
>
> }
>
>
>
>
>
> Would cause the cache of “MyWrapperOfString” instances to be available on 
> 192.168.1.2 and on 192.168.1.3 .   Also be able to observe the cache using 
> visor, etc ….
>
>
>
> However – I instead get an error that the class “MyWrapperOfString” is not 
> available on 192.168.1.2.   Now if I take the jar that the class is packed, 
> and place it in the lib folder, all is happy.
>
>
>
> Should I have to do this?
>
> If yes – how do I update the jar if I have a cluster of nodes doing this?   
> Do I have to shutdown the entire cluster in order to not have class loader 
> problems?
>
> I thought the peer class loading is supposed to solve this problem.
>
>
>
> I think it would be VERY INSTRUCTIVE for the snippet that I anchored to not 
> use a standard java library cache object, but to demonstrate the need to 
> package value object into a jar and stuff it into the lib folder (If this is 
> what is expected). Running lambdas that use basic java primitives is 
> cool, but is this the normal?
>
>
>
> Switching up …. Is there interest in me creating class loader that would load 
> java classes into the vm that could be incorporated into ignite?   So instead 
> of reading a jar, you load the class bytes into a cache .  You want to hot 
> load a new class?  Fine ! pump into the DISTRIBUTED_CLASS_PATH_CACHE .
>
>
>
> Cheers.
>
>
>
> SCott
>
>



--
Best regards,
Ivan Pavlukhin

Toxiproxy

[Delian]

Is anyone aware whether Toxiproxy can be set up to sit between Ignite nodes
in order to look at how things behave under various network conditions ? I
am new to Ignite and am wondering whether my thinking is flawed.

I have a simple 3 node cluster using the static TCP IP finder - all fine.
I've now set up Toxiproxy with a proxy (per node) for discovery,
communication, shared mem and timeserver as the config file for each node
allows me to explicitly set ports for these.  Finally, the ip finders in the
node configs point to the cluster nodes going through ToxiProxy - not
direct.

Nodes fire up but don't cluster, I'm seeing a lot of activity in Toxiproxy
console where by nodes are sending requests on ports other than the above 
(in some cases incrementing so I assume a range is being attempted). As I
have not explicitly set these up in Toxiproxy the requests seem to get
routed to the upstream node on 47500 (service disco) which is obviously
wrong in some cases. I see a number of open ports for the process - some of
which I have set but some not and they are not the same across the nodes. 

1) Can I statically set all these ports (even if I knew what they were) so I
can create proxies for them with the hope that allows me to cluster up ? 

2) I believe a ring topology is in play - are the hosts/ip's set up in the
service disco config always used, i.e. so everything goes through Toxiproxy
or is there the possibility they will connect direct and bypass ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Cache Preload from database using IgniteDatastreamer

[Om Thacker]

I have searched a lot to find out how to use IgniteDataStreamer to preload
cache from database, but all i couldn't find any. 
So I tried to do it myslef. Following is my approach:
1. A spring boot application having java based server configuration  for
cache - Ignite server application
2. A spring boot application having java based client configuration  for
cache - Ignite client application
3. I start the server app and then start the client app.
4. on startup of client app, after context is refreshed, i am loading ignite
cache by fetching data from database using spring data jpa.


@Autowired 
private Ignite ignite

public void preload(){
   try(IgniteDataStremer stmr =
ignite.dataStremer("UsersCache")){
   List usersList = userRepository.preload();
   for(Users users: usersList){
   stmr.addData(users.getUsersKey(),users);
   }
   }
}


I am using this process to load all the cache.There are total 14 caches
created and loaded like this.

The tables are having around 60 records.And using spring data JPA gives
me gc overhead and jvm pause issues.
I have read about data streamer that it is performance wise better, but i
don't know whether i am using it correctly.

Also i am not sure that loadcache() approach is better than datastreamer
approach.
Can anyone please help me on this issue.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture

[Ilya Kasnacheev]

Hello!

Do you happen to use SQL MVCC?
https://apacheignite.readme.io/docs/multiversion-concurrency-control#section-multiversion-concurrency-control

Looks like you've stumbled on a previously unknown issue. Can you please
file a ticket against Apache Ignite JIRA, with some description of steps to
reproduce?

Regards,
-- 
Ilya Kasnacheev


чт, 23 мая 2019 г. в 12:19, garima.j :

> Hello,
>
> We have a 3 node cluster in production for Apache Ignite version 2.7. RAM
> 128GB. A Spark streaming service (with thick Ignite clients) writes data
> into Ignite cache in a transaction (first get and then put).
> Now, 3 spark clients stopped and hence NODE_FAILED events were received. My
> Ignite crashed on one node and had the below stack trace :
>
> [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near
> enlist
> request from unknown node (will ignore) [txId=GridCacheVersion
> [topVer=169659586, order=1558471024158, nodeOrder=23],
> node=1be3bce3-7220-45bc-9863-4f16d97ea22b]
> [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager]
> Failed
> processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> msg=GridNearTxEnlistRequest [threadId=5872,
> futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false,
> miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
> lockVer=GridCacheVersion [topVer=169659586, order=1558471024158,
> nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240,
> crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176],
> cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000,
> taskNameHash=0, op=UPSERT, needRes=false]]
> java.lang.NullPointerException: null
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.(GridDhtTxAbstractEnlistFuture.java:237)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.(GridDhtTxEnlistFuture.java:84)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
> [ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [ignite-core-2.7.0.jar:2.7.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
> [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> 

Re: Efficiency of key queries

[Ilya Kasnacheev]

Hello!

- I don't think Ignite can get indexed columns' values from index. It will
load the key-value pair either way.
- Ignite stores key-value pairs in pages together, so when you iterate on
_key, all pages will be loaded into offheap.

I think that there is no easy solution for your use case.

Regards,
-- 
Ilya Kasnacheev


вт, 28 мая 2019 г. в 12:09, seiferma :

> Dear all,
>
> we are evaluating techniques for finding all keys contained in a cache. So
> far, we tried the most simple approaches mentioned in this mailing list
> like
> iterating over all cache entries or creating a SqlFieldsQuery asking for
> the
> _key column. While the results are correct, the performance is not
> satisfying if some cache content is not held in memory.
>
> Our setup is as follows: We got a small spring boot application that is
> running an embedded ignite instance. We manually activated the cluster
> consisting of this single node. We create all caches using the partitioned
> mode, so we can modify indexed fields during runtime by issuing sql
> statements. We are aware of the fact that using partitioned mode with a
> single node is useless but it should not do any harm. We limit the
> available
> heap memory of the application to one gigabyte and pushed more data to the
> cache than the heap memory can hold. Therefore, some cache entries are
> written to the hard drive. In addition, we enabled persistent storage to
> make the cache survive restarts.
>
> What we see is that the query as well as iterating cache entries takes a
> long time to complete. While running the queries, we could see some decent
> load on the hard drive. This makes sense for iterating cache entries but
> not
> so much for the query. We expected that the query just aks the database for
> the content of the (indexed?) _key column, which should not require the
> whole entity being loaded. This request should be pretty fast even if most
> of the entities are not available in the heap. With less data, the requests
> are faster than we could explain by the smaller amount of data.
>
> Could you give us some hints about what we could have done wrong or how we
> could retrieve all keys used in a cache in a more efficient way? If you
> need
> more information, I am keen to provide it.
>
> Best regards
> Stephan
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

ignite work directory: binary_meta and marshaller

[shivakumar]

Hi 
we have deployed ignite on kubernetes environment and we have set ignite
work directory (i.e., binary_meta and marshaller) to ephemeral disk which
will not be available after kubernetes node (pod) restart, we have seen
sometime "*class org.apache.ignite.binary.BinaryObjectException: Cannot find
metadata for object with compact footer*: -146362649 "
is this exception because we are setting ignite work directory where it
stores  *binary_meta* and *marshaller* to ephemeral disk and not persistence
disk ?
should we set work directory to mounted persistence volume in kubernetes ??

regards,
shiva



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: FW: class loading, peer class loading, jars, fun times in ignite

[Павлухин Иван]

Hi Scott,

As far as I know, peer class loading does not work for data classes
(which are stored in a cache). It works for tasks sended for execution
using IgniteCompute.

It is only a partial answer. Could you describe your use case in more details?

вт, 28 мая 2019 г. в 23:35, Scott Cote :
>
> Whoops – sent to the wrong list …
>
>
>
> From: Scott Cote
> Sent: Tuesday, May 28, 2019 1:04 PM
> To: d...@ignite.apache.org
> Subject: class loading, peer class loading, jars, fun times in ignite
>
>
>
> I am fairly certain that I don’t know how to use peer class loading properly.
>
>
>
> Am using Apache Ignite 2.7.  If I have a node running on 192.168.1.2 with a 
> peer class loading enabled, and I start up a second node – 192.168.1.3, 
> client mode enabled and peer class loading enabled, then I expected the 
> following:
>
>
>
> Running the snippet (based on 
> https://apacheignite.readme.io/docs/getting-started#section-first-ignite-data-grid-application
>  ) on the client (192.168.1.3):
>
>
>
> try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) {
>
> IgniteCache cache = 
> ignite.getOrCreateCache("myCacheName");
>
>
>
> // Store keys in cache (values will end up on different cache nodes).
>
> for (int i = 0; i < 10; i++)
>
> cache.put(i,new MyWrapperOfString( Integer.toString(i)));
>
>
>
> for (int i = 0; i < 10; i++)
>
> System.out.println("Got [key=" + i + ", val=" + cache.get(i) + ']');
>
> }
>
>
>
>
>
> Would cause the cache of “MyWrapperOfString” instances to be available on 
> 192.168.1.2 and on 192.168.1.3 .   Also be able to observe the cache using 
> visor, etc ….
>
>
>
> However – I instead get an error that the class “MyWrapperOfString” is not 
> available on 192.168.1.2.   Now if I take the jar that the class is packed, 
> and place it in the lib folder, all is happy.
>
>
>
> Should I have to do this?
>
> If yes – how do I update the jar if I have a cluster of nodes doing this?   
> Do I have to shutdown the entire cluster in order to not have class loader 
> problems?
>
> I thought the peer class loading is supposed to solve this problem.
>
>
>
> I think it would be VERY INSTRUCTIVE for the snippet that I anchored to not 
> use a standard java library cache object, but to demonstrate the need to 
> package value object into a jar and stuff it into the lib folder (If this is 
> what is expected). Running lambdas that use basic java primitives is 
> cool, but is this the normal?
>
>
>
> Switching up …. Is there interest in me creating class loader that would load 
> java classes into the vm that could be incorporated into ignite?   So instead 
> of reading a jar, you load the class bytes into a cache .  You want to hot 
> load a new class?  Fine ! pump into the DISTRIBUTED_CLASS_PATH_CACHE .
>
>
>
> Cheers.
>
>
>
> SCott
>
>



-- 
Best regards,
Ivan Pavlukhin