Re: Would this be crazy?

[nino martinez wael]

The idea are that we will have 2 nodes per call in a telephony setup..
A primary node for external service calls and a secondary node to
follow the call around the system (agent/humans)..

This however can amount a lot..

On Mon, Jan 18, 2016 at 2:40 PM, Alexey Kuznetsov
 wrote:
> Hi, Nino!
>
> I think it is quite crazy idea to setup 1000 concurrent clusters (if I
> understand you correctly).
> Starting 1000 nodes will be quite expensive.
>
> Could you please describe what business logic you are going to implement?
>
> It is not clear for me what are you expecting from Ignite and what Ignite
> components do you need.
>
> From first glance you need Ignite Compute (to do some computations with
> calls), but not need caching?
>
> If you need a compute only, I would recommend start 2-5 nodes and they will
> easily process such small number (1000) of tasks.
>
>
> On Mon, Jan 18, 2016 at 5:45 PM, nino martinez wael
>  wrote:
>>
>> Hi
>>
>> We are considering using Ignite as a backend for our client / server
>> environment (for Ignites key/value store). IT's a telephony based
>> system, and the idea are that as a telephony CALL passes through the
>> system Ignite nodes will follow the call adding and manipulating data.
>> Each call would have it's own node cluster (as we don't need to share
>> data between calls). There could be a lot of concurrent calls, max
>> would probably be around 1000.
>>
>> So the question are how does Ignite handle
>> * Cluster size of up to 1000 concurrent clusters (all based on a
>> predicate)
>> * A cluster where nodes will come and go and probably are error-prone
>> * Realtime, it would be a problem if one node dies and the others have
>> to wait for confirmation of death etc
>> * OSGI, any issues?
>>
>>
>> Please ask if I need to rephrase some of my questions
>>
>> --
>> Best regards / Med venlig hilsen
>> Nino Martinez
>
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com



-- 
Best regards / Med venlig hilsen
Nino Martinez

Execution hangs at IgniteRDD.savePairs()

[Shane Kinsella]

Hi All, I've created a StackOverflow post
(http://stackoverflow.com/questions/34815652/igniterdd-freezes-at-savepairs)
but thought I might share it here also.



I have a Spark cluster of three machines and am trying to use Apache Ignite
for caching data. On each Spark machine I have an Ignite node running and am
using the Spark REPL for testing (problem originally found using
Spark-submit so it not the REPL).

The problem is that my execution freezes at IgniteRDD.savePairs. Here is my
CacheConfig:






I previously had this working but ran out of memory in Ignite so I
(temporarily) added some options for tiered storage:


 












These were removed to try to debug the current issue. After adding these
changes was when savePairs stopped working. I have not found anything in the
logs.

Has anyone came across this issue, any work-arounds/solutions?

I believe there could be some hidden state involved. Is there a way to
restore my cluster (delete certain directory etc.)? I have restarted the
whole cluster numerous times.

Notes: HDFS is configured as the under filesystem. When I create my
IgniteContext (in the REPL) with an Ignite Node running on the same machine
I get a warning that the IGFS/IGFS-management endpoints already in use: I
have tested this with and without an Ignite node running on driver machine.

P.S. Here is the thread trace:

Frozen threads found (potential deadlock)

It seems that the following threads have not changed their stack for more
than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

shmem-worker-#93%null% <--- Frozen for at least 21 sec
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryUtils.readSharedMemory(long,
byte[], long, long, long) IpcSharedMemoryUtils.java (native)
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemorySpace.read(byte[],
int, int, long) IpcSharedMemorySpace.java:220
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryInputStream.read(byte[],
int, int) IpcSharedMemoryInputStream.java:62
org.apache.ignite.internal.util.ipc.IpcToNioAdapter.serve()
IpcToNioAdapter.java:114
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ShmemWorker.body()
TcpCommunicationSpi.java:2943
org.apache.ignite.internal.util.worker.GridWorker.run() GridWorker.java:110
java.lang.Thread.run() Thread.java:745

I see references to Shmem; FYI the endpoint that is configured is a TCP
endpoint.

P.P.S. Forget the 21 seconds above: the thread was still hung after an hour.




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Execution-hangs-at-IgniteRDD-savePairs-tp2615.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Would this be crazy?

[Alexey Kuznetsov]

Hi, Nino!

I think it is quite crazy idea to setup 1000 concurrent clusters (if I
understand you correctly).
Starting 1000 nodes will be quite expensive.

Could you please describe what business logic you are going to implement?

It is not clear for me what are you expecting from Ignite and what Ignite
components do you need.

>From first glance you need Ignite Compute (to do some computations with
calls), but not need caching?

If you need a compute only, I would recommend start 2-5 nodes and they will
easily process such small number (1000) of tasks.


On Mon, Jan 18, 2016 at 5:45 PM, nino martinez wael <
nino.martinez.w...@gmail.com> wrote:

> Hi
>
> We are considering using Ignite as a backend for our client / server
> environment (for Ignites key/value store). IT's a telephony based
> system, and the idea are that as a telephony CALL passes through the
> system Ignite nodes will follow the call adding and manipulating data.
> Each call would have it's own node cluster (as we don't need to share
> data between calls). There could be a lot of concurrent calls, max
> would probably be around 1000.
>
> So the question are how does Ignite handle
> * Cluster size of up to 1000 concurrent clusters (all based on a predicate)
> * A cluster where nodes will come and go and probably are error-prone
> * Realtime, it would be a problem if one node dies and the others have
> to wait for confirmation of death etc
> * OSGI, any issues?
>
>
> Please ask if I need to rephrase some of my questions
>
> --
> Best regards / Med venlig hilsen
> Nino Martinez
>



-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Would this be crazy?

[nino martinez wael]

Hi

We are considering using Ignite as a backend for our client / server
environment (for Ignites key/value store). IT's a telephony based
system, and the idea are that as a telephony CALL passes through the
system Ignite nodes will follow the call adding and manipulating data.
Each call would have it's own node cluster (as we don't need to share
data between calls). There could be a lot of concurrent calls, max
would probably be around 1000.

So the question are how does Ignite handle
* Cluster size of up to 1000 concurrent clusters (all based on a predicate)
* A cluster where nodes will come and go and probably are error-prone
* Realtime, it would be a problem if one node dies and the others have
to wait for confirmation of death etc
* OSGI, any issues?


Please ask if I need to rephrase some of my questions

-- 
Best regards / Med venlig hilsen
Nino Martinez

Re: Cache.put latencies high

[Denis Magda]

Hi,

If you're interested in the cache loading rather than in pure benchmarking
then I would suggest you using IgniteDataStreamer [1] as one of the fastest
way of a cache preloading.

Next if you're not going to use transaction then you can switch a cache to
ATOMIC mode. It will work faster.

Finally, it's always better to use cache.putAll() rather than cache.put().

However, if you're interested in pure benchmarking then you can refer to the
details of configs and benchmarks of Yardstick (official benchmarking
toolkit used by Ignite).

[1] https://apacheignite.readme.io/docs/data-loading#ignitedatastreamer
[2] https://github.com/apacheignite/yardstick-ignite





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Cache-put-latencies-high-tp2524p2611.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: issue with 2-node ignite server cluster

[Denis Magda]

Hi,

There are tons of the reasons why this can happen. When the Windows client
is started does it being to use the cluster somehow? If it does then your
code can lead to the failure of the server.

In any case please share the logs from all the nodes from us and the source
of the client code that is executed across the cluster.

Regards,
Denis 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/issue-with-2-node-ignite-server-cluster-tp2600p2609.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Using Ignite

[Murthy Kakarlamudi]

Thanks all for your responses. Will look into these alternatives.

On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke  wrote:

> You seem to look for streaming solutions , such as Spark Streaming or
> Flink Streaming or Storm
>
> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
> >
> > Kafka may suit your needs as a "queue" with producer/consumer and
> persistence capabilities also.
> >
> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
> >> Hi,
> >>   We have a scenario where in we have a c++ application that pumps out
> data ticks multiple times in a second. These data ticks needs to be
> displayed in the web front end. We need some middleware platform to hold
> those events temporarily before being consumed by web-front end. Can Ignite
> work as a viable middleware option in this scenario? In future we are
> expecting that there will be multiple consumers for these data ticks. Can
> Ignite Messaging be used in this scenario?
> >>
> >> Thanks,
> >> Satya.
> >
>

Re: How to get rid of the message( Received message without registered handler ) ?

[Denis Magda]

Ignite 1.6 should be released in Q1 2016. However it's up to the community to
make a decision on an actual release date. Usually a release happens when
some significant feature is implemented or many bugs are fixed.

However, GridGain quite often assembles its community releases, that are
based on Apache Ignite. As far as I know the next one should be released
this week and the fix for this bug should be there. You can track GridGain
community releases on this page
http://www.gridgain.com/download/editions/#community-edition

Regards,
Denis



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/CacheObjectImpl-pile-up-in-the-CacheContinuousQueryHandler-backupQueue-and-OOM-occurred-tp2412p2612.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Alternate way of using DBH2ServerStartup ??

[Ravi]

thanx now its working.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Alternate-way-of-using-DBH2ServerStartup-tp2599p2614.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Would this be crazy?

[Alexey Goncharuk]

Nino,

Can you please elaborate why you want to start several nodes per call in
your architecture? Generally, the number of nodes is a factor of number of
machines you have, in rare cases you may need to start an Ignite client per
external call, however, usually it is not very effective.

If you describe your use-case in more details, the community may suggest
right architecture.


AG

Ignite Cache best practices

[Kevin Daly]

I am building an ORM/Graph Database on top of Ignite.. Just wanted to know if
it is best practice on machines to close a cache after using it. Is it safe
to keep a map of handles to a bunch of classes? Or should we close or bound
the Ignite Caches?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-best-practices-tp2622.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Using Ignite

[Dood]

Kafka may suit your needs as a "queue" with producer/consumer and 
persistence capabilities also.


On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:

Hi,
   We have a scenario where in we have a c++ application that pumps 
out data ticks multiple times in a second. These data ticks needs to 
be displayed in the web front end. We need some middleware platform to 
hold those events temporarily before being consumed by web-front end. 
Can Ignite work as a viable middleware option in this scenario? In 
future we are expecting that there will be multiple consumers for 
these data ticks. Can Ignite Messaging be used in this scenario?


Thanks,
Satya.

Re: Using Ignite

[Denis Magda]

Hi Murthy,

It looks like that you can use Ignite Streaming feature [1] with
pre-configured sliding window [2] basing on what you mean under
"temporarily". One of the advantages of Ignite's sliding windows is that
you can query particular events using advanced SQL, text or other queries.

[1] https://apacheignite.readme.io/docs/streaming--cep
[2] https://apacheignite.readme.io/docs/sliding-windows

--
Denis

On Mon, Jan 18, 2016 at 8:50 PM, Murthy Kakarlamudi 
wrote:

> Thanks all for your responses. Will look into these alternatives.
>
> On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke 
> wrote:
>
>> You seem to look for streaming solutions , such as Spark Streaming or
>> Flink Streaming or Storm
>>
>> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
>> >
>> > Kafka may suit your needs as a "queue" with producer/consumer and
>> persistence capabilities also.
>> >
>> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
>> >> Hi,
>> >>   We have a scenario where in we have a c++ application that pumps out
>> data ticks multiple times in a second. These data ticks needs to be
>> displayed in the web front end. We need some middleware platform to hold
>> those events temporarily before being consumed by web-front end. Can Ignite
>> work as a viable middleware option in this scenario? In future we are
>> expecting that there will be multiple consumers for these data ticks. Can
>> Ignite Messaging be used in this scenario?
>> >>
>> >> Thanks,
>> >> Satya.
>> >
>>
>
>

Re: Working with Ignite Cache on Scala Play

[a d]

Thanks for the inputs Alexey.


From: Alexey Kuznetsov 
Sent: Thursday, January 14, 2016 8:22 PM
To: user@ignite.apache.org
Subject: Re: Working with Ignite Cache on Scala Play

Avi,

I think there is no need to start/stop nodes on each api interaction.
Start/stop nodes is a quite heavy and also will trigger rebalancing in caches.
I think you should just start several nodes only once.

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
[http://www.gridgain.com/wp-content/uploads/2015/04/enterprise_edition_box.png]

GridGain - In-Memory Data Fabric
www.gridgain.com
GridGain's In-Memory Data Fabric provides a comprehensive solution enabling 
speed and scale for transactional, analytical applications across any data store

Re: TcpDiscoveryVmIpFinder's isShared property

[Valentin Kulichenko]

Yakov,

JavaDoc is OK, but it seems to me that the confusion is caused by the fact
that setShared() method is placed on the IpFinderAdapter, while it actually
makes sense only for VmIpFinder. Can we deprecate it and always return
false from isShared() method for JDBC, S3 and others?

-Val

On Mon, Jan 18, 2016 at 1:56 AM, Yakov Zhdanov  wrote:

> Val, can you please review my changes to javadoc in master and update if
> necessary?
>
> --Yakov
>
> 2016-01-15 23:13 GMT+03:00 Andrey Kornev :
>
>> It does now! Thank you!
>>
>> Andrey
>>
>> --
>> Date: Fri, 15 Jan 2016 12:10:15 -0800
>> Subject: Re: TcpDiscoveryVmIpFinder's isShared property
>> From: valentin.kuliche...@gmail.com
>> To: user@ignite.apache.org
>>
>>
>> Andrey,
>>
>> Setting shared=true for TcpDiscoveryVmIpFinder means that nodes can
>> discover each other only within one JVM, when all nodes use the same
>> instance of IP finder. The shared "storage" in this case is just a local
>> collection. We use this heavily in unit tests, for example.
>>
>> TcpDiscoveryVmIpFinder with shared=false is how it's usually used. There
>> is no shared storage, so addresses have to be statically provided in the
>> configuration.
>>
>> shared=false for any other IP finder doesn't make any sense, because they
>> use some kind of storage by definition (e.g., JDBC or S3). This should be
>> clarified in docs.
>>
>> Makes sense?
>>
>> -Val
>>
>> On Fri, Jan 15, 2016 at 9:16 AM, Andrey Kornev 
>> wrote:
>>
>> Yakov,
>>
>> Thank you for the clarification, but I must admit I'm still not
>> completely out of the woods with respect to intended usage. Setting the
>> property to "false" seems to be most natural and only reasonable option,
>> and I wonder when would one want to set it to true? I must be missing
>> something.
>>
>> Also, just to clarify. In the last sentence you're saying: "This way user
>> doesn't have to list any IPs before start..." How would then the new nodes
>> know where to look for a node to connect to? They need to get the list of
>> seed nodes from somewhere, right? If so, then setting isShared to true
>> doesn't really make much difference - an initial list of seeds still must
>> be provided to every node.
>>
>> Thanks
>> Andrey
>>
>> --
>> Date: Thu, 14 Jan 2016 12:55:52 +0300
>> Subject: Re: TcpDiscoveryVmIpFinder's isShared property
>> From: yzhda...@apache.org
>> To: user@ignite.apache.org
>>
>>
>> Guys, this property is supported in VM IP finder for simplifying
>> discovery in single VM. I agree, that name could be better, but I would not
>> mess with it for now and just fix the javadocs (pls review, I did that in
>> master).
>>
>> "isShared" is a property of any IP finder. If it is "true" then IP finder
>> allows to add and remove addresses in runtime and this is how, for example,
>> S3 IP finder works. If "isShared" is "false" then IP finder is immutable
>> and all the addresses should be listed in configuration. This is the most
>> use case for VM IP finder. Since, usually VM IP finder is created per each
>> Ignite instance and all the known IPs are listed right away, but there is
>> also an option to make it shared - set "isShared" to true and literally
>> share it between local VM Ignite instances. This way user does not have to
>> list any IPs before start, instead all starting nodes add their addresses
>> to the finder, then get the registered addresses and continue with
>> discovery procedure.
>>
>> --Yakov
>>
>> 2016-01-13 22:45 GMT+03:00 Dmitriy Setrakyan :
>>
>> Any chance we could get an explanation here, so we can update the docs?
>> Yakov, I think you would know how this flag works.
>>
>> On Wed, Jan 13, 2016 at 11:40 AM, Vladimir Ozerov 
>> wrote:
>>
>> +1 to the question. Very confusing property. At the very least JavaDocs
>> should be reworked significantly.
>>
>> On Wed, Jan 13, 2016 at 8:32 PM, Andrey Kornev 
>> wrote:
>>
>> Hi there,
>>
>> I'm a bit confused about the purpose and usage of this property. What is
>> being shared with who? What are the consequences of setting the property to
>> true or false? Under what circumstances would one want to set it to either
>> true or false? Does one care at all?
>>
>> Thanks
>> Andrey
>>
>>
>>
>>
>>
>>
>

Runtime error at IgniteSpiThread

[Ambha]

I am using Ignite 1.4.0, Xmx is set to 5G and other JVM settings like
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC etc are also enabled. Apache Storm
cluster creates 5 client connections (client mode=true) to Ignite server
(single node). We also store the incoming data to Cassandra (it is running
on different system) by implementing CacheStore interface. The setting works
fine with 50,000 load. But when the input size increases to nearly 1 lakh
(.1 million) system repeatedly fails because of runtime error. The entire
stack trace is shown below.  

1. Is there any restriction on the number of clients which can be created
per ignite server node? five connections in my case are created by different
storm worker node
2.  The runtime error does not happen so often if I disable the cache store,
any reason for this
3. The client connections are opened and kept alive till the end, do you see
any side effect of this?
4. Is it better to create multimode ignite cluster (more than one server
node) for an input of 5-10 lakhs (1 million)
5. Please suggest any other way of connecting different components (storm,
ignite, Cassandra) which can produce the better result

stack trace**
[20:45:18,050][SEVERE][tcp-disco-msg-worker-#2%testGrid][TcpDiscoverySpi]
Runtime error caught during grid runnable execution: IgniteSpiThread
[name=tcp-disco-msg-worker-#2%testGrid]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDiscoveryDataReceived(GridCacheProcessor.java:1899)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$6.onExchange(GridDiscoveryManager.java:659)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1706)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:3310)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processJoinRequestMessage(ServerImpl.java:3002)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2095)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:5382)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Runtime-error-at-IgniteSpiThread-tp2630.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Cache best practices

[vkulichenko]

Hi Kevin,

I would say that it makes sense to call IgniteCache.close() only if you do
not intend to use this cache anymore on a particular client. If you're still
going to use it, it's absolutely fine to store the reference somewhere.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-best-practices-tp2622p2626.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Using Ignite

[Murthy Kakarlamudi]

Hi Denis,
 That kind of querying will be extremely helpful if it is supported.
Because in our use case, events sitting in the cache need to be queried
based on time as the criteria before it can be published to the clients.
Looks like the sliding window feature fits perfectly for this. Thanks for
sharing.



On Mon, Jan 18, 2016 at 2:27 PM, Denis Magda  wrote:

> Hi Murthy,
>
> It looks like that you can use Ignite Streaming feature [1] with
> pre-configured sliding window [2] basing on what you mean under
> "temporarily". One of the advantages of Ignite's sliding windows is that
> you can query particular events using advanced SQL, text or other queries.
>
> [1] https://apacheignite.readme.io/docs/streaming--cep
> [2] https://apacheignite.readme.io/docs/sliding-windows
>
> --
> Denis
>
> On Mon, Jan 18, 2016 at 8:50 PM, Murthy Kakarlamudi 
> wrote:
>
>> Thanks all for your responses. Will look into these alternatives.
>>
>> On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke 
>> wrote:
>>
>>> You seem to look for streaming solutions , such as Spark Streaming or
>>> Flink Streaming or Storm
>>>
>>> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
>>> >
>>> > Kafka may suit your needs as a "queue" with producer/consumer and
>>> persistence capabilities also.
>>> >
>>> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
>>> >> Hi,
>>> >>   We have a scenario where in we have a c++ application that pumps
>>> out data ticks multiple times in a second. These data ticks needs to be
>>> displayed in the web front end. We need some middleware platform to hold
>>> those events temporarily before being consumed by web-front end. Can Ignite
>>> work as a viable middleware option in this scenario? In future we are
>>> expecting that there will be multiple consumers for these data ticks. Can
>>> Ignite Messaging be used in this scenario?
>>> >>
>>> >> Thanks,
>>> >> Satya.
>>> >
>>>
>>
>>
>