Re: Ignite 1.6.0 binary - compilation Issues

[Agneeswaran]

Hi Igor,

Thanks a lot for your support.

Will apply this code changes.

With regards,
Agneeswaran



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-binary-compilation-Issues-tp5202p5266.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Affinity Key Mapping

[amitpa]

Hey Thanks for the reply.

I am copying the example here :- 

/ Instantiate person keys with the same company ID which is used as affinity
key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
 
Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);
 
// Both, the company and the person objects will be cached on the same node.
cache.put("myCompanyId", new Company(...));
cache.put(personKey1, p1);
cache.put(personKey2, p2);

>From the example it doesnt appear that we can put it in different caches.
See both are putting to the same cache.

Is there any exampel of using different caches for storing related domain
objects, how does Ignite figure out colocation in that case? Sorry for the
questions, but it got me confused.

Also how do you configure partitions for a cache?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

write behind question

[Zhengqingzheng]

Hi there,
I have one scenario that I need to make sure changes inside the cache need to 
be persisted before I do some other sql queries to database.
However when the cache is in "write behind" mode, I am not sure whether all 
data has been updated. So, is there any way to persist the date before calling 
sql to the database?


Best regards,
Kevin

Re: How do I know the cache rebalance is finished?

[Andrew]

Thanks, Val.

I want make each server node execute the cache data only for local primary
partitions.
If a node failed or joined then local primary partitions start changing.
I made the each server node holds processing during cache rebalance, after
cache rebalance is finished, check the changed local primary partitions then
starts new threads for newly added partitions and sweep objects out related
to missed partitions.

So, I wish I could detect the point of moment the cache rebalance has been
finished.
Is there any way for me to achieve it?

Sincerely,
Andrew.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-do-I-know-the-cache-rebalance-is-finished-tp5219p5263.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

IGFS YARN setup

[Haithem Turki]

Hello,

I'm interested in using IGFS as a Hadoop caching layer - the usecase
revolves largely around Spark jobs running on a YARN cluster that persist
data to S3 (although I have some non-Spark stuff running too so would
ideally integrate at the Hadoop filesystem layer). I'm excited about the
potential speedups that this could bring :)

I took a stab at deploying this for the first time, and had some questions:

- I ideally was envisioning deploying nodes via YARN to take advantage of
dynamic scaling and use any available memory on the cluster, I wanted to
make sure that this was indeed a supported workflow / on the roadmap as I
hit a few bumps along the way:
* I ended up needing to dump pretty much all of my Hadoop-related jars to
HDFS for my nodes to startup correctly (or else I was getting
ClassNotFoundExceptions ranging from guava to hadoop to asm to ignite
classes not being there). Am I doing something horribly wrong / have you
guys considered package a fat jar for the non-hadoop dependencies at least?
* Couldn't specify the yarn queue despite attempting to
set -Dmapreduce.job.queuename via IGNITE_JVM_OPTS variable (
https://issues.apache.org/jira/browse/IGNITE-2738?)
* Seems like dynamic allocation isn't supported? Wanted to get a sense of
whether this was in the roadmap
* Since YARN allocates containers at random it's pretty onerous to figure
out which hostnames have Ignite nodes running on them and specifying those
in the URL. For now I have TCP enabled (Ignite doesn't seem to die on port
conflicts if multiple nodes are running on the same machine) and I guess I
can set up a reverse proxy so that I can point towards a stable URL but
it's not great / doesn't scale well so I was wondering if there were other
suggestions on how to configure discovery (maybe spin up a local node
outside of YARN that leverages the cluster discovery?)
* I also wasn't clear on how cluster routing/balancing worked. If I specify
my hadoop jobs to point at host1:10500 via TCP, will all read/writes route
through that node or do the reads/writes somehow get balanced?

Or is this completely crazy / should I just deploy IGFS outside of YARN?

- Is there a way of configuring the local filesystem as a tiered storage
layer (or is it on the roadmap)? Usecase is that even reading from an SSD
is much faster than S3.

Thanks in advance!
- Haithem

Re: Connect H2 Console to a running cluster

[Sergi Vladykin]

No, this is impossible. Also keep in mind, that H2 debug console runs
queries locally without any distributive capabilities. To do the same you
can send closures to your cluster nodes and run local SQL queries.

But may be it is a good idea to be able to start the console on all the
nodes. This may be useful for development environment.

Sergi

2016-05-26 11:23 GMT-07:00 edwardkblk :

> The "-DIGNITE_H2_DEBUG_CONSOLE=true" option requires a startup of a local
> node.  My question is about the options to connect H2 Console to a remote
> cluster without starting a local node.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Connect-H2-Console-to-a-running-cluster-tp5242p5255.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Connect H2 Console to a running cluster

[Denis Magda]

H2 console doesn’t let you to work with the whole cluster. The cluster consists 
of many nodes each of which has its own instance of H2 engine running. H2 
console connects to a particular H2 server running on a concrete node and will 
operate over the dataset that is stored on the node only.

To execute SQL queries over Ignite cluster you should use GridGain Web Console
https://ignite.apache.org/addons.html#web-console 


—
Denis

> On May 26, 2016, at 9:23 PM, edwardkblk  
> wrote:
> 
> The "-DIGNITE_H2_DEBUG_CONSOLE=true" option requires a startup of a local
> node.  My question is about the options to connect H2 Console to a remote
> cluster without starting a local node.
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Connect-H2-Console-to-a-running-cluster-tp5242p5255.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Denis Magda]

Here is a link with rough estimation
https://apacheignite.readme.io/docs/capacity-planning 


—
Denis

> On May 26, 2016, at 8:09 PM, Tomek W  wrote:
> 
> | Make sure you have enough memory for your dataset.
> How to check it  ?
> 
> 2016-05-26 18:46 GMT+02:00 Alexei Scherbakov  >:
> You should measure performance on the real-life cases and see if it's enough 
> for you.
> Ignite performs good in both modes.
> If you really want to use ONHEAP_TIERED, you must tune GC and heap size, as 
> described here [1]
> Make sure you have enough memory for your dataset.
> The goal is to avoid long GC pauses.
> 
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning 
> 
> 
> 2016-05-26 19:40 GMT+03:00 Tomek W  >:
> Ok, I will try it. However, Why OFF_HEAP_TIERED ?  It seem to be not fast as 
> ON HEAP
> 
> 2016-05-26 18:32 GMT+02:00 Alexei Scherbakov  >:
> We are talking about count(*) query performance, right ?
> WriteBehind is for writing to CacheStore in the async mode.
> 
> If yes, do the following:
> 
> 1) Set OFFHEAP_TIERED mode and reduce max heap memory on example to 4Gb.
> 2) Update to Ignite 1.6
> 3) Measure query performance. Run the query several times and use average 
> value as the estimation.
> 4) If it's not as expected, show me GC logs.
> 
> 
> 
> 2016-05-26 18:28 GMT+03:00 Tomek W  >:
> No, I am using ON_HEAP_TIERED.
> 
> Maybe WriteBehind should be turned on ?
> My App do exactly one thing:  initialize hot loading.
> 
> When it comes to JDBC client, I did show fragment of code in previous post.
> 
> 2016-05-26 16:15 GMT+02:00 Alexei Scherbakov  >:
> I see long pauses in your GC log (> 3 seconds)
> This means your app have high pressure on the heap.
> It's hard to tell why without knowing what your app is doing.
> 
> Are you using OFFHEAP_TIERED?
> If yes, try to reduce sqlOnheapRowCacheSize value.
> 
> 
> 
> 
> 2016-05-26 14:57 GMT+03:00 Tomek W  >:
> Ok,
> i am going to add new machines to ignite cluster. Firstly, please look at my 
> gc file log - previous message.
> 
> 2016-05-26 13:39 GMT+02:00 Alexei Scherbakov  >:
> Hi,
> 
> The initial question was about setSqlOnheapRowCacheSize and I think
> now it is clear how to improve SQL performance using with parameter.
> 
> If you dissatisfied with the Ignite performance, I suggest you to start a new 
> thread on this,
> providing detailed info about your performance test like
> cluster configuration, server GC settings, and test sources.
> 
> As already mentioned, Ignite SQL engine(H2) has the same(or slightly) less 
> performance when Postresql.
> Ignite really starts to shine when used as distributed data grid having large 
> amount of data in memory on several nodes.
> 
> SELECT count(*) from table is not very good test query.
> Postgres may have the result cached, whereas Ignite always do the full table 
> traversal.
> Recently I implemented an improvement for this case.
> See https://issues.apache.org/jira/browse/IGNITE-2751 
>  for details.
> 
> I strongly recommend to test Ignite performance on the real case.
> Dont' forget to configure GC properly [1]
> 
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning 
> 
> 
> 
> 
> 
> 
> 
> 2016-05-26 2:09 GMT+03:00 Tomek W  >:
> | Also it would be interesting to see result of 
> | SELECT count(*) from the query above in both cases.
> (number of rows = 2 798 685)
> SELECT count(*) FROM postgresTable;
>  456 ms
> SELECT count(*) FROM postgresTable;
> 314 ms
> 
> SELECT count(*) FROM igniteTable;
> 9746 ms
> SELECT count(*) FROM igniteTable;
> 9664 ms
> 
> 
> Code of Jdbc Drvier (the same code for Ignite and postgresql - url connection 
> is given from command line):
> http://pastebin.com/mYDSjziN 
> My start sh file:
> http://pastebin.com/VmRM2sPQ 
> 
> My gc log file (following hint Magda):
> (file generated during hot loading and query via JDBC).
> http://pastebin.com/XicnNczV 
> 
> 
> If you would like to see something else let me know.
> 
> PS How to launch H2 debug console ? I followed docs, but it doesn't help.  
> I set enviroment variable:
> echo $IGNITE_H2_DEBUG_CONSOLE
> true
> now, ./ignite.sh conf.xml
> 
> sudo netstat -tulpn | grep 61214
> No 

Re: Two nodes in cluster

[Denis Magda]

How this topic is related to SQL queries? :)

I see that you’ve already started a bunch of SQL related topics on the user 
list. In short SQL queries can be executed via JDBC client or  using Java API. 

Refer to this page for more details - 
https://apacheignite.readme.io/docs/sql-queries 


In addition it makes sense to investigate CacheQueryExample that is delivered 
with Ignite releases
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java
 


—
Denis

> On May 26, 2016, at 7:18 PM, Tomek W  wrote:
> 
> Ok,
> Where should I follow my SQL queries (via JDBC interface) ?  
> I have two nodes with data - I would like to get full answer to SQL query.
> 
> 2016-05-26 10:08 GMT+02:00 Denis Magda  >:
> Ignite will automatically redistribute (rebalance) data across all available 
> nodes. So everything is scaling horizontally automatically.
> You just need to start a new node and portion of dataset will be rebalanced 
> on it.
> 
> —
> Denis
> 
>> On May 26, 2016, at 10:30 AM, Tomek W > > wrote:
>> 
>> Yes,
>> but I consider how to force ignite to distribute rows on other machines (not 
>> only this  machine that initialized hot loading).
>> 
>> 2016-05-26 8:46 GMT+02:00 Vladislav Pyatkov > >:
>> Hello,
>> 
>> If I understand correctly, you want to load data from a database in 
>> PARTITIONED type cache. This case is well described here
>> https://apacheignite.readme.io/docs/data-loading 
>> 
>> 
>> On Thu, May 26, 2016 at 1:45 AM, tomk > > wrote:
>> Hello,
>> At now I have only one node, but I am going to add more machines (in one
>> local network).
>> What should I know to do it ?
>> I have PARTITIONED type of cache.
>> Hot loading (my use case is caching postgresql) is done on one machine.
>> How to force ignite to distribute these rows to all machines - not only that
>> where client is executed (client initialize hot loading from postgresql).
>> 
>> 
>> 
>> 
>> 
>> --
>> View this message in context: 
>> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
>>  
>> 
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com 
>> .
>> 
>> 
>> 
>> -- 
>> Vladislav Pyatkov
>> 
> 
> 

Re: Exception during querying by jdbc - occurs on average once every 5 times

[Denis Magda]

> https://issues.apache.org/jira/browse/IGNITE-2751 
> 
> Does it mean that my SQL quries speed up ?

Yes, it does.

—
Denis

> On May 26, 2016, at 7:16 PM, Tomek W  wrote:
> 
> Ok, so firstly upgrade.
> If it don't solve problem, increase timeout (on new version too).
> 
> https://issues.apache.org/jira/browse/IGNITE-2751 
> 
> Does it mean that my SQL quries speed up ?
> 
> 2016-05-26 17:53 GMT+02:00 Denis Magda  >:
>> So what I should to do ?  
>> Increase timeout or try new version ?  
>> When it comes to increase timeout - is it possible to do it using xml config 
>> file?
> 
> Try switching to the new version first. And yes it’s possible to set 
> ConnectorConfiguration in XML.
> 
> —
> Denis
> 
>> On May 26, 2016, at 6:32 PM, Tomek W > > wrote:
>> 
>> | How long does it usually take to execute such query? I’m quite sure that 
>> you’re facing with an I/O timeout that terminates the connection between the 
>> client and server because query execution takes long time on the server side.
>> 
>> Yeah, I have the same guess.  
>> 
>> So what I should to do ?  
>> Increase timeout or try new version ?  
>> When it comes to increase timeout - is it possible to do it using xml config 
>> file?
>> 
>> 2016-05-26 15:53 GMT+02:00 Denis Magda > >:
>> Hi,
>> 
>> How long does it usually take to execute such query? I’m quite sure that 
>> you’re facing with an I/O timeout that terminates the connection between the 
>> client and server because query execution takes long time on the server side.
>> 
>> The first thing to do is to give a try to the latest Ignite release [1] that 
>> contains the fix for the issue [2] you most probably facing with.
>> The second thing to do is to increase ConnectorConfiguration.idleTimeout on 
>> the server side. This can be done in the following way:
>> 
>> IgniteConfiguration cfg = new IgniteConfiguration();
>> 
>> ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration();
>> 
>> connectorConfiguration.setIdleTimeout(30_000);
>> 
>> cfg.setConnectorConfiguration(connectorConfiguration);
>> 
>> [1] apache-ignite-fabric-1.6.0-bin.zip 
>> 
>> [2] https://issues.apache.org/jira/browse/IGNITE-2751 
>> 
>> 
>> —
>> Denis
>> 
>>> On May 25, 2016, at 3:14 AM, tomk >> > wrote:
>>> 
>>> Hello, 
>>> below, I attached some exception that I get during querying by JDBC client. 
>>> It occurs for SELECT * COUNT.
>>> It is very important that this exception occurs about once every three times
>>> - twice getting the correct result. 
>>> My cache has exactly 300 entries.
>>> 
>>> 
>>> java.sql.SQLException: Failed to query Ignite.
>>>at
>>> org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:122)
>>>at Tester.main(Tester.java:41)
>>> Caused by: class
>>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
>>> Failed to perform request (connection failed): /127.0.0.1:11212 
>>> 
>>>at
>>> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
>>>at
>>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:336)
>>>at
>>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:296)
>>>at
>>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:605)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:249)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:2115)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
>>>at
>>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:1659)
>>>at
>>> 

Re: Connect H2 Console to a running cluster

[edwardkblk]

The "-DIGNITE_H2_DEBUG_CONSOLE=true" option requires a startup of a local
node.  My question is about the options to connect H2 Console to a remote
cluster without starting a local node.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Connect-H2-Console-to-a-running-cluster-tp5242p5255.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite 1.6.0 binary - compilation Issues

[Igor Sapego]

Agneeswaran,

I have submitted pull request for your issue: [1]. It has not been accepted
yet
but it makes compilation possible in your case. So if it is critical for
you then
you can apply the same changes for your code.

[1] - https://github.com/apache/ignite/pull/759/files

Best Regards,
Igor

On Thu, May 26, 2016 at 4:11 PM, Igor Sapego  wrote:

> Agneeswaran,
>
> I've submitted a ticket for this issue: [1]. I'll reply you here once it's
> going to
> be resolved.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-3205
>
> Best Regards,
> Igor
>
> On Thu, May 26, 2016 at 2:48 PM, Agneeswaran <
> agneeswaran.ponnuraman...@nielsen.com> wrote:
>
>> Thanks Igor,
>>
>> Please find the compiler version details,
>>
>> g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>>
>> Thanks,
>> Agneeswaran
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-binary-compilation-Issues-tp5202p5233.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Tomek W]

| Make sure you have enough memory for your dataset.
How to check it  ?

2016-05-26 18:46 GMT+02:00 Alexei Scherbakov :

> You should measure performance on the real-life cases and see if it's
> enough for you.
> Ignite performs good in both modes.
> If you really want to use ONHEAP_TIERED, you must tune GC and heap size,
> as described here [1]
> Make sure you have enough memory for your dataset.
> The goal is to avoid long GC pauses.
>
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
>
> 2016-05-26 19:40 GMT+03:00 Tomek W :
>
>> Ok, I will try it. However, Why OFF_HEAP_TIERED ?  It seem to be not fast
>> as ON HEAP
>>
>> 2016-05-26 18:32 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> We are talking about count(*) query performance, right ?
>>> WriteBehind is for writing to CacheStore in the async mode.
>>>
>>> If yes, do the following:
>>>
>>> 1) Set OFFHEAP_TIERED mode and reduce max heap memory on example to 4Gb.
>>> 2) Update to Ignite 1.6
>>> 3) Measure query performance. Run the query several times and use
>>> average value as the estimation.
>>> 4) If it's not as expected, show me GC logs.
>>>
>>>
>>>
>>> 2016-05-26 18:28 GMT+03:00 Tomek W :
>>>
 No, I am using ON_HEAP_TIERED.

 Maybe WriteBehind should be turned on ?
 My App do exactly one thing:  initialize hot loading.

 When it comes to JDBC client, I did show fragment of code in previous
 post.

 2016-05-26 16:15 GMT+02:00 Alexei Scherbakov <
 alexey.scherbak...@gmail.com>:

> I see long pauses in your GC log (> 3 seconds)
> This means your app have high pressure on the heap.
> It's hard to tell why without knowing what your app is doing.
>
> Are you using OFFHEAP_TIERED?
> If yes, try to reduce sqlOnheapRowCacheSize value.
>
>
>
>
> 2016-05-26 14:57 GMT+03:00 Tomek W :
>
>> Ok,
>> i am going to add new machines to ignite cluster. Firstly, please
>> look at my gc file log - previous message.
>>
>> 2016-05-26 13:39 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> Hi,
>>>
>>> The initial question was about setSqlOnheapRowCacheSize and I think
>>> now it is clear how to improve SQL performance using with parameter.
>>>
>>> If you dissatisfied with the Ignite performance, I suggest you to
>>> start a new thread on this,
>>> providing detailed info about your performance test like
>>> cluster configuration, server GC settings, and test sources.
>>>
>>> As already mentioned, Ignite SQL engine(H2) has the same(or
>>> slightly) less performance when Postresql.
>>> Ignite really starts to shine when used as distributed data grid
>>> having large amount of data in memory on several nodes.
>>>
>>> SELECT count(*) from table is not very good test query.
>>> Postgres may have the result cached, whereas Ignite always do the
>>> full table traversal.
>>> Recently I implemented an improvement for this case.
>>> See https://issues.apache.org/jira/browse/IGNITE-2751 for details.
>>>
>>> I strongly recommend to test Ignite performance on the real case.
>>> Dont' forget to configure GC properly [1]
>>>
>>> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2016-05-26 2:09 GMT+03:00 Tomek W :
>>>
 | Also it would be interesting to see result of
 | SELECT count(*) from the query above in both cases.
 (number of rows = 2 798 685)
 SELECT count(*) FROM postgresTable;
  456 ms
 SELECT count(*) FROM postgresTable;
 314 ms

 SELECT count(*) FROM igniteTable;
 9746 ms
 SELECT count(*) FROM igniteTable;
 9664 ms


 Code of Jdbc Drvier (the same code for Ignite and postgresql - url
 connection is given from command line):
 http://pastebin.com/mYDSjziN
 My start sh file:
 http://pastebin.com/VmRM2sPQ

 My gc log file (following hint Magda):
 (file generated during hot loading and query via JDBC).
 http://pastebin.com/XicnNczV


 If you would like to see something else let me know.

 PS How to launch H2 debug console ? I followed docs, but it doesn't
 help.
 I set enviroment variable:
 echo $IGNITE_H2_DEBUG_CONSOLE
 true
 now, ./ignite.sh conf.xml

 sudo netstat -tulpn | grep 61214
 No opened ports.

 BTW, during starting ignite it give me information:
 [01:03:02]  Performance suggestions for grid
 'turbines_table_cluster' (fix if possible)
 [01:03:02] To disable, set
 

Re: Unexpected performance issue with SQL query followed by error

[Alexei Scherbakov]

I'll try to reproduce your case tomorrow using these cardinalities.
Meanwhile  try to execute the following query.

EXPLAIN ANALYZE SELECT DISTINCT * FROM "Activity".activity activity0
LEFT OUTER JOIN "Activity".activityuseraccountrole activityuseraccountrole0
ON activity0.activity_Id=activityuseraccountrole0.activity_Id

I suspect the IN condition is the culprit.

2016-05-26 17:26 GMT+03:00 jan.swaelens :

> And indeed, I am running with these options:
>
> /MEM_ARGS="-XX:PermSize=256m -Xms1g -Xmx8g -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:NewSize=128m -XX:MaxNewSize=128m
> -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60
> -XX:+DisableExplicitGC -DIGNITE_H2_DEBUG_CONSOLE=true"/
>
> thanks
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Unexpected-performance-issue-with-SQL-query-followed-by-error-tp4726p5240.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 

Best regards,
Alexei Scherbakov

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Alexei Scherbakov]

You should measure performance on the real-life cases and see if it's
enough for you.
Ignite performs good in both modes.
If you really want to use ONHEAP_TIERED, you must tune GC and heap size, as
described here [1]
Make sure you have enough memory for your dataset.
The goal is to avoid long GC pauses.

[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning

2016-05-26 19:40 GMT+03:00 Tomek W :

> Ok, I will try it. However, Why OFF_HEAP_TIERED ?  It seem to be not fast
> as ON HEAP
>
> 2016-05-26 18:32 GMT+02:00 Alexei Scherbakov  >:
>
>> We are talking about count(*) query performance, right ?
>> WriteBehind is for writing to CacheStore in the async mode.
>>
>> If yes, do the following:
>>
>> 1) Set OFFHEAP_TIERED mode and reduce max heap memory on example to 4Gb.
>> 2) Update to Ignite 1.6
>> 3) Measure query performance. Run the query several times and use average
>> value as the estimation.
>> 4) If it's not as expected, show me GC logs.
>>
>>
>>
>> 2016-05-26 18:28 GMT+03:00 Tomek W :
>>
>>> No, I am using ON_HEAP_TIERED.
>>>
>>> Maybe WriteBehind should be turned on ?
>>> My App do exactly one thing:  initialize hot loading.
>>>
>>> When it comes to JDBC client, I did show fragment of code in previous
>>> post.
>>>
>>> 2016-05-26 16:15 GMT+02:00 Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com>:
>>>
 I see long pauses in your GC log (> 3 seconds)
 This means your app have high pressure on the heap.
 It's hard to tell why without knowing what your app is doing.

 Are you using OFFHEAP_TIERED?
 If yes, try to reduce sqlOnheapRowCacheSize value.




 2016-05-26 14:57 GMT+03:00 Tomek W :

> Ok,
> i am going to add new machines to ignite cluster. Firstly, please look
> at my gc file log - previous message.
>
> 2016-05-26 13:39 GMT+02:00 Alexei Scherbakov <
> alexey.scherbak...@gmail.com>:
>
>> Hi,
>>
>> The initial question was about setSqlOnheapRowCacheSize and I think
>> now it is clear how to improve SQL performance using with parameter.
>>
>> If you dissatisfied with the Ignite performance, I suggest you to
>> start a new thread on this,
>> providing detailed info about your performance test like
>> cluster configuration, server GC settings, and test sources.
>>
>> As already mentioned, Ignite SQL engine(H2) has the same(or slightly)
>> less performance when Postresql.
>> Ignite really starts to shine when used as distributed data grid
>> having large amount of data in memory on several nodes.
>>
>> SELECT count(*) from table is not very good test query.
>> Postgres may have the result cached, whereas Ignite always do the
>> full table traversal.
>> Recently I implemented an improvement for this case.
>> See https://issues.apache.org/jira/browse/IGNITE-2751 for details.
>>
>> I strongly recommend to test Ignite performance on the real case.
>> Dont' forget to configure GC properly [1]
>>
>> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
>>
>>
>>
>>
>>
>>
>> 2016-05-26 2:09 GMT+03:00 Tomek W :
>>
>>> | Also it would be interesting to see result of
>>> | SELECT count(*) from the query above in both cases.
>>> (number of rows = 2 798 685)
>>> SELECT count(*) FROM postgresTable;
>>>  456 ms
>>> SELECT count(*) FROM postgresTable;
>>> 314 ms
>>>
>>> SELECT count(*) FROM igniteTable;
>>> 9746 ms
>>> SELECT count(*) FROM igniteTable;
>>> 9664 ms
>>>
>>>
>>> Code of Jdbc Drvier (the same code for Ignite and postgresql - url
>>> connection is given from command line):
>>> http://pastebin.com/mYDSjziN
>>> My start sh file:
>>> http://pastebin.com/VmRM2sPQ
>>>
>>> My gc log file (following hint Magda):
>>> (file generated during hot loading and query via JDBC).
>>> http://pastebin.com/XicnNczV
>>>
>>>
>>> If you would like to see something else let me know.
>>>
>>> PS How to launch H2 debug console ? I followed docs, but it doesn't
>>> help.
>>> I set enviroment variable:
>>> echo $IGNITE_H2_DEBUG_CONSOLE
>>> true
>>> now, ./ignite.sh conf.xml
>>>
>>> sudo netstat -tulpn | grep 61214
>>> No opened ports.
>>>
>>> BTW, during starting ignite it give me information:
>>> [01:03:02]  Performance suggestions for grid
>>> 'turbines_table_cluster' (fix if possible)
>>> [01:03:02] To disable, set
>>> -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
>>> [01:03:02]   ^-- Disable grid events (remove 'includeEventTypes'
>>> from configuration)
>>> [01:03:02]   ^-- Enable ATOMIC mode if not using transactions (set
>>> 'atomicityMode' to ATOMIC)
>>> [01:03:02]   ^-- Enable write-behind 

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Alexei Scherbakov]

We are talking about count(*) query performance, right ?
WriteBehind is for writing to CacheStore in the async mode.

If yes, do the following:

1) Set OFFHEAP_TIERED mode and reduce max heap memory on example to 4Gb.
2) Update to Ignite 1.6
3) Measure query performance. Run the query several times and use average
value as the estimation.
4) If it's not as expected, show me GC logs.



2016-05-26 18:28 GMT+03:00 Tomek W :

> No, I am using ON_HEAP_TIERED.
>
> Maybe WriteBehind should be turned on ?
> My App do exactly one thing:  initialize hot loading.
>
> When it comes to JDBC client, I did show fragment of code in previous post.
>
> 2016-05-26 16:15 GMT+02:00 Alexei Scherbakov  >:
>
>> I see long pauses in your GC log (> 3 seconds)
>> This means your app have high pressure on the heap.
>> It's hard to tell why without knowing what your app is doing.
>>
>> Are you using OFFHEAP_TIERED?
>> If yes, try to reduce sqlOnheapRowCacheSize value.
>>
>>
>>
>>
>> 2016-05-26 14:57 GMT+03:00 Tomek W :
>>
>>> Ok,
>>> i am going to add new machines to ignite cluster. Firstly, please look
>>> at my gc file log - previous message.
>>>
>>> 2016-05-26 13:39 GMT+02:00 Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com>:
>>>
 Hi,

 The initial question was about setSqlOnheapRowCacheSize and I think
 now it is clear how to improve SQL performance using with parameter.

 If you dissatisfied with the Ignite performance, I suggest you to start
 a new thread on this,
 providing detailed info about your performance test like
 cluster configuration, server GC settings, and test sources.

 As already mentioned, Ignite SQL engine(H2) has the same(or slightly)
 less performance when Postresql.
 Ignite really starts to shine when used as distributed data grid having
 large amount of data in memory on several nodes.

 SELECT count(*) from table is not very good test query.
 Postgres may have the result cached, whereas Ignite always do the full
 table traversal.
 Recently I implemented an improvement for this case.
 See https://issues.apache.org/jira/browse/IGNITE-2751 for details.

 I strongly recommend to test Ignite performance on the real case.
 Dont' forget to configure GC properly [1]

 [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning






 2016-05-26 2:09 GMT+03:00 Tomek W :

> | Also it would be interesting to see result of
> | SELECT count(*) from the query above in both cases.
> (number of rows = 2 798 685)
> SELECT count(*) FROM postgresTable;
>  456 ms
> SELECT count(*) FROM postgresTable;
> 314 ms
>
> SELECT count(*) FROM igniteTable;
> 9746 ms
> SELECT count(*) FROM igniteTable;
> 9664 ms
>
>
> Code of Jdbc Drvier (the same code for Ignite and postgresql - url
> connection is given from command line):
> http://pastebin.com/mYDSjziN
> My start sh file:
> http://pastebin.com/VmRM2sPQ
>
> My gc log file (following hint Magda):
> (file generated during hot loading and query via JDBC).
> http://pastebin.com/XicnNczV
>
>
> If you would like to see something else let me know.
>
> PS How to launch H2 debug console ? I followed docs, but it doesn't
> help.
> I set enviroment variable:
> echo $IGNITE_H2_DEBUG_CONSOLE
> true
> now, ./ignite.sh conf.xml
>
> sudo netstat -tulpn | grep 61214
> No opened ports.
>
> BTW, during starting ignite it give me information:
> [01:03:02]  Performance suggestions for grid 'turbines_table_cluster'
> (fix if possible)
> [01:03:02] To disable, set
> -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> [01:03:02]   ^-- Disable grid events (remove 'includeEventTypes' from
> configuration)
> [01:03:02]   ^-- Enable ATOMIC mode if not using transactions (set
> 'atomicityMode' to ATOMIC)
> [01:03:02]   ^-- Enable write-behind to persistent store (set
> 'writeBehindEnabled' to true)
>
>
> 2016-05-25 12:23 GMT+02:00 Alexei Scherbakov <
> alexey.scherbak...@gmail.com>:
>
>> For postgres test I mean initial jdbc query and result set traversal.
>> For Ignite I mean sql query and iterator traversal.
>> Also it would be interesting to see result of
>> *SELECT count(*) from the query above in both cases.*
>>
>> 2016-05-25 12:00 GMT+03:00 Tomek W :
>>
>>> [image: Obraz w treści 1]
>>>
>>> What code do you mean ? JDBC client ?
>>>
>>> 2016-05-25 10:25 GMT+02:00 Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com>:
>>>
 What's the batch size for postgresql ?
 What's the size of one entry ?
 Could you provide the test code for both postgres and Ignite (just

Re: Two nodes in cluster

[Tomek W]

Ok,
Where should I follow my SQL queries (via JDBC interface) ?
I have two nodes with data - I would like to get full answer to SQL query.

2016-05-26 10:08 GMT+02:00 Denis Magda :

> Ignite will automatically redistribute (rebalance) data across all
> available nodes. So everything is scaling horizontally automatically.
> You just need to start a new node and portion of dataset will be
> rebalanced on it.
>
> —
> Denis
>
> On May 26, 2016, at 10:30 AM, Tomek W  wrote:
>
> Yes,
> but I consider how to force ignite to *distribute* rows on other machines
> (*not only this  machine that initialized hot loading*).
>
> 2016-05-26 8:46 GMT+02:00 Vladislav Pyatkov :
>
>> Hello,
>>
>> If I understand correctly, you want to load data from a database in
>> PARTITIONED type cache. This case is well described here
>> https://apacheignite.readme.io/docs/data-loading
>>
>> On Thu, May 26, 2016 at 1:45 AM, tomk  wrote:
>>
>>> Hello,
>>> At now I have only one node, but I am going to add more machines (in one
>>> local network).
>>> What should I know to do it ?
>>> I have PARTITIONED type of cache.
>>> Hot loading (my use case is caching postgresql) is done on one machine.
>>> How to force ignite to distribute these rows to all machines - not only
>>> that
>>> where client is executed (client initialize hot loading from postgresql).
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com
>>> .
>>>
>>
>>
>>
>> --
>> Vladislav Pyatkov
>>
>
>
>

Re: Exception during querying by jdbc - occurs on average once every 5 times

[Tomek W]

Ok, so firstly upgrade.
If it don't solve problem, increase timeout (on new version too).

https://issues.apache.org/jira/browse/IGNITE-2751
Does it mean that my SQL quries speed up ?

2016-05-26 17:53 GMT+02:00 Denis Magda :

> So what I should to do ?
> Increase timeout or try new version ?
> When it comes to increase timeout - is it possible to do it using xml
> config file?
>
>
> Try switching to the new version first. And yes it’s possible to set
> ConnectorConfiguration in XML.
>
> —
> Denis
>
> On May 26, 2016, at 6:32 PM, Tomek W  wrote:
>
> | How long does it usually take to execute such query? I’m quite sure that
> you’re facing with an I/O timeout that terminates the connection between
> the client and server because query execution takes long time on the server
> side.
>
> Yeah, I have the same guess.
>
> So what I should to do ?
> Increase timeout or try new version ?
> When it comes to increase timeout - is it possible to do it using xml
> config file?
>
> 2016-05-26 15:53 GMT+02:00 Denis Magda :
>
>> Hi,
>>
>> How long does it usually take to execute such query? I’m quite sure that
>> you’re facing with an I/O timeout that terminates the connection between
>> the client and server because query execution takes long time on the server
>> side.
>>
>> The first thing to do is to give a try to the latest Ignite release [1]
>> that contains the fix for the issue [2] you most probably facing with.
>> The second thing to do is to increase ConnectorConfiguration.idleTimeout
>> on the server side. This can be done in the following way:
>>
>>
>> IgniteConfiguration cfg = new IgniteConfiguration();
>>
>> ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration();
>>
>> connectorConfiguration.setIdleTimeout(30_000);
>>
>> cfg.setConnectorConfiguration(connectorConfiguration);
>>
>>
>> [1] apache-ignite-fabric-1.6.0-bin.zip
>> 
>> [2] https://issues.apache.org/jira/browse/IGNITE-2751
>>
>> —
>> Denis
>>
>> On May 25, 2016, at 3:14 AM, tomk  wrote:
>>
>> Hello,
>> below, I attached some exception that I get during querying by JDBC
>> client.
>> It occurs for SELECT * COUNT.
>> It is very important that this exception occurs about once every three
>> times
>> - twice getting the correct result.
>> My cache has exactly 300 entries.
>>
>>
>> java.sql.SQLException: Failed to query Ignite.
>>at
>>
>> org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:122)
>>at Tester.main(Tester.java:41)
>> Caused by: class
>>
>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
>> Failed to perform request (connection failed): /127.0.0.1:11212
>>at
>>
>> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
>>at
>>
>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:336)
>>at
>>
>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:296)
>>at
>>
>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:605)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:249)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:2115)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:1659)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:731)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1463)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1398)
>>at
>>
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1280)
>>at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>at java.lang.Thread.run(Thread.java:745)
>> ?
>>
>>
>>
>>

Re: Exception during querying by jdbc - occurs on average once every 5 times

[Denis Magda]

> So what I should to do ?  
> Increase timeout or try new version ?  
> When it comes to increase timeout - is it possible to do it using xml config 
> file?

Try switching to the new version first. And yes it’s possible to set 
ConnectorConfiguration in XML.

—
Denis

> On May 26, 2016, at 6:32 PM, Tomek W  wrote:
> 
> | How long does it usually take to execute such query? I’m quite sure that 
> you’re facing with an I/O timeout that terminates the connection between the 
> client and server because query execution takes long time on the server side.
> 
> Yeah, I have the same guess.  
> 
> So what I should to do ?  
> Increase timeout or try new version ?  
> When it comes to increase timeout - is it possible to do it using xml config 
> file?
> 
> 2016-05-26 15:53 GMT+02:00 Denis Magda  >:
> Hi,
> 
> How long does it usually take to execute such query? I’m quite sure that 
> you’re facing with an I/O timeout that terminates the connection between the 
> client and server because query execution takes long time on the server side.
> 
> The first thing to do is to give a try to the latest Ignite release [1] that 
> contains the fix for the issue [2] you most probably facing with.
> The second thing to do is to increase ConnectorConfiguration.idleTimeout on 
> the server side. This can be done in the following way:
> 
> IgniteConfiguration cfg = new IgniteConfiguration();
> 
> ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration();
> 
> connectorConfiguration.setIdleTimeout(30_000);
> 
> cfg.setConnectorConfiguration(connectorConfiguration);
> 
> [1] apache-ignite-fabric-1.6.0-bin.zip 
> 
> [2] https://issues.apache.org/jira/browse/IGNITE-2751 
> 
> 
> —
> Denis
> 
>> On May 25, 2016, at 3:14 AM, tomk > > wrote:
>> 
>> Hello, 
>> below, I attached some exception that I get during querying by JDBC client. 
>> It occurs for SELECT * COUNT.
>> It is very important that this exception occurs about once every three times
>> - twice getting the correct result. 
>> My cache has exactly 300 entries.
>> 
>> 
>> java.sql.SQLException: Failed to query Ignite.
>>at
>> org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:122)
>>at Tester.main(Tester.java:41)
>> Caused by: class
>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
>> Failed to perform request (connection failed): /127.0.0.1:11212 
>> 
>>at
>> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
>>at
>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:336)
>>at
>> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:296)
>>at
>> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:605)
>>at
>> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:249)
>>at
>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>at
>> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
>>at
>> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:2115)
>>at
>> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:1659)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:731)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1463)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1398)
>>at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1280)
>>at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>at java.lang.Thread.run(Thread.java:745)
>> ?
>> 
>> 
>> 
>> 
>> --
>> View this message in context: 
>> 

Re: Exception during querying by jdbc - occurs on average once every 5 times

[Tomek W]

| How long does it usually take to execute such query? I’m quite sure that
you’re facing with an I/O timeout that terminates the connection between
the client and server because query execution takes long time on the server
side.

Yeah, I have the same guess.

So what I should to do ?
Increase timeout or try new version ?
When it comes to increase timeout - is it possible to do it using xml
config file?

2016-05-26 15:53 GMT+02:00 Denis Magda :

> Hi,
>
> How long does it usually take to execute such query? I’m quite sure that
> you’re facing with an I/O timeout that terminates the connection between
> the client and server because query execution takes long time on the server
> side.
>
> The first thing to do is to give a try to the latest Ignite release [1]
> that contains the fix for the issue [2] you most probably facing with.
> The second thing to do is to increase ConnectorConfiguration.idleTimeout
> on the server side. This can be done in the following way:
>
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration();
>
> connectorConfiguration.setIdleTimeout(30_000);
>
> cfg.setConnectorConfiguration(connectorConfiguration);
>
>
> [1] apache-ignite-fabric-1.6.0-bin.zip
> 
> [2] https://issues.apache.org/jira/browse/IGNITE-2751
>
> —
> Denis
>
> On May 25, 2016, at 3:14 AM, tomk  wrote:
>
> Hello,
> below, I attached some exception that I get during querying by JDBC
> client.
> It occurs for SELECT * COUNT.
> It is very important that this exception occurs about once every three
> times
> - twice getting the correct result.
> My cache has exactly 300 entries.
>
>
> java.sql.SQLException: Failed to query Ignite.
>at
>
> org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:122)
>at Tester.main(Tester.java:41)
> Caused by: class
>
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
> Failed to perform request (connection failed): /127.0.0.1:11212
>at
>
> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
>at
>
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:336)
>at
>
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:296)
>at
>
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:605)
>at
>
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:249)
>at
>
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>at
>
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
>at
>
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:2115)
>at
>
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:1659)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:731)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1463)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1398)
>at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1280)
>at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>at java.lang.Thread.run(Thread.java:745)
> ?
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Exception-during-querying-by-jdbc-occurs-on-average-once-every-5-times-tp5156.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com
> .
>
>
>

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Tomek W]

No, I am using ON_HEAP_TIERED.

Maybe WriteBehind should be turned on ?
My App do exactly one thing:  initialize hot loading.

When it comes to JDBC client, I did show fragment of code in previous post.

2016-05-26 16:15 GMT+02:00 Alexei Scherbakov :

> I see long pauses in your GC log (> 3 seconds)
> This means your app have high pressure on the heap.
> It's hard to tell why without knowing what your app is doing.
>
> Are you using OFFHEAP_TIERED?
> If yes, try to reduce sqlOnheapRowCacheSize value.
>
>
>
>
> 2016-05-26 14:57 GMT+03:00 Tomek W :
>
>> Ok,
>> i am going to add new machines to ignite cluster. Firstly, please look at
>> my gc file log - previous message.
>>
>> 2016-05-26 13:39 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> Hi,
>>>
>>> The initial question was about setSqlOnheapRowCacheSize and I think
>>> now it is clear how to improve SQL performance using with parameter.
>>>
>>> If you dissatisfied with the Ignite performance, I suggest you to start
>>> a new thread on this,
>>> providing detailed info about your performance test like
>>> cluster configuration, server GC settings, and test sources.
>>>
>>> As already mentioned, Ignite SQL engine(H2) has the same(or slightly)
>>> less performance when Postresql.
>>> Ignite really starts to shine when used as distributed data grid having
>>> large amount of data in memory on several nodes.
>>>
>>> SELECT count(*) from table is not very good test query.
>>> Postgres may have the result cached, whereas Ignite always do the full
>>> table traversal.
>>> Recently I implemented an improvement for this case.
>>> See https://issues.apache.org/jira/browse/IGNITE-2751 for details.
>>>
>>> I strongly recommend to test Ignite performance on the real case.
>>> Dont' forget to configure GC properly [1]
>>>
>>> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2016-05-26 2:09 GMT+03:00 Tomek W :
>>>
 | Also it would be interesting to see result of
 | SELECT count(*) from the query above in both cases.
 (number of rows = 2 798 685)
 SELECT count(*) FROM postgresTable;
  456 ms
 SELECT count(*) FROM postgresTable;
 314 ms

 SELECT count(*) FROM igniteTable;
 9746 ms
 SELECT count(*) FROM igniteTable;
 9664 ms


 Code of Jdbc Drvier (the same code for Ignite and postgresql - url
 connection is given from command line):
 http://pastebin.com/mYDSjziN
 My start sh file:
 http://pastebin.com/VmRM2sPQ

 My gc log file (following hint Magda):
 (file generated during hot loading and query via JDBC).
 http://pastebin.com/XicnNczV


 If you would like to see something else let me know.

 PS How to launch H2 debug console ? I followed docs, but it doesn't
 help.
 I set enviroment variable:
 echo $IGNITE_H2_DEBUG_CONSOLE
 true
 now, ./ignite.sh conf.xml

 sudo netstat -tulpn | grep 61214
 No opened ports.

 BTW, during starting ignite it give me information:
 [01:03:02]  Performance suggestions for grid 'turbines_table_cluster'
 (fix if possible)
 [01:03:02] To disable, set
 -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
 [01:03:02]   ^-- Disable grid events (remove 'includeEventTypes' from
 configuration)
 [01:03:02]   ^-- Enable ATOMIC mode if not using transactions (set
 'atomicityMode' to ATOMIC)
 [01:03:02]   ^-- Enable write-behind to persistent store (set
 'writeBehindEnabled' to true)


 2016-05-25 12:23 GMT+02:00 Alexei Scherbakov <
 alexey.scherbak...@gmail.com>:

> For postgres test I mean initial jdbc query and result set traversal.
> For Ignite I mean sql query and iterator traversal.
> Also it would be interesting to see result of
> *SELECT count(*) from the query above in both cases.*
>
> 2016-05-25 12:00 GMT+03:00 Tomek W :
>
>> [image: Obraz w treści 1]
>>
>> What code do you mean ? JDBC client ?
>>
>> 2016-05-25 10:25 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> What's the batch size for postgresql ?
>>> What's the size of one entry ?
>>> Could you provide the test code for both postgres and Ignite (just
>>> the query + read with the time estimation) ?
>>>
>>> 2016-05-25 11:13 GMT+03:00 Tomek W :
>>>
 | How many entries are downloaded to the client in both cases?
 3000 000

 | Do the both queries involve network I/O ?
 No, I have only local one server (for testing purpose).


 2016-05-25 9:59 GMT+02:00 Alexei Scherbakov <
 alexey.scherbak...@gmail.com>:

> SELECT * is not really a good test query.
> It's result can be affected not only by engine 

Connect H2 Console to a running cluster

[edwardkblk]

Is there a way to find out url, user name, password to connect H2 Console to
already running cluster?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Connect-H2-Console-to-a-running-cluster-tp5242.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Nodes running on different operating systems

[Graham Bull]

I've disabled the firewall on each of the three machines, but unfortunately
it makes no difference :(

I need to leave this for now, but should return to it at some point later
on.

Thanks for your help Val,

Graham



On 25 May 2016 at 12:57, vkulichenko  wrote:

> Hi Graham,
>
> Note that all nodes in topology have to be able to connect to each other in
> both directions. Looks like one of your nodes can accept connections, but
> can't create them (or other way around). Probably your Windows box has a
> firewall enabled?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Nodes-running-on-different-operating-systems-tp5098p5181.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Unexpected performance issue with SQL query followed by error

[jan.swaelens]

Hello,

The activity table is 121823 records and the activityuseraccountrole 1027839
records.

First one runs under a second and produces:

/SELECT DISTINCT
ACTIVITY0._KEY,
ACTIVITY0._VAL,
ACTIVITY0.ACTIVITY_ID,
ACTIVITY0.TIMESTAMP,
ACTIVITY0.CONTAINER_ID,
ACTIVITY0.ACTIVITYTYPE_ID,
ACTIVITY0.REALIZATION_ID,
ACTIVITY0.KERNEL_ID,
ACTIVITY0.PREDECESSORTYPE_ENUMID,
ACTIVITY0.SUCCESSORTYPE_ENUMID,
ACTIVITY0.DURATIONUNIT_ENUMID,
ACTIVITY0.NAME,
ACTIVITY0.NAME_MLID,
ACTIVITY0.DESCRIPTION,
ACTIVITY0.DESCRIPTION_MLID,
ACTIVITY0.DURATION,
ACTIVITY0.REQUIRED,
ACTIVITY0.ESTIMSTARTDATE,
ACTIVITY0.ESTIMSTARTHOUR,
ACTIVITY0.ESTIMENDHOUR,
ACTIVITY0.ESTIMENDDATE,
ACTIVITY0.REMOVEFROMWORKLIST,
ACTIVITY0.SEQUENCENR,
ACTIVITY0.SESSION_ID,
ACTIVITY0.LASTACTIVITY_ID,
ACTIVITY0.SYSREPOPERATION_ID,
ACTIVITY0.LIFECYCLEREPORTING,
ACTIVITY0.DUEDATE,
ACTIVITY0.PRIORITY_ENUMID,
ACTIVITY0.NOTIFY
FROM "Activity".ACTIVITY ACTIVITY0
/* "Activity".ACTIVITY.__SCAN_ */
/* scanCount: 121824 *//

The seconds one also very fast and produces:

/SELECT DISTINCT
ACTIVITYUSERACCOUNTROLE0._KEY,
ACTIVITYUSERACCOUNTROLE0._VAL,
ACTIVITYUSERACCOUNTROLE0.ACTIVITY_ID,
ACTIVITYUSERACCOUNTROLE0.USERACCOUNTROLE_ID
FROM "Activity".ACTIVITYUSERACCOUNTROLE ACTIVITYUSERACCOUNTROLE0
/* "Activity".ACTIVITYUSERACCOUNTROLE.__SCAN_ */
/* scanCount: 1027840 */
WHERE ACTIVITYUSERACCOUNTROLE0.USERACCOUNTROLE_ID IN(1, 3)/

So basically, the join seems to kill it?

br
jan




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Unexpected-performance-issue-with-SQL-query-followed-by-error-tp4726p5239.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Use apache Ignite outside spark-shell

[luismattor]

Hi,

The problem is in IgniteContext.scala:

// Start ignite server node on each worker in server mode.
sparkContext.parallelize(1 to workers, workers).foreachPartition(it ⇒
ignite())

As far as I understand, this line works as far as each executor holds a
single partition. That is, if executor owns more than one partition
IgniteContext will try to launch several Ignite nodes (in the same
process); if there are executors with no partitions, no Ignite instance
will be launched.

For example, Image a two worker cluster (node1, node2), each worker having
two cores. The idea is to launch an Ignite node on each executor. So we
call:
sparkContext.parallelize(1 to 2, 2).foreachPartition(it ⇒ ignite())

It might be the chance the two partitions are handled by node1. Or it might
be the case node1 handle one partition, and node2 the other. (this is what
we want). So the question is how to make sure the latter scenario happens?

Regards
Luis




On 26 May 2016 at 05:19, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n5228...@n6.nabble.com> wrote:

> Hi,
>
> Why do you think this is happening? Do you have any exception? Generally,
> IgniteContext always reuses the Ignite client if it already exists in the
> current process.
>
> Note that you can still have several clients in several processes. Usually
> you will have a client on the driver and a client per each executor (not
> worker!).
>
> -Val
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
> To unsubscribe from Use apache Ignite outside spark-shell, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5238.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Exception during querying by jdbc - occurs on average once every 5 times

[Denis Magda]

Hi,

How long does it usually take to execute such query? I’m quite sure that you’re 
facing with an I/O timeout that terminates the connection between the client 
and server because query execution takes long time on the server side.

The first thing to do is to give a try to the latest Ignite release [1] that 
contains the fix for the issue [2] you most probably facing with.
The second thing to do is to increase ConnectorConfiguration.idleTimeout on the 
server side. This can be done in the following way:

IgniteConfiguration cfg = new IgniteConfiguration();

ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration();

connectorConfiguration.setIdleTimeout(30_000);

cfg.setConnectorConfiguration(connectorConfiguration);

[1] apache-ignite-fabric-1.6.0-bin.zip 

[2] https://issues.apache.org/jira/browse/IGNITE-2751 


—
Denis

> On May 25, 2016, at 3:14 AM, tomk  wrote:
> 
> Hello, 
> below, I attached some exception that I get during querying by JDBC client. 
> It occurs for SELECT * COUNT.
> It is very important that this exception occurs about once every three times
> - twice getting the correct result. 
> My cache has exactly 300 entries.
> 
> 
> java.sql.SQLException: Failed to query Ignite.
>at
> org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:122)
>at Tester.main(Tester.java:41)
> Caused by: class
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
> Failed to perform request (connection failed): /127.0.0.1:11212
>at
> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
>at
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:336)
>at
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:296)
>at
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:605)
>at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:249)
>at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
>at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:2115)
>at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:1659)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:731)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1463)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1398)
>at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1280)
>at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>at java.lang.Thread.run(Thread.java:745)
> ?
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Exception-during-querying-by-jdbc-occurs-on-average-once-every-5-times-tp5156.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How do I know the cache rebalance is finished?

[vkulichenko]

Hi,

There were some changes in affinity assignment that could cause the
difference in behavior, but actually rebalancing in a separate process and
there was never a guarantee that partitions will be assigned prior to
EVT_CACHE_REBALANCE_STOPPED is fired.

Can you provide more details about your use case? What are you trying to
achieve?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-do-I-know-the-cache-rebalance-is-finished-tp5219p5235.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite 1.6.0 binary - compilation Issues

[Igor Sapego]

Agneeswaran,

I've submitted a ticket for this issue: [1]. I'll reply you here once it's
going to
be resolved.

[1] - https://issues.apache.org/jira/browse/IGNITE-3205

Best Regards,
Igor

On Thu, May 26, 2016 at 2:48 PM, Agneeswaran <
agneeswaran.ponnuraman...@nielsen.com> wrote:

> Thanks Igor,
>
> Please find the compiler version details,
>
> g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
> Copyright (C) 2010 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
> Thanks,
> Agneeswaran
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-binary-compilation-Issues-tp5202p5233.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Ignite 1.6.0 binary - compilation Issues

[Agneeswaran]

Thanks Igor,

Please find the compiler version details,

g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Thanks,
Agneeswaran



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-binary-compilation-Issues-tp5202p5233.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Tomek W]

Ok,
i am going to add new machines to ignite cluster. Firstly, please look at
my gc file log - previous message.

2016-05-26 13:39 GMT+02:00 Alexei Scherbakov :

> Hi,
>
> The initial question was about setSqlOnheapRowCacheSize and I think
> now it is clear how to improve SQL performance using with parameter.
>
> If you dissatisfied with the Ignite performance, I suggest you to start a
> new thread on this,
> providing detailed info about your performance test like
> cluster configuration, server GC settings, and test sources.
>
> As already mentioned, Ignite SQL engine(H2) has the same(or slightly) less
> performance when Postresql.
> Ignite really starts to shine when used as distributed data grid having
> large amount of data in memory on several nodes.
>
> SELECT count(*) from table is not very good test query.
> Postgres may have the result cached, whereas Ignite always do the full
> table traversal.
> Recently I implemented an improvement for this case.
> See https://issues.apache.org/jira/browse/IGNITE-2751 for details.
>
> I strongly recommend to test Ignite performance on the real case.
> Dont' forget to configure GC properly [1]
>
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
>
>
>
>
>
>
> 2016-05-26 2:09 GMT+03:00 Tomek W :
>
>> | Also it would be interesting to see result of
>> | SELECT count(*) from the query above in both cases.
>> (number of rows = 2 798 685)
>> SELECT count(*) FROM postgresTable;
>>  456 ms
>> SELECT count(*) FROM postgresTable;
>> 314 ms
>>
>> SELECT count(*) FROM igniteTable;
>> 9746 ms
>> SELECT count(*) FROM igniteTable;
>> 9664 ms
>>
>>
>> Code of Jdbc Drvier (the same code for Ignite and postgresql - url
>> connection is given from command line):
>> http://pastebin.com/mYDSjziN
>> My start sh file:
>> http://pastebin.com/VmRM2sPQ
>>
>> My gc log file (following hint Magda):
>> (file generated during hot loading and query via JDBC).
>> http://pastebin.com/XicnNczV
>>
>>
>> If you would like to see something else let me know.
>>
>> PS How to launch H2 debug console ? I followed docs, but it doesn't
>> help.
>> I set enviroment variable:
>> echo $IGNITE_H2_DEBUG_CONSOLE
>> true
>> now, ./ignite.sh conf.xml
>>
>> sudo netstat -tulpn | grep 61214
>> No opened ports.
>>
>> BTW, during starting ignite it give me information:
>> [01:03:02]  Performance suggestions for grid 'turbines_table_cluster'
>> (fix if possible)
>> [01:03:02] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
>> [01:03:02]   ^-- Disable grid events (remove 'includeEventTypes' from
>> configuration)
>> [01:03:02]   ^-- Enable ATOMIC mode if not using transactions (set
>> 'atomicityMode' to ATOMIC)
>> [01:03:02]   ^-- Enable write-behind to persistent store (set
>> 'writeBehindEnabled' to true)
>>
>>
>> 2016-05-25 12:23 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> For postgres test I mean initial jdbc query and result set traversal.
>>> For Ignite I mean sql query and iterator traversal.
>>> Also it would be interesting to see result of
>>> *SELECT count(*) from the query above in both cases.*
>>>
>>> 2016-05-25 12:00 GMT+03:00 Tomek W :
>>>
 [image: Obraz w treści 1]

 What code do you mean ? JDBC client ?

 2016-05-25 10:25 GMT+02:00 Alexei Scherbakov <
 alexey.scherbak...@gmail.com>:

> What's the batch size for postgresql ?
> What's the size of one entry ?
> Could you provide the test code for both postgres and Ignite (just the
> query + read with the time estimation) ?
>
> 2016-05-25 11:13 GMT+03:00 Tomek W :
>
>> | How many entries are downloaded to the client in both cases?
>> 3000 000
>>
>> | Do the both queries involve network I/O ?
>> No, I have only local one server (for testing purpose).
>>
>>
>> 2016-05-25 9:59 GMT+02:00 Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>>
>>> SELECT * is not really a good test query.
>>> It's result can be affected not only by engine performance.
>>>
>>> How many entries are downloaded to the client in both cases?
>>> Do the both queries involve network I/O ?
>>>
>>> 2016-05-25 7:58 GMT+03:00 Denis Magda :
>>>
 In general Ignite is designed to be used in a distributed
 environment when gigabytes or terabytes of dataset is spread across 
 many
 cluster nodes and SQL queries executed across the cluster should be 
 faster
 since resources of all the machines will be used and as a result a 
 query
 should be completed quicker. In your scenario you just have only a 
 single
 cluster node and in fact comparing performance of PostgreSQL and H2 
 (engine
 that is used by Ignite SQL) and I can consider that Ignite SQL can work
 slightly slowly but this in is 

Re: Unexpected performance issue with SQL query followed by error

[Alexei Scherbakov]

How much data do you have in both tables?

30 minutes is too long even for full scan in memory.

Please run

EXPLAIN ANALYZE SELECT DISTINCT * FROM "Activity".activity activity0

EXPLAIN ANALYZE SELECT DISTINCT * FROM "Activity".activityuseraccountrole
activityuseraccountrole0
WHERE activityuseraccountrole0.useraccountrole_Id IN (1, 3)

Don't you forget to configure GC property, as discussed earlier in this
topic ?
Do you have any activity on cache while running the query ?


2016-05-25 16:48 GMT+03:00 jan.swaelens :

> Thanks,
>
> Added the code and executed the following:
>
> EXPLAIN ANALYZE SELECT DISTINCT * FROM "Activity".activity activity0
> LEFT OUTER JOIN "Activity".activityuseraccountrole activityuseraccountrole0
> ON activityuseraccountrole0.activity_Id = activity0.activity_Id
> AND activityuseraccountrole0.useraccountrole_Id IN (1, 3)
>
> This also keeps running until eternity, well at least for 30 minutes after
> which I give up. Should I wait it out or does it make sense to add a clause
> to limit the data?
>
> best regards
> jan
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Unexpected-performance-issue-with-SQL-query-followed-by-error-tp4726p5194.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 

Best regards,
Alexei Scherbakov

Re: off heap indexes - setSqlOnheapRowCacheSize - how does it improve efficiency ?

[Alexei Scherbakov]

Hi,

The initial question was about setSqlOnheapRowCacheSize and I think
now it is clear how to improve SQL performance using with parameter.

If you dissatisfied with the Ignite performance, I suggest you to start a
new thread on this,
providing detailed info about your performance test like
cluster configuration, server GC settings, and test sources.

As already mentioned, Ignite SQL engine(H2) has the same(or slightly) less
performance when Postresql.
Ignite really starts to shine when used as distributed data grid having
large amount of data in memory on several nodes.

SELECT count(*) from table is not very good test query.
Postgres may have the result cached, whereas Ignite always do the full
table traversal.
Recently I implemented an improvement for this case.
See https://issues.apache.org/jira/browse/IGNITE-2751 for details.

I strongly recommend to test Ignite performance on the real case.
Dont' forget to configure GC properly [1]

[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning






2016-05-26 2:09 GMT+03:00 Tomek W :

> | Also it would be interesting to see result of
> | SELECT count(*) from the query above in both cases.
> (number of rows = 2 798 685)
> SELECT count(*) FROM postgresTable;
>  456 ms
> SELECT count(*) FROM postgresTable;
> 314 ms
>
> SELECT count(*) FROM igniteTable;
> 9746 ms
> SELECT count(*) FROM igniteTable;
> 9664 ms
>
>
> Code of Jdbc Drvier (the same code for Ignite and postgresql - url
> connection is given from command line):
> http://pastebin.com/mYDSjziN
> My start sh file:
> http://pastebin.com/VmRM2sPQ
>
> My gc log file (following hint Magda):
> (file generated during hot loading and query via JDBC).
> http://pastebin.com/XicnNczV
>
>
> If you would like to see something else let me know.
>
> PS How to launch H2 debug console ? I followed docs, but it doesn't help.
> I set enviroment variable:
> echo $IGNITE_H2_DEBUG_CONSOLE
> true
> now, ./ignite.sh conf.xml
>
> sudo netstat -tulpn | grep 61214
> No opened ports.
>
> BTW, during starting ignite it give me information:
> [01:03:02]  Performance suggestions for grid 'turbines_table_cluster' (fix
> if possible)
> [01:03:02] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> [01:03:02]   ^-- Disable grid events (remove 'includeEventTypes' from
> configuration)
> [01:03:02]   ^-- Enable ATOMIC mode if not using transactions (set
> 'atomicityMode' to ATOMIC)
> [01:03:02]   ^-- Enable write-behind to persistent store (set
> 'writeBehindEnabled' to true)
>
>
> 2016-05-25 12:23 GMT+02:00 Alexei Scherbakov  >:
>
>> For postgres test I mean initial jdbc query and result set traversal.
>> For Ignite I mean sql query and iterator traversal.
>> Also it would be interesting to see result of
>> *SELECT count(*) from the query above in both cases.*
>>
>> 2016-05-25 12:00 GMT+03:00 Tomek W :
>>
>>> [image: Obraz w treści 1]
>>>
>>> What code do you mean ? JDBC client ?
>>>
>>> 2016-05-25 10:25 GMT+02:00 Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com>:
>>>
 What's the batch size for postgresql ?
 What's the size of one entry ?
 Could you provide the test code for both postgres and Ignite (just the
 query + read with the time estimation) ?

 2016-05-25 11:13 GMT+03:00 Tomek W :

> | How many entries are downloaded to the client in both cases?
> 3000 000
>
> | Do the both queries involve network I/O ?
> No, I have only local one server (for testing purpose).
>
>
> 2016-05-25 9:59 GMT+02:00 Alexei Scherbakov <
> alexey.scherbak...@gmail.com>:
>
>> SELECT * is not really a good test query.
>> It's result can be affected not only by engine performance.
>>
>> How many entries are downloaded to the client in both cases?
>> Do the both queries involve network I/O ?
>>
>> 2016-05-25 7:58 GMT+03:00 Denis Magda :
>>
>>> In general Ignite is designed to be used in a distributed
>>> environment when gigabytes or terabytes of dataset is spread across many
>>> cluster nodes and SQL queries executed across the cluster should be 
>>> faster
>>> since resources of all the machines will be used and as a result a query
>>> should be completed quicker. In your scenario you just have only a 
>>> single
>>> cluster node and in fact comparing performance of PostgreSQL and H2 
>>> (engine
>>> that is used by Ignite SQL) and I can consider that Ignite SQL can work
>>> slightly slowly but this in is not Ignite usage scenario.
>>>
>>> However if you try to create a cluster of several nodes running on
>>> different physical machines, pre-load gigabytes of data there and 
>>> compare
>>> Ignite SQL and PostgresSQL you should see performance improvements on
>>> Ignite side.
>>>
>>> In any case taking into account the advise above do 

Re: Ignite 1.6.0 binary - compilation Issues

[Igor Sapego]

Hi Agneeswaran,

It seems like we need to add fix for your compiler. Can you please tell me
your compiler
version so I can test with it? You can find it out using 'g++ --version'
shell command.

Best Regards,
Igor

On Thu, May 26, 2016 at 11:43 AM, Agneeswaran <
agneeswaran.ponnuraman...@nielsen.com> wrote:

> Hi team,
>
> Please help me on this compilation error.
>
> Thanks,
> Agneeswaran
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-binary-compilation-Issues-tp5202p5223.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Use apache Ignite outside spark-shell

[vkulichenko]

Hi,

Why do you think this is happening? Do you have any exception? Generally,
IgniteContext always reuses the Ignite client if it already exists in the
current process.

Note that you can still have several clients in several processes. Usually
you will have a client on the driver and a client per each executor (not
worker!).

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: meaning of backup options.

[vkulichenko]

'readFromBackup' defines if it's allowed to read the local value from the
backup. I.e. if you're executing get(K) on the server node, this node is
backup for the K and readFromBackup is false, Ignite will send the request
to primary node and get the value from there. By default the value is true,
so is the entry is available locally, no network trips are required.

'copyOnRead' defines whether the copy of the value is created before it's
returned from the cache. Of course, this makes sense only if the value is
read locally (e.g., if in case of get() on primary or backup node, in case
of invoke(), etc.). Setting this to false can improve performance, because
you avoid additional deserialization step, but it's not safe to mutate the
value that you read in this case, so usually it's used only for immutable
objects.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/meaning-of-backup-options-tp5134p5227.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Transaction performance

[amitpa]

Val, 

Will do that, but meanwhile can I increase performance incresing System
Thread pool etc? 

My machine has CPU and bandwidth limit is still not reached



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transaction-performance-tp5085p5226.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite and H2?

[vkulichenko]

Hi,

Your understanding is correct. H2 is mostly used for query parsing and
execution planning. Indexes and tables are highly customized to work
directly with the data stored in cache and to provide better performance in
highly concurrent scenarios.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-and-H2-tp5200p5225.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Transaction performance

[vkulichenko]

Hi,

With a single node without a client, there is no network, so latency is of
course lower than when you add network.

According to you description, your test is more complicated than code
snippets your provided before. So please provide a project that I will be
able to run and reproduce the performance drop. In this case I will be able
to tell something regarding your particular benchmark.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transaction-performance-tp5085p5224.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Transaction performance

[amitpa]

Lastly is there any thread count number to be increased to help in high
throughput commit messages...liek does increasing the system threads
increase performance?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transaction-performance-tp5085p5222.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two nodes in cluster

[Denis Magda]

Ignite will automatically redistribute (rebalance) data across all available 
nodes. So everything is scaling horizontally automatically.
You just need to start a new node and portion of dataset will be rebalanced on 
it.

—
Denis

> On May 26, 2016, at 10:30 AM, Tomek W  wrote:
> 
> Yes,
> but I consider how to force ignite to distribute rows on other machines (not 
> only this  machine that initialized hot loading).
> 
> 2016-05-26 8:46 GMT+02:00 Vladislav Pyatkov  >:
> Hello,
> 
> If I understand correctly, you want to load data from a database in 
> PARTITIONED type cache. This case is well described here
> https://apacheignite.readme.io/docs/data-loading 
> 
> 
> On Thu, May 26, 2016 at 1:45 AM, tomk  > wrote:
> Hello,
> At now I have only one node, but I am going to add more machines (in one
> local network).
> What should I know to do it ?
> I have PARTITIONED type of cache.
> Hot loading (my use case is caching postgresql) is done on one machine.
> How to force ignite to distribute these rows to all machines - not only that
> where client is executed (client initialize hot loading from postgresql).
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
>  
> 
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
> 
> 
> 
> -- 
> Vladislav Pyatkov
> 

Re: Two nodes in cluster

[Vladislav Pyatkov]

Data themselves are distributed across nodes (in the PARTITIONED cache) is
responsible for this affinity function.
You can follow any responses to write his affinity functions are
implemented org.apache.ignite.cache.affinity.AffinityFunction and install
it via the configuration cache
org.apache.ignite.configuration.CacheConfiguration # setAffinity

On Thu, May 26, 2016 at 10:30 AM, Tomek W  wrote:

> Yes,
> but I consider how to force ignite to *distribute* rows on other machines
> (*not only this  machine that initialized hot loading*).
>
> 2016-05-26 8:46 GMT+02:00 Vladislav Pyatkov :
>
>> Hello,
>>
>> If I understand correctly, you want to load data from a database in
>> PARTITIONED type cache. This case is well described here
>> https://apacheignite.readme.io/docs/data-loading
>>
>> On Thu, May 26, 2016 at 1:45 AM, tomk  wrote:
>>
>>> Hello,
>>> At now I have only one node, but I am going to add more machines (in one
>>> local network).
>>> What should I know to do it ?
>>> I have PARTITIONED type of cache.
>>> Hot loading (my use case is caching postgresql) is done on one machine.
>>> How to force ignite to distribute these rows to all machines - not only
>>> that
>>> where client is executed (client initialize hot loading from postgresql).
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Vladislav Pyatkov
>>
>
>


-- 
Vladislav Pyatkov

How do I know the cache rebalance is finished?

[Andrew]

Hi,

I was using REBALANCE_STOPPED event to recognize which primary partitions
are assigned to local with Ignite v1.5.
As the REBALANCE_STOPPED event occurs a couple of times when a server node
has been failed,
I made my code to check discoveryEvent of received events because I want my
code be executed only once per a node failed or joined.

It worked well but after upgrade to Ignite v1.6 my code does not perform
correctly.
I found the partitions are not rebalanced yet, although the event listener
received a REBALANCE_STOPPED event.

Here is my code.
How do I fix my code? I need your advice.

-
Ignite ignite = IgniteUtil.getIgnite();

IgnitePredicate locLsnr = new
IgnitePredicate() {
@Override
public boolean apply(EventAdapter evt) {
Boolean isClient = null;

switch (evt.type()) {
case EventType.EVT_CACHE_REBALANCE_STOPPED:
if (evt instanceof 
CacheRebalancingEvent) {
CacheRebalancingEvent cacheEvt 
= (CacheRebalancingEvent) evt;
isClient =
cacheEvt.discoveryNode().attribute(IgniteNodeAttributes.ATTR_CLIENT_MODE);
if (isClient != null && 
!isClient
&& 
cacheEvt.cacheName().equals(CacheConfig.messageCache().getName())
&& 
(cacheEvt.discoveryEventType() == EventType.EVT_NODE_JOINED

|| cacheEvt.discoveryEventType() == EventType.EVT_NODE_LEFT 

|| cacheEvt.discoveryEventType() == EventType.EVT_NODE_FAILED)) {

//Rebalance finished
Affinity aff = 
ignite.affinity(CacheConfig.messageCache().getName());
int[] partitions =
aff.primaryPartitions(ignite.cluster().localNode());

}
}
break;
default:
break;
}

return true;
}
};

ignite.events().localListen(locLsnr,
EventType.EVT_CACHE_REBALANCE_STOPPED);
-

Sincerely,
Andrew. 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-do-I-know-the-cache-rebalance-is-finished-tp5219.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two nodes in cluster

[Tomek W]

Yes,
but I consider how to force ignite to *distribute* rows on other machines (*not
only this  machine that initialized hot loading*).

2016-05-26 8:46 GMT+02:00 Vladislav Pyatkov :

> Hello,
>
> If I understand correctly, you want to load data from a database in
> PARTITIONED type cache. This case is well described here
> https://apacheignite.readme.io/docs/data-loading
>
> On Thu, May 26, 2016 at 1:45 AM, tomk  wrote:
>
>> Hello,
>> At now I have only one node, but I am going to add more machines (in one
>> local network).
>> What should I know to do it ?
>> I have PARTITIONED type of cache.
>> Hot loading (my use case is caching postgresql) is done on one machine.
>> How to force ignite to distribute these rows to all machines - not only
>> that
>> where client is executed (client initialize hot loading from postgresql).
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Vladislav Pyatkov
>

Re: Required Spring jars for Configuration

[Vladislav Pyatkov]

Hello Kamal,
I think you can rule out a, g, h and i.
But I could be wrong, it should be checked. Simply remove jars from the
assembly and try to run.

Re: Two nodes in cluster

[Vladislav Pyatkov]

Hello,

If I understand correctly, you want to load data from a database in
PARTITIONED type cache. This case is well described here
https://apacheignite.readme.io/docs/data-loading

On Thu, May 26, 2016 at 1:45 AM, tomk  wrote:

> Hello,
> At now I have only one node, but I am going to add more machines (in one
> local network).
> What should I know to do it ?
> I have PARTITIONED type of cache.
> Hot loading (my use case is caching postgresql) is done on one machine.
> How to force ignite to distribute these rows to all machines - not only
> that
> where client is executed (client initialize hot loading from postgresql).
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Two-nodes-in-cluster-tp5204.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Required Spring jars for Configuration

[Kamal C]

Hi all,

I'm using Ignite Spring configuration files to configure the Ignite and
cache instances.

The `Ignite-Spring` folder consists of many jars. In my application, I'm
not using spring framework. How to find only the required jars which loads
the configuration files?

a. commons-logging-1.1.1.jar
b. ignite-spring-1.6.0.jar
c. spring-aop-4.1.0.RELEASE.jar
d. spring-beans-4.1.0.RELEASE.jar
e. spring-context-4.1.0.RELEASE.jar
f. spring-core-4.1.0.RELEASE.jar
g. spring-expression-4.1.0.RELEASE.jar
h. spring-jdbc-4.1.0.RELEASE.jar
i. spring-tx-4.1.0.RELEASE.jar


Regards,
Kamal C