Re: "ArrayIndexOutOfBoundsException" happened when doing qurey in version 1.6.0

[Semyon Boikov]

We already have similar issue reproduced in our benchmarks -
https://issues.apache.org/jira/browse/IGNITE-3300. I think this is related
to optimziation done in 1.6 to store key partition in the cache key. I
believe fix will be included in 1.7 release.

On Tue, Jul 26, 2016 at 2:13 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Crossposting to dev@
>
> Folks,
>
> I noticed the exception below in couple of user threads already. This
> actually means that partition number for the key is -1, which doesn't make
> much sense to me. Does anyone has any ideas?
>
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
>   at java.util.ArrayList.elementData(ArrayList.java:400)
>   at java.util.ArrayList.get(ArrayList.java:413)
>   at 
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignment.get(GridAffinityAssignment.java:152)
>   at 
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:387)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodes(GridCacheAffinityManager.java:251)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:287)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:278)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:302)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7$3.apply(IgniteH2Indexing.java:1730)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:251)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:200)
>   at 
> org.apache.ignite.internal.util.lang.GridFilteredIterator.hasNext(GridFilteredIterator.java:59)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Cursor.next(GridH2Cursor.java:59)
>   at org.h2.index.IndexCursor.next(IndexCursor.java:274)
>   at org.h2.table.TableFilter.next(TableFilter.java:359)
>   at org.h2.command.dml.Select.queryFlat(Select.java:527)
>   at org.h2.command.dml.Select.queryWithoutCache(Select.java:632)
>   at org.h2.command.dml.Query.query(Query.java:297)
>   at org.h2.command.dml.Query.query(Query.java:284)
>   at org.h2.command.dml.Query.query(Query.java:36)
>   at org.h2.command.CommandContainer.query(CommandContainer.java:91)
>   at org.h2.command.Command.executeQuery(Command.java:196)
>   ... 26 more
>
>
> -Val
>
>
> On Mon, Jul 25, 2016 at 5:07 AM, ght230  wrote:
>
>> ignite-related.xml
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/n6516/ignite-related.xml
>> >
>> Please refer to the attachment.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/ArrayIndexOutOfBoundsException-happened-when-doing-qurey-in-version-1-6-0-tp6245p6516.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: How to use BackupFilter to assign all backups to different machine groups?

[Jason]

Thanks Val.

so the attributes are per node, and after exchange when new node starts,
each node has a copy of attributes for every node in the cluster, right?

Thanks,
-Jason



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-use-BackupFilter-to-assign-all-backups-to-different-machine-groups-tp6513p6535.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Can not start another client when doing ContinuousAsyncQuery

[ght230]

ExampleNodeStartup.java

  
CacheContinuousAsyncQueryExample.java

  
ExampleNodeClient.java

  
example-cache.xml
  

step 1. running ExampleNodeStartup.java
step 2. running CacheContinuousAsyncQueryExample.java
step 3. running ExampleNodeClient.java at the same time of step 2.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Can-not-start-another-client-when-doing-ContinuousAsyncQuery-tp6375p6534.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Can not start another client when doing ContinuousAsyncQuery

[ght230]

Anyone has ideas?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Can-not-start-another-client-when-doing-ContinuousAsyncQuery-tp6375p6530.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to use BackupFilter to assign all backups to different machine groups?

[Jason]

Thanks Val.

Because the user attributes are a map of , so "attributes
are static and can't be changed after the node is started. Nodes exchange
attributes with each other during the join process" means that the maps  are
merged together with new key added, and old key unchanged?  

If there's change for old key, will it raise an exception? e.g. there're
three nodes in cluster now: A1, B2, C3 and their user attribute map is [, , ]. Then add a new node D2 whose map is [, ,
, ] where A's group is changed from 1 to 2, what will happen? all
the nodes will have a map of [, , , ] or [, , , ] or throw exception?

Thanks,
-Jason



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-use-BackupFilter-to-assign-all-backups-to-different-machine-groups-tp6513p6529.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: QuerySqlFunction methods with variable arguments do not seem to work

[vkulichenko]

Hi,

Please show your cache configuration. Did you provide the MySql class in the
CacheConfiguration.setSqlFunctionClasses(..) property?

BTW, instead of this query, it's much better to use IgniteCache.get(..)
method, providing the required key.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/QuerySqlFunction-methods-with-variable-arguments-do-not-seem-to-work-tp6524p6528.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to use BackupFilter to assign all backups to different machine groups?

[vkulichenko]

Hi Jason,

a). Yes, you can provide attributes in the XML file in the same way as with
Java.
b). This is true. With the current API there is a chance that two backups
will end up in the same group.
c,d,e). As I responded in another thread, attributes are static and can't be
changed after the node is started. Nodes exchange attributes with each other
during the join process, so all existing nodes know in which group the new
node is. Affinity will be assigned correctly according to the backup filter.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-use-BackupFilter-to-assign-all-backups-to-different-machine-groups-tp6513p6527.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Exception scanning cache

[vkulichenko]

Hi,

Please properly subscribe to the mailing list so that the community can
receive email notifications for your messages. Here is the instruction:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


hlustosa wrote
> -When we start a client node, its local memory is used to store keys and
> values for a distributed cache?

No, unless you create a near cache on this client. See
https://apacheignite.readme.io/docs/near-caches


hlustosa wrote
> -In case when there is no replication, the cache.put method is
> synchronous? I mean, once the method is executed in the client, the key
> and the value are transferred to some cluster node? 

Yes, by default cache.put updates the primary node synchronously. If you
want to execute asynchronously, refer to this page:
https://apacheignite.readme.io/docs/async-support

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Exception-scanning-cache-tp6493p6520.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Transactions and non-committed entries

[vkulichenko]

There are operations that are enlisted in a transaction (get, put, remove,
etc.) and non-transactional (like size, clear, etc.). JavaDoc for methods
usually specify that. If a transactional operation is executed without a
transaction, an implicit transaction will be created by Ignite for this
single operation. Non-transactional operations are always executed outside
of the transactional context, regardless of weather explicit transaction
exists in the current thread or not.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transactions-and-non-committed-entries-tp6451p6519.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite - SQL queries aggregation

[Andrey Gura]

Ignite SQL engine built on top of H2 engine and supports aggregation
functions like cnt, max, etc.

Also Ignite supports user defined function. See QuerySqlFunction annotation
documentation and its usages in Ignite tests (e.g.
https://github.com/apache/ignite/blob/68891e89dd0e0f19321d6a4d45ae7372279b8b08/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/IgniteCacheAbstractQuerySelfTest.java#L1877
)

In order to define functions you should create class with some set of
functions where each function represented as method annotated by
QuerySqlFunction and configure cache using
CacheConfiguration.setSqlFunctionClasses() method.

On Mon, Jul 25, 2016 at 7:41 PM, M Singh  wrote:

> Hi:
>
> I wanted to find out if ignite sql/other queries support aggregation or
> udf functions (like count, etc).
>
> Thanks
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Ignite - SQL queries aggregation

[M Singh]

Hi:
I wanted to find out if ignite sql/other queries support aggregation or udf 
functions (like count, etc).
Thanks

How to monitor the cache info?

[Jason]

hi Ignite team,

In the community version, how to monitor the below info:

1. which nodes are one cache assigned to? which partitions for each node?

2. how to monitor the memory usage in each node for one cache?

3. how to monitor the off-heap memory usage for one node? seems the node
command in IgniteVisor can only show the heap and non-heap which isn't the
same with off-heap memory.

Thanks,
-Jason





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-monitor-the-cache-info-tp6515.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Question about GridGain will typically add around 200 bytes overhead to each entry

[Sergi Vladykin]

I don't think that enterprise edition will help much here. I hope soon we
will be releasing 1.7, there we made some improvements in this area, you
can try it when it will be out.

Sergi

2016-07-25 11:20 GMT+03:00 November :

> Hi Sergi,
>
> I have tried try setting setMemoryMode to OFFHEAP_TIERED on
> CacheConfiguratio. It save about 2GB, but still use about 30GB memory.
>
> I wonder if there is any method to reduce the overhead?
> I notice Ignite has enterprise edition. Will them differ in overhead?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Question-about-GridGain-will-typically-add-around-200-bytes-overhead-to-each-entry-tp6411p6509.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Question about GridGain will typically add around 200 bytes overhead to each entry

[November]

Hi Sergi,

I have tried try setting setMemoryMode to OFFHEAP_TIERED on
CacheConfiguratio. It save about 2GB, but still use about 30GB memory.

I wonder if there is any method to reduce the overhead?
I notice Ignite has enterprise edition. Will them differ in overhead?

Thanks



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Question-about-GridGain-will-typically-add-around-200-bytes-overhead-to-each-entry-tp6411p6509.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Question about GridGain will typically add around 200 bytes overhead to each entry

[Sergi Vladykin]

Yes, it looks like Ignite introduces the most of overhead here.

Did you try setting setMemoryMode to OFFHEAP_TIERED on CacheConfiguration?
I think it may help a bit.

Sergi

2016-07-25 10:37 GMT+03:00 November :

> Hi Sergi
>
> Thanks for reply.
>
> I have tried to store them in hashmap. It use about 5GB memory.
>
> By the way, my ignite cache set backups = 0 and has one index field.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Question-about-GridGain-will-typically-add-around-200-bytes-overhead-to-each-entry-tp6411p6507.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Question about GridGain will typically add around 200 bytes overhead to each entry

[November]

Hi Sergi

Thanks for reply.

I have tried to store them in hashmap. It use about 5GB memory.

By the way, my ignite cache set backups = 0 and has one index field.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Question-about-GridGain-will-typically-add-around-200-bytes-overhead-to-each-entry-tp6411p6507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Question about GridGain will typically add around 200 bytes overhead to each entry

[Sergi Vladykin]

I believe you already have some logic to parse this file and put all the
entries to cache. To test overhead you can store all your entries in simple
HashMap instead of Ignite cache and see the difference in memory
consumption.

Sergi

2016-07-25 9:09 GMT+03:00 November :

> Thanks for reply.
>
> I use jvisualvm.exe to see the heap dump of ignite node, but it will cause
> the ignite node crash.
>
> That's my situation.
> I have a file about 1GB. It has 70 million lines. I use each line (about 15
> bytes one line) as a cache entry in ignite.  Finally it use about 32GB
> memory. That's why I think it's the overhead use the most memory.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Question-about-GridGain-will-typically-add-around-200-bytes-overhead-to-each-entry-tp6411p6503.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: How to control the data unbalance among all the server nodes?

[Vladislav Pyatkov]

Hello Jason,

1) Yes, data (cache entry) distributed in partition, depends on hash code
of key (key.hashCode() % parts).

2) The basis for the rebalancing of partitions are leave node from cluster
or join a node to cluster. RendezvousAffinityFunction does so that each
node has roughly the same count of partitions. Also
(RendezvousAffinityFunction)  partition distribution as less as possible
difference than the previous.

Thus we can assume (in case the hash code has uniform distribution) that
each node has about a different part of the data.

On Mon, Jul 25, 2016 at 8:41 AM, Jason  wrote:

> hi Ignite team,
>
> 1. Ignite does the data balance based on the cache partitions' balance not
> the data, right? E.g. if there's data skew in some keys, how to handle?
>
> 2. RendezvousAffinityFunction cannot ensure all the nodes have the
> completely same partitions and this should be an advantage to reduce the
> impact of network fluctuations, such as Node leave/join, but how to control
> the unbalance rate of all the nodes, say partition # in the node with max
> partitions / that of min? If the max node reaches the memory limit, will it
> do re-balance? Is there any config to control this, say 10%, 20%?
>
> Thanks,
> -Jason
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/How-to-control-the-data-unbalance-among-all-the-server-nodes-tp6502.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov