Re: Is Ignite persistency store row store or column store?

[Roman Guseinov]

Hi,

Ignite persistency store is not a column-oriented storage. In simple terms,
it stores data like a row-oriented one: K1,V1,K2,V2... To get more details
please refer [1], [2].

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
[2]
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-IgnitePersitentStore

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Query on Ignite web console

[Alexey Kuznetsov]

 Vishwas,

There is no such API to get security token of Web Console.

What are you tying to implement?

Actually there no need in web console and web agent if you just need to
execute some SQL.
You may use JDBC/ODBC/REST see:

https://apacheignite.readme.io/docs/rest-api
https://apacheignite-sql.readme.io/docs/sql-reference-overview


On Mon, Apr 16, 2018 at 11:21 PM, vbm  wrote:

> Hi Denis,
>
> Thanks for the information. I have started using web console now.
>
> When the user is created as part of web console, a security token is
> created.
> The ignite-web-agent is dependent on this security token.
>
> Currently the starting of a ignite-web agent looks more of a manual step
> because of this security token. I wanted to automate the process of ignite
> web agent connecting to the console.
>
> Are there any rest api for web console with which we can create user and
> also get the security token for a particular user.
>
>
> Regards,
> Vishwas
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Alexey Kuznetsov

Is Ignite persistency store row store or column store?

[zhouxy1123]

hi , is Ignite persistency store   row store or column store?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Efficiently determining if cache keys belong to the local server node

[Raymond Wilson]

Hi Stan,

Your understanding is correct.

I'm aware of the AffinityRun and AffinityCall methods, and their simple key
limitation.

My use case may require 100,000 or more elements of information to be
processed, so I don't want to call AffinityRun/Call that often. Each of
these elements is identified by a key that is very efficiently encoded into
the request (at the ~1 bit per key  level)

Further, each of those elements identifies work units that in themselves
could have 100,000 or more different elements to be processed.

One approach would be to explicitly break up the request into smaller ones,
each targeted at a server node. But that requires the requestor to have
intimate knowledge of the composition of the grid resources deployed, which
is not desirable.

The approach I'm looking into here is to have each server node receive the
same request via Cluster.Broadcast(), and for those nodes to determine which
elements in the overall request via the Key -> Partition affinity mapping.
The mapping itself is very efficient, and as I noted in my original post
determining the partition -> node map seems simple enough to do.

I'm unsure of the performance of requesting that mapping for every request,
versus caching it and adding watchers for rebalancing and topology change
events to invalidate that cache mapping as needed (and how to wire those
up).

Thanks,
Raymond.

-Original Message-
From: Stanislav Lukyanov [mailto:stanlukya...@gmail.com]
Sent: Tuesday, April 17, 2018 12:02 AM
To: user@ignite.apache.org
Subject: RE: Efficiently determining if cache keys belong to the local
server node

// Bcc’ing off dev@ignite list for now as it seems to be rather a user-space
discussion.

Hi,

Let me take a step back first. It seems a bit like an XY problem
(https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem),
so I’d like to clarify the goals before diving into your current solution.

AFAIU you want to process certain entries in your cache locally on the
server that caches these entries. Is that correct?
Have you looked at affinityRun and affinityCall
(https://apacheignite.readme.io/docs/collocate-compute-and-data)? If yes,
why they don’t work for you?
One limitation with these methods is that they accept a single key to
process. Can you process your keys one by one, or do you need to access
multiple keys at once?

Thanks,
Stan

From: Raymond Wilson
Sent: 15 апреля 2018 г. 10:55
To: user@ignite.apache.org
Cc: d...@ignite.apache.org
Subject: Efficiently determining if cache keys belong to the local server
node

I have a type of query that asks for potentially large numbers of
information elements to be computed. Each element has an affinity key that
maps it to a server node through an IAffinityFunction.



The way the question is asked means that a single query broadcast to the
compute projection (owning the cache containing the source data for the
request) contains the identities of all the pieces of information needed to
be processed.



Each server node then scans the elements requested and identifies which ones
are its responsibility according to the affinity key.



Calculating the partition ID from the affinity key is simple (I have an
affinity function set up and supplied to the cache configuration, or I could
use IAffinity.GetPartition()), so the question became: How do I know the
server node executing the query is responsible for that partition, and so
should process this element? IE: I need to derive the vector of primary or
backup  partitions that this node is responsible for.



I can query the partition map and return it, like this:



ICacheAffinity affinity = Cache.Ignite.GetAffinity(Cache.Name);

public Dictionary primaryPartitions =
affinity.GetPrimaryPartitions(Cache.Ignite.GetCluster().GetLocalNode()).ToDictionary(k
=> k, v => true);



This lets me do a dictionary lookup, but its less efficient that having a
complete partition map with simple array lookup semantics, like this:



ICacheAffinity affinity = Cache.Ignite.GetAffinity(Cache.Name);

bool[] partitionMap = new bool[affinity.Partitions];



foreach (int partition in
affinity.GetBackupPartitions(Cache.Ignite.GetCluster().GetLocalNode()))

partitionMap[partition] = true;



This is a nice lookup for the query to determine which elements are its
responsibility from the overall request.



I’m not sure of the performance profile of this approach if I end up doing
it a lot, so I’m considering caching this lookup and invalidate it if any
event occurs that could modify the key -> partition map.



Questions:



   1. How big is the penalty when determining the full partition map like
   this?
   2. If I decide to invalidate the cached map, what are all the events I’d
   need to listen to?
  1. Rebalancing events?:I found CacheRebalancingEvent, but I’m not
  sure if this gives visibility to the points in time when a rebalanced
  

Re: Ignite query statement reused/cached

[dkarachentsev]

Hi,

For sure Ignite caches queries, that's why first request runs much longer
than rest ones.

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Query on Ignite web console

[vbm]

Hi Denis,

Thanks for the information. I have started using web console now. 

When the user is created as part of web console, a security token is
created.
The ignite-web-agent is dependent on this security token.

Currently the starting of a ignite-web agent looks more of a manual step
because of this security token. I wanted to automate the process of ignite
web agent connecting to the console. 

Are there any rest api for web console with which we can create user and
also get the security token for a particular user. 


Regards,
Vishwas



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: The thread which is inserting data into Ignite is hung

[dkarachentsev]

Hi Praveen,

Stack traces only show that thread is waiting for response, to get the full
picture, please attach full logs and thread dumps at the moment of hang from
all nodes. I need from all nodes, because actual issue happened on remote
node. 

Also, according to last exception, there might be connectivity issue, when
client cannot get response from cluster.

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Hibernate 5.2 support

[Andrey Mashenkov]

Hi,

As ticket still has no patch, most likely it will be postponed to the next
version.

On Mon, Apr 16, 2018 at 4:27 PM, kestas  wrote:

> I see there is ticket registered for Hibernate 5.2 support:
> https://issues.apache.org/jira/browse/IGNITE-5848
> It is planned for 2.5 release. And release page says
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.5 the
> 2.5
> version should come out on Apr 30, so it is only 2 weeks away, however,
> there is no activity on Hibernate integration ticket.   So should we expect
> Hibernate 5.2 support in upcoming Apr 30 release?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Andrey V. Mashenkov

Hibernate 5.2 support

[kestas]

I see there is ticket registered for Hibernate 5.2 support:
https://issues.apache.org/jira/browse/IGNITE-5848
It is planned for 2.5 release. And release page says
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.5 the 2.5
version should come out on Apr 30, so it is only 2 weeks away, however,
there is no activity on Hibernate integration ticket.   So should we expect
Hibernate 5.2 support in upcoming Apr 30 release? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Efficiently determining if cache keys belong to the local server node

[Stanislav Lukyanov]

// Bcc’ing off dev@ignite list for now as it seems to be rather a user-space 
discussion.

Hi,

Let me take a step back first. It seems a bit like an XY problem 
(https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem),
so I’d like to clarify the goals before diving into your current solution.

AFAIU you want to process certain entries in your cache locally on the server 
that caches these entries. Is that correct?
Have you looked at affinityRun and affinityCall 
(https://apacheignite.readme.io/docs/collocate-compute-and-data)? If yes, why 
they don’t work for you?
One limitation with these methods is that they accept a single key to process. 
Can you process your keys one by one, or do you need to access multiple keys at 
once?

Thanks,
Stan 

From: Raymond Wilson
Sent: 15 апреля 2018 г. 10:55
To: user@ignite.apache.org
Cc: d...@ignite.apache.org
Subject: Efficiently determining if cache keys belong to the local server node

I have a type of query that asks for potentially large numbers of
information elements to be computed. Each element has an affinity key that
maps it to a server node through an IAffinityFunction.



The way the question is asked means that a single query broadcast to the
compute projection (owning the cache containing the source data for the
request) contains the identities of all the pieces of information needed to
be processed.



Each server node then scans the elements requested and identifies which
ones are its responsibility according to the affinity key.



Calculating the partition ID from the affinity key is simple (I have an
affinity function set up and supplied to the cache configuration, or I
could use IAffinity.GetPartition()), so the question became: How do I know
the server node executing the query is responsible for that partition, and
so should process this element? IE: I need to derive the vector of primary
or backup  partitions that this node is responsible for.



I can query the partition map and return it, like this:



ICacheAffinity affinity = Cache.Ignite.GetAffinity(Cache.Name);

public Dictionary primaryPartitions =
affinity.GetPrimaryPartitions(Cache.Ignite.GetCluster().GetLocalNode()).ToDictionary(k
=> k, v => true);



This lets me do a dictionary lookup, but its less efficient that having a
complete partition map with simple array lookup semantics, like this:



ICacheAffinity affinity = Cache.Ignite.GetAffinity(Cache.Name);

bool[] partitionMap = new bool[affinity.Partitions];



foreach (int partition in
affinity.GetBackupPartitions(Cache.Ignite.GetCluster().GetLocalNode()))

partitionMap[partition] = true;



This is a nice lookup for the query to determine which elements are its
responsibility from the overall request.



I’m not sure of the performance profile of this approach if I end up doing
it a lot, so I’m considering caching this lookup and invalidate it if any
event occurs that could modify the key -> partition map.



Questions:



   1. How big is the penalty when determining the full partition map like
   this?
   2. If I decide to invalidate the cached map, what are all the events I’d
   need to listen to?
  1. Rebalancing events?:I found CacheRebalancingEvent, but I’m not
  sure if this gives visibility to the points in time when a rebalanced
  partition becomes active on the new node and so the partition map changes
  2. Topology change events? (eg: adding a new backup node without
  rebalancing (if that is a thing) I looked for an event like that but have
  not found it so far, though I do know the affinity function can
respond to
  this via AssignPartitions()
   3. How do I provide my own affinity key mapper to for keys to partition
   IDs, but allow Ignite to map the partitions to nodes. The IAffinityFunction
   implementation requires both steps to be implemented. I’d prefer not to
   have the partition -> server mapping responsibility as this requires
   persistent configuration on the nodes to ensure stable mapping.



Thanks,

Raymond.

Re: Apache Ignite nightly release builds

[Petr Ivanov]

Hi, Igniters!


I’m glad to inform that starting today Apache Ignite Nightly Builds provide 
Nuget and Maven stagings at MyGet [1].

Instructions on connecting to these feeds are here:
 — Nuget [2]
 — Maven [3]


Enjoy.


[1] https://www.myget.org/gallery/apache-ignite-nightly 

[2] http://docs.myget.org/docs/walkthrough/getting-started-with-nuget 

[3] http://docs.myget.org/docs/walkthrough/getting-started-with-maven 


> On 24 Mar 2018, at 11:42, Dmitriy Setrakyan  wrote:
> 
> Thanks, Denis. It should be added to the download page, I updated the ticket.
> 
> On Sat, Mar 24, 2018 at 5:48 AM, Denis Magda  > wrote:
> Created a JIRA ticket for that:
> https://issues.apache.org/jira/browse/IGNITE-8040 
> 
> 
> --
> Denis
> 
> On Fri, Mar 23, 2018 at 1:27 AM, Dmitriy Setrakyan  >
> wrote:
> 
> > Awesome! Finally instead of asking our users to build from the master, we
> > can provide a link to the nightly build instead.
> >
> > Denis, can you please add these links to the website?
> >
> > D.
> >
> > On Thu, Mar 22, 2018 at 1:27 PM, Petr Ivanov  > > wrote:
> >
> >> It works, thanks!
> >>
> >>
> >> Here is updated links for Artifacts and Changes respectively with silent
> >> guest login (can be added to bookmarks):
> >> * https://ci.ignite.apache.org/viewLog.html?buildId=lastSucces 
> >> 
> >> sful=Releases_NightlyRelease_RunApacheIgnite
> >> NightlyRelease=artifacts=1
> >> * https://ci.ignite.apache.org/viewLog.html?buildId=lastSucces 
> >> 
> >> sful=Releases_NightlyRelease_RunApacheIgnite
> >> NightlyRelease=buildChangesDiv=1
> >>
> >>
> >>
> >> > On 22 Mar 2018, at 13:06, Vitaliy Osipov  >> > > wrote:
> >> >
> >> > 
> >>
> >>
> >
> 

Re: how to use bigbench test ignite performance

[aealexsandrov]

Hi,

As I know Ignite provides its own benchmarks that used Yardstick framework.

Documentation:

https://apacheignite.readme.io/docs/perfomance-benchmarking

yardstick:

https://github.com/gridgain/yardstick

In Ignite sources exists IgniteAbstractBenchmark class that could be used
for creation of the custom benchmarks. 

Best Regards.
Andrei










--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite query statement reused/cached

[kotamrajuyashasvi]

Hi

Does Ignite cache query Strings and reuse cached statements if same query
String is to be executed?
Suppose query String is same and the same query is run multiple times with
different binding values, Does Ignite just reuse the cached query statement
with the new bindings? Does it impact performance ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/