What is H2 Database.

[Malashree]

What is H2 Database how it helps us in Apache Ignite.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: YCSB Benchmark

[Ivan Artiukhov]

Hi!

1) Are you running Ignite and YCSB nodes on the same machine? How many YCSB
nodes do you start?
2) What DataRegionConfiguration do you set in ignite.xml config file? Does
the whole dataset of 1M entries fit in RAM ()?
3) Do other Ignite parameters have the same value as in the blog post you
are referencing to?

вт, 18 сент. 2018 г. в 9:18, Shay Alon :

> Thanks for your reply.
> I was comparing to the results published here:
>
> https://www.gridgain.com/resources/blog/apacher-ignitetm-and-apacher-cassandratm-benchmarks-power-in-memory-computing
> I run the benchmark on very powered machine with 72 cpu and 500G RAM. I
> used 32 threads.
>
>
>
>
>
> On Tue, Sep 18, 2018 at 8:38 AM Ivan Artiukhov  wrote:
>
>> Hi!
>>
>> What results are you using for comparison? Please provide URL. There are
>> plenty of factors which affects performance. Hardware, number of server
>> and
>> client nodes, Ignite memory mode, replication factor (number of backups),
>> number of benchmarking threads, data amount (record count) -- just to
>> mention a few.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Setting performance expectations

[Daryl Stultz]

is my understanding correct, that you first select 400k records from DB, DB
is remote and this takes 700ms, then you select all these 400k records from
Ignite and now it takes 500ms, but in this case ignite is a local embedded
node?

All of this runs on my laptop, no network involved. MacBook Pro 2018, 
PostgreSQL 9.6. VM flags "-XX:+PrintGC -Xmx5000m -Xms4000m"

Your suggestion to print GC logging led me to increase min and max memory and 
free up RAM by closing other apps. Things sped up quite a bit. Does this mean 
Ignite was serializing and writing to disk? I didn't have onHeapCacheEnabled 
for this run, does that mean it's serializing but to off-heap memory, if memory 
is available? After adding setOnheapCacheEnabled(true) the metrics didn't 
change much. So here, database to map is 953 ms, and get from cache one at a 
time is 723ms, not much faster. Getting all in one batch is twice as fast as 
loading from the DB, but I still expect reading from Ignite to be far faster 
than the DB.

Here is the output for the below example:

2018-09-20 20:14:23.079 DEBUG - c.o.i.IgniteSpec : load 428765 rows into local 
map in 953 ms
[GC (Allocation Failure)  1047513K->282437K(4275200K), 0.1436511 secs]
2018-09-20 20:14:25.363 DEBUG - c.o.i.IgniteSpec : putAll map to cache in 2284 
ms
2018-09-20 20:14:26.086 DEBUG - c.o.i.IgniteSpec : get 428765 elements one at a 
time in 723 ms
2018-09-20 20:14:26.575 DEBUG - c.o.i.IgniteSpec : get all keys at once in 489 
ms
[20:14:26] Ignite node stopped OK [uptime=00:00:04.770]
2018-09-20 20:14:26.805 DEBUG - c.o.w.d.Profiler : [[ cleanDatabase >>> END: 1; 
Total:1 ]]
2018-09-20 20:14:26.805 DEBUG - c.o.u.s.EbeanExtension : END TransactionEmpty 
test suite.
2018-09-20 20:14:26.820 DEBUG - i.e.s.l.ShutdownManager : Shutting down
2018-09-20 20:14:26.820 DEBUG - i.e.s.c.DefaultServer : Shutting down 
EbeanServer unittest-ws
[GC (System.gc())  992502K->265050K(4616704K), 0.0797061 secs]
[Full GC (System.gc())  265050K->171852K(4616704K), 0.3171326 secs]

Here's the code. It's Kotlin. (I did do a very basic get/put test of Kotlin vs 
Java, though no DB involved, performance was comparable.)

data class ScheduledAssignmentDto(
val id: Int,
val userId: Int,
val roleId: Int,
val date: Date
)

class IgniteSpec : Specification({

val log = Log(IgniteSpec::class.java)

val config = IgniteConfiguration()
config.setClientMode(false)

val ignite = Ignition.start(config)

val cacheCfg = CacheConfiguration()
cacheCfg.name = "ScheduledAssignmentDto"
val cache = ignite.getOrCreateCache(cacheCfg)

val sql = """select scheduledAssignmentId as id, userId, roleId, 
ScheduleDays.scheduleDate as date
from ScheduledAssignments
join ScheduleDays on ScheduleDays.scheduleDayId = 
ScheduledAssignments.scheduleDayId
order by scheduledAssignmentId
""".trimMargin()

var rowCount = 1
val keys = mutableSetOf()

val map = mutableMapOf()
val loadTime = measureTimeMillis {
TenantResourceSupplier.getEbeanServer().createSqlQuery(sql)
.findEachRow { resultSet: ResultSet, rowNumber: Int ->
val id = resultSet.getInt("id")
val userId = resultSet.getInt("userId")
val roleId = resultSet.getInt("roleId")
val date = Date(resultSet.getDate("date").time)
val dto = ScheduledAssignmentDto(id, userId, roleId, date)
map.put(id, dto)
rowCount = rowNumber
keys += id
}
}
log.debug("load $rowCount rows into local map in $loadTime ms")

val cacheMapTime = measureTimeMillis {
cache.putAll(map)
}
log.debug("putAll map to cache in $cacheMapTime ms")


val readTime = measureTimeMillis {
keys.forEach {
cache.get(it)
}
}
log.debug("get $rowCount elements one at a time in $readTime ms")

val readTime2 = measureTimeMillis {
cache.getAll(keys)
}
log.debug("get all keys at once in $readTime2 ms")

ignite.close()

})

Thanks for the help.

/Daryl

Re: Too Many open files (native persistence bin files)

[Dmitriy Pavlov]

Hi,

Is it reproduced if you remove direct io from class path?

Sincerely
Dmitry Pavlov

вт, 18 сент. 2018 г., 16:01 Evgenii Zhuravlev :

> Hi,
>
> How many caches do you have?
>
> Evgenii
>
> вт, 18 сент. 2018 г. в 15:38, kvenkatramtreddy  >:
>
>> one correction: lsof showing the information for threads as well, so that
>> is
>> why it is showing number of Threads * file descriptor.
>>
>> My current ulimit is 32635, open files in 6000 but still we are receiving
>> too many files open.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: IgniteSparkSession exception:Ignite instance with provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite instance? [name=null]

[vkulichenko]

Does the configuration file exist on worker nodes? I looks like it actually
fails to start there for some reason, and then you eventually get this
exception. Are there any other exception in worker/executor logs?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is there a way to use Ignite optimization and Spark optimization together when using Spark Dataframe API?

[vkulichenko]

If join is indexed and collocated, it still can be pretty fast. Do you have a
particular query that is slower with optimization than without?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ignite.events().remoteListen unmarshal0 throw class not found

[vkulichenko]

Can you show the full trace? Most likely it's failing on server side because
you don't actually have the class there.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Query 3x slower with index

[eugene miretsky]

Thanks Ilya,

Tried it, no luck. It performs the same as when using category_id index
alone (slow).
  Any combindation I try either uses AFFINITY_KEY or category index. When
it uses category index it runs slowers.

Also, when AFFINITY_KEY key is used, the jobs runs on 32 threads (my query
parallelism settings ) when category_id is used, the jobs runs on one
thread most of the time (first few seconds it looks like more threads are
doing work).

Please help on this. It seems like a very simple use case (using affinity
key and another index), either I am doing something extremly silly, or I
stumbled on a bug in Ignite that's effecting a lot of people.

Cheers,
Eugene

On Thu, Sep 20, 2018 at 6:22 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> > 2) ga_customer_and_category_id: on customer_id and category_id
>
> Have you tried to do an index on category_id first, customer_id second?
> Note that Ignite will use only one index when joining two tables and that
> in your case it should start with category_id.
>
> You can also try adding affinity key to this index in various places, see
> if it helps further.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 сент. 2018 г. в 21:27, eugene miretsky :
>
>> Hi Ilya,
>>
>> I created 4 indexs on the table:
>> 1) ga_pKey: on customer_id, dt, category_id (that's our primary key
>> columns)
>> 2) ga_customer_and_category_id: on customer_id and category_id
>> 2) ga_customer_id: on customer_id
>> 4) ga_category_id: on category_id
>>
>>
>> For the first query (category in ()), the execution plan when using the
>> first 3 index is exactly the same  - using /* PUBLIC.AFFINITY_KEY */
>> When using #4 (alone or in combination with any of the other 3)
>>
>>1. /* PUBLIC.AFFINITY_KEY */ is replaced with  /*
>>PUBLIC.GA_CATEGORY_ID: CATEGORY_ID IN(117930, 175930, 175940, 175945,
>>101450) */
>>2. The query runs slower.
>>
>> For the second query (join on an inlined table) the behaviour is very
>> similar. Using the first 3 indexes results in the same plan - using  /*
>> PUBLIC.AFFINITY_KEY */ and  /* function: CATEGORY_ID = GA__Z0.CATEGORY_ID
>> */.
>> When using #4 (alone or in combination with any of the other 3)
>>
>>1. /* function */ and /* PUBLIC.GA_CATEGORY_ID: CATEGORY_ID =
>>CATS__Z1.CATEGORY_ID */ are used
>>2. The query is much slower.
>>
>>
>> Theoretically the query seems pretty simple
>>
>>1. Use affinity key  to make sure the query runs in parallel and
>>there are no shuffles
>>2. Filter rows that match category_id using the category_id index
>>3. Used customer_id index for the group_by (not sure if this step
>>makes sense)
>>
>> But I cannot get it to work.
>>
>> Cheers,
>> Eugene
>>
>>
>>
>>
>> On Tue, Sep 18, 2018 at 10:56 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I can see you try to use _key_PK as index. If your primary key is
>>> composite, it won't work properly for you. I recommend creating an explicit
>>> (category_id, customer_id) index.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> вт, 18 сент. 2018 г. в 17:47, eugene miretsky >> >:
>>>
 Hi Ilya,

 The different query result was my mistake - one of the categoy_ids was
 duplicate, so in the query that used join, it counted rows for that
 category twice. My apologies.

 However, we are still having an issue with query time, and the index
 not being applied to category_id. Would appreciate if you could take a
 look.

 Cheers,
 Eugene

 On Mon, Sep 17, 2018 at 9:15 AM Ilya Kasnacheev <
 ilya.kasnach...@gmail.com> wrote:

> Hello!
>
> Why don't you diff the results of those two queries, tell us what the
> difference is?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 17 сент. 2018 г. в 16:08, eugene miretsky <
> eugene.miret...@gmail.com>:
>
>> Hello,
>>
>> Just wanted to see if anybody had time to look into this.
>>
>> Cheers,
>> Eugene
>>
>> On Wed, Sep 12, 2018 at 6:29 PM eugene miretsky <
>> eugene.miret...@gmail.com> wrote:
>>
>>> Thanks!
>>>
>>> Tried joining with an inlined table instead of IN as per the second
>>> suggestion, and it didn't quite work.
>>>
>>> Query1:
>>>
>>>- Select COUNT(*) FROM( Select customer_id from GATABLE3  use
>>>Index( ) where category_id in (9005, 175930, 175930, 
>>> 175940,175945,101450,
>>>6453) group by customer_id having SUM(product_views_app) > 2 OR
>>>SUM(product_clicks_app) > 1 )
>>>- exec time = 17s
>>>- *Result: 3105868*
>>>- Same exec time if using AFFINITY_KEY index or "_key_PK_hash or
>>>customer_id index
>>>- Using an index on category_id increases the query time 33s
>>>
>>> Query2:
>>>
>>>- Select COUNT(*) FROM( Select customer_id from GATABLE3 ga  use
>>>index (PUBLIC."_key_PK") inner join 

Re: Message grid failure due to userVersion setting

[Ilya Kasnacheev]

Hello!

I think we should just wait for someone to tackle IGNITE-7905
 (or actively contribute
for a chance).

Regards,
-- 
Ilya Kasnacheev


вт, 18 сент. 2018 г. в 21:59, Dave Harvey :

> Thanks Ilya,
>
> As I understand  this a bit more, it seems like IGNITE-7905
>  is really the same
> basic flaw in user version not working as documented. Ignite-7905
> reproduction is simply to set a non-zero userVersion in a ignite.xml (
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/DeploymentMode.html),
> and then
>
> // Connect to the cluster.Ignite ignite = Ignition.start();
> // Activate the cluster. Automatic topology initialization occurs // only if 
> you manually activate the cluster for the very first time. 
> ignite.cluster().active(true);
>
>
> The activation then throws an exception on the server, because the server
> already has the same built-in Ignite class.
>
> As I understand the documentation, since the built in Ignite class is not
> excluded, it should not even consider peer class loading because the class
> exists locally.It should just use the local class.
>
> -DH
>
> On Tue, Sep 18, 2018 at 9:00 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> I'm not familiar with these areas very much, but if you had a reproducer
>> project I could take a look.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 17 сент. 2018 г. в 19:32, Dave Harvey :
>>
>>> I probably did not explain this clearly.  When sending a message from
>>> server to client using the message grid, from a context unrelated to any
>>> client call, the server, as you would expect uses its installed libraries,
>>> and userVersion 0.For some reason, when the client receives this
>>> message, it require that the user version match it's current user version.
>>>
>>> The use case is we have a stable set of libraries on the server, and the
>>> server wants to send a topic based message to the client, using only the
>>> type "String".   Unrelated to this, the client is using the compute grid,
>>> where P2P is used, but that is interfering with basic functionality.
>>> This,  IGNITE-7905  ,
>>> and the paucity  of  results when I google for "ignite userVersion"  makes
>>> it clear that shooting down classes in CONTINUOUS mode with userVersion is
>>> not completely thought through.  We certainly never want to set a
>>> userVersion on the servers.
>>>
>>> The documentation for P2P says:
>>> "
>>>
>>>1. Ignite will check if class is available on local classpath (i.e.
>>>if it was loaded at system startup), and if it was, it will be returned. 
>>> No
>>>class loading from a peer node will take place in this case."
>>>
>>> Clearly, java.lang.String is on the local class path.So it seems
>>> like a user version mismatch should not be a reason to reject a class that
>>> is on the local classpath.
>>>
>>> On Mon, Sep 17, 2018 at 11:01 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 I think that Ignite cannot unload old version of code, unless it is
 loaded with something like URI deployment module.
 Version checking is there but server can't get rid of old code if it's
 on classpath.

 Regards,
 --
 Ilya Kasnacheev


 пн, 17 сент. 2018 г. в 16:47, Dave Harvey :

> We have a client that uses the compute grid and message grid, as well
> as the discovery API.  It communicates with a server plugin.   The cluster
> is configured for CONTINUOUS peer class loading.  In order to force the
> proper code to be loaded for the compute tasks, we change the user 
> version,
> e.g., to 2.
>
> If the server sends the client a message on the message grid, using
> java.lang.string, the client fails because the user version sent for
> java.lang.string is 0, but the client insists on 2.
>
> How is this supposed to work?   Our expectation was that the message
> grid should not be affected by peer class loading settings.
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and 
> may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as
> a Service (SaaS) for business. Providing a *safer* and *more useful*
> place for your human generated data. Specializing in; Security, archiving
> and compliance. To find out more Click 

ignite.events().remoteListen unmarshal0 throw class not found

[wangsan]

A server and a client, then client call:
   line_1: CacheNodeFilter rmtFilter = new CacheNodeFilter(serviceType,
uuid);
   line_2: ignite.events().remoteListen(listenerDispatcher, rmtFilter,
EventType.EVT_CACHE_OBJECT_EXPIRED);

I am quite sure all the  server and client have the class CacheNodeFilter
with same version.
And CacheNodeFilter implements IgnitePredicate
But in client it will throw:
catch (ClassNotFoundException e) {
throw new IgniteCheckedException("Failed to find class with
given class loader for unmarshalling " +
"(make sure same versions of all classes are available on
all nodes or enable peer-class-loading) " +
"[clsLdr=" + clsLdr + ", cls=" + e.getMessage() + "]", e);
}

It is weird that exception throw on line_2 but not line_1.If line_1 run
successful,the class "CacheNodeFilter" must be loaded by classloader, So why
line_2 still throw ClassNotFoundException?

And the classloader is  appclassloader.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unable to get the ignite cache metrics

[ezhuravlev]

Hi,

Looks like you try to access local cache metrics on client node, which will
be empty, because clients doesn't store cache. Try to use
CacheClusterMetricsMXBeanImpl instead.

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Inconsistent data.

[Mikhail]

Hi Shrikant,

What kind of requests do you mean? If you make affinity calls, ignite always
sends the task to a node which has data, if new node doesn't have data,
tasks just won't go to this node.
Could you please explain in more details what you mean by requests?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Setting performance expectations

[Mikhail]

Hi Daryl,

Could you please share with us your benchmark?

is my understanding correct, that you first select 400k records from DB, DB
is remote and this takes 700ms, then you select all these 400k records from
Ignite and now it takes 500ms, but in this case ignite is a local embedded
node?

Numbers that you provided don't look good for me, even if your database
cached all record in memory and sent them to your app right from RAM there's
a network, which definitely should be slower.
I want to check your code and check how you make measures, also there might
be something wrong with the configuration. 
Also could you please gather GC logs, I want sure that we don't spend the
time for GC.

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is there a way to use Ignite optimization and Spark optimization together when using Spark Dataframe API?

[Ray]

Hi Val, thanks for the reply.

I'll try again and let you know if I missed something.

By "Ignite is not optimized for join", I mean currently Ignite only supports
nest loop join which is very inefficient when joining two large table.
Please refer to these two tickets for details.
https://issues.apache.org/jira/browse/IGNITE-6201
https://issues.apache.org/jira/browse/IGNITE-6202



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data migration using persistent Queues

[ipsxd]

Yes, Ignite Queue. In the mean time I did some tests and it seems that adding
fields is not a problem but removing or doing some more complicated stuff
would be. 

Another option is to implement Externalizable and implement the logic there.
So I think a good approach would be a combination between methods. Additions
are backwards compatible so this feature will be used. For incompatible
changes migrations of data to other queues will be used. To have a
complicated code in serialization/deserialization is not desirable either as
this would affect performance. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ScanQuery throwing Exception for java Thin client while peerclassloading is enabled

[Ilya Kasnacheev]

Hello!

Not unless you have value class on server nodes. If you don't have value
class available (or key for that matter) you are restricted ro
BinaryObject's.

Regards,
-- 
Ilya Kasnacheev


ср, 19 сент. 2018 г. в 10:26, Saby :

> Hi Ilya,
> Thanks for your response, yes I had tried with withKeepBinary() option. If
> I
> call with withKeepBinary() option then there no deserialization related
> exception is coming but the returned entries are being wrapped to
> 'BinaryObjectImpl' and hence getting ClassCastException. Is there any
> option
> to get the entries of rawtype?
>
> I'm using the following code snippet to fetch the data:
>
>
> public class Predicate implements IgniteBiPredicate
> {
>
> /**
>  *
>  */
> private static final long serialVersionUID = 1L;
>
> @Override
> public boolean apply(K e1, V e2) {
> return true;
> }
> }
>
> public class IgniteValueClass implements Binarylizable, Serializable {
> private static final long serialVersionUID = 50283244369719L;
> @QuerySqlField(index = true)
> String id;
> @QuerySqlField(index = true)
> String val1;
>
> @QuerySqlField
> String val2;
>
> public String getId() {
> return id;
> }
>
> public void setId(String id) {
> this.id = id;
> }
>
> public String getVal1() {
> return val1;
> }
>
> public void setVal1(String val1) {
> this.val1 = val1;
> }
>
> public String getVal2() {
> return val2;
> }
>
> public void setVal2(String val2) {
> this.val2 = val2;
> }
>
> @Override
> public void writeBinary(BinaryWriter writer) throws
> BinaryObjectException {
> writer.writeString("id", id);
> writer.writeString("val1", val1);
> writer.writeString("val2", val2);
>
> }
>
> @Override
> public void readBinary(BinaryReader reader) throws
> BinaryObjectException {
> id = reader.readString("id");
> val1 = reader.readString("val1");
> val2 = reader.readString("val2");
>
> }
>
> }
>
> public Iterator> getAllEntries(String areaName)
> throws NotClientException {
> checkConnectedAndExists(areaName);
> Query> sql = new
> ScanQuery<>(new
> Predicate<>());
> try (QueryCursor> cursor =
> ignite.cache(areaName).withKeepBinary().query(sql)) {
> return returnEntry(cursor);
> }
> }
>
> Thanks
> Saby
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite service for redhat/centos

[Ilya Kasnacheev]

Hello!

We have RPM build which may be a good starting point. It has service
template.

Regards,
-- 
Ilya Kasnacheev


чт, 20 сент. 2018 г. в 13:59, Nadba :

> Do we have this feature in 2.6? How do we run ignite as a service or a
> process detached from console? Any solution other than nohup and &?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: .net decimal being stored as Other in ignite.

[Ilya Kasnacheev]

Hello!

This problem I do not observe. I can see values with decimal points in
dbeaver.

Regards.
-- 
Ilya Kasnacheev


чт, 20 сент. 2018 г. в 13:43, wt :

> thanks, just another FYI , the values in that column cut off anything after
> the decimal point while it is other so 3.456 becomes 3
>
> for now we are storing as a string and casting to decimal when we query
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite service for redhat/centos

[Nadba]

Do we have this feature in 2.6? How do we run ignite as a service or a
process detached from console? Any solution other than nohup and &?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: .net decimal being stored as Other in ignite.

[wt]

thanks, just another FYI , the values in that column cut off anything after
the decimal point while it is other so 3.456 becomes 3

for now we are storing as a string and casting to decimal when we query



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Query 3x slower with index

[Ilya Kasnacheev]

Hello!

> 2) ga_customer_and_category_id: on customer_id and category_id

Have you tried to do an index on category_id first, customer_id second?
Note that Ignite will use only one index when joining two tables and that
in your case it should start with category_id.

You can also try adding affinity key to this index in various places, see
if it helps further.

Regards,
-- 
Ilya Kasnacheev


ср, 19 сент. 2018 г. в 21:27, eugene miretsky :

> Hi Ilya,
>
> I created 4 indexs on the table:
> 1) ga_pKey: on customer_id, dt, category_id (that's our primary key
> columns)
> 2) ga_customer_and_category_id: on customer_id and category_id
> 2) ga_customer_id: on customer_id
> 4) ga_category_id: on category_id
>
>
> For the first query (category in ()), the execution plan when using the
> first 3 index is exactly the same  - using /* PUBLIC.AFFINITY_KEY */
> When using #4 (alone or in combination with any of the other 3)
>
>1. /* PUBLIC.AFFINITY_KEY */ is replaced with  /*
>PUBLIC.GA_CATEGORY_ID: CATEGORY_ID IN(117930, 175930, 175940, 175945,
>101450) */
>2. The query runs slower.
>
> For the second query (join on an inlined table) the behaviour is very
> similar. Using the first 3 indexes results in the same plan - using  /*
> PUBLIC.AFFINITY_KEY */ and  /* function: CATEGORY_ID = GA__Z0.CATEGORY_ID
> */.
> When using #4 (alone or in combination with any of the other 3)
>
>1. /* function */ and /* PUBLIC.GA_CATEGORY_ID: CATEGORY_ID =
>CATS__Z1.CATEGORY_ID */ are used
>2. The query is much slower.
>
>
> Theoretically the query seems pretty simple
>
>1. Use affinity key  to make sure the query runs in parallel and there
>are no shuffles
>2. Filter rows that match category_id using the category_id index
>3. Used customer_id index for the group_by (not sure if this step
>makes sense)
>
> But I cannot get it to work.
>
> Cheers,
> Eugene
>
>
>
>
> On Tue, Sep 18, 2018 at 10:56 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> I can see you try to use _key_PK as index. If your primary key is
>> composite, it won't work properly for you. I recommend creating an explicit
>> (category_id, customer_id) index.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вт, 18 сент. 2018 г. в 17:47, eugene miretsky > >:
>>
>>> Hi Ilya,
>>>
>>> The different query result was my mistake - one of the categoy_ids was
>>> duplicate, so in the query that used join, it counted rows for that
>>> category twice. My apologies.
>>>
>>> However, we are still having an issue with query time, and the index not
>>> being applied to category_id. Would appreciate if you could take a look.
>>>
>>> Cheers,
>>> Eugene
>>>
>>> On Mon, Sep 17, 2018 at 9:15 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 Why don't you diff the results of those two queries, tell us what the
 difference is?

 Regards,
 --
 Ilya Kasnacheev


 пн, 17 сент. 2018 г. в 16:08, eugene miretsky <
 eugene.miret...@gmail.com>:

> Hello,
>
> Just wanted to see if anybody had time to look into this.
>
> Cheers,
> Eugene
>
> On Wed, Sep 12, 2018 at 6:29 PM eugene miretsky <
> eugene.miret...@gmail.com> wrote:
>
>> Thanks!
>>
>> Tried joining with an inlined table instead of IN as per the second
>> suggestion, and it didn't quite work.
>>
>> Query1:
>>
>>- Select COUNT(*) FROM( Select customer_id from GATABLE3  use
>>Index( ) where category_id in (9005, 175930, 175930, 
>> 175940,175945,101450,
>>6453) group by customer_id having SUM(product_views_app) > 2 OR
>>SUM(product_clicks_app) > 1 )
>>- exec time = 17s
>>- *Result: 3105868*
>>- Same exec time if using AFFINITY_KEY index or "_key_PK_hash or
>>customer_id index
>>- Using an index on category_id increases the query time 33s
>>
>> Query2:
>>
>>- Select COUNT(*) FROM( Select customer_id from GATABLE3 ga  use
>>index (PUBLIC."_key_PK") inner join table(category_id int = (9005, 
>> 175930,
>>175930, 175940,175945,101450, 6453)) cats on cats.category_id =
>>ga.category_id   group by customer_id having SUM(product_views_app) > 
>> 2 OR
>>SUM(product_clicks_app) > 1 )
>>- exec time = 38s
>>- *Result: 3113921*
>>- Same exec time if using AFFINITY_KEY index or "_key_PK_hash or
>>customer_id index or category_id index
>>- Using an index on category_id doesnt change the run time
>>
>> Query plans are attached.
>>
>> 3 questions:
>>
>>1. Why is the result differnt for the 2 queries - this is quite
>>concerning.
>>2. Why is the 2nd query taking longer
>>3. Why  category_id index doesn't work in case of query 2.
>>
>>
>> On Wed, Sep 5, 2018 at 8:31 AM Ilya Kasnacheev <
>> 

Re: IgniteUtils enables strictPostRedirect in a static block

[Ilya Kasnacheev]

Hello!

I have a feeling that those lines were left there inadvertently. The
correct insertion of this prop is in GridUpdateNotifier class.

Can you file a ticket about this problem?

Regards,
-- 
Ilya Kasnacheev


вт, 18 сент. 2018 г. в 23:31, xero :

> Hi,
> Does anyone have information about this?
>
> Thanks.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: connecting to Zookeeper cluster using SSL/TLS Connection

[Ilya Kasnacheev]

Hello!

Have you tried specifying

zookeeper.client.secure

If you want to connect to the server secure client port, you need to set
this property to *true* on the client. This will connect to server using
SSL with specified credentials. Note that it requires the Netty client.
zookeeper.ssl.keyStore.location and zookeeper.ssl.keyStore.password

Specifies the file path to a JKS containing the local credentials to be
used for SSL connections, and the password to unlock the file.
zookeeper.ssl.trustStore.location and zookeeper.ssl.trustStore.password

Specifies the file path to a JKS containing the remote credentials to be
used for SSL connections, and the password to unlock the file.
as JVM system properties?

Regards,
-- 
Ilya Kasnacheev


чт, 20 сент. 2018 г. в 6:01, Raghavan, Aravind :

>
>
> Hi All,
>
>
>
> I am trying to use Zookeeper for node discovery with Apache Ignite. I have
> configured Zookeeper to only accept SSL/TLS connections. How do I provide
> Zookeeper keystore detail to Apache Ignite ZookeeperDiscoverySpi? I have
> checked the documentation and source code of ignite-zookeeper.jar and I do
> not see any options to supply these details? Should I be providing these
> details elsewhere in the ignite config?
>
>
>
> Thanks,
>
> Aravind
>
>
>

Re: ***UNCHECKED*** new CacheConfiguration always returns same instance

[Ilya Kasnacheev]

Hello!

hashCode is implemented by MutableConfiguration (from JCache) which only
takes a subset of fields into account.

Why is this a problem for you?

Regards,
-- 
Ilya Kasnacheev


ср, 19 сент. 2018 г. в 10:21, kcheng.mvp :

> here is a simple test code
>
> CacheConfiguration cacheCfg = new CacheConfiguration<>(IG_P);
> cacheCfg.setBackups(1);
>
>
> cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
> logger.info("instance {}", cacheCfg.hashCode());
>
> cacheCfg = new CacheConfiguration<>(IG_T);
> cacheCfg.setBackups(1);
>
>
> cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.PRIMARY_SYNC);
> logger.info("instance {}", cacheCfg.hashCode());
>
>
> the log shows even I create CacheConfiguration two times, but they have the
> same hashcode. is this correct?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Off heap constantly grow on use BinaryObject as field of cached data

[Serg]

Hello Ilya,

I have found that cause of problem the objects which have different size of
data. 
As result I have situation where pageFillFactor increased and offHeapSize is
decreased and this trend never end.

Grafana screen:
https://www.awesomescreenshot.com/image/3619800/ad604d44bae0f241a2618197c2be8af4

I updated reproducer with static data set in csv (
ignite-server-docker/src/main/docker/5.csv ) . 
Reproducer : https://github.com/SergeyMagid/ignite-reproduce-grow-memory

Can you look at this ?
Can this be a bug or we should use different data model?





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSparkSession exception:Ignite instance with provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite instance? [name=null]

[yangjiajun]

 The configuration is
from:https://github.com/apache/ignite/blob/master/examples/config/example-default.xml.

The code runs well with spark local[*].But it throws exception when I run it
with my spark standalone cluster  which has a master node and a worker
node.The exception is in the sql execution stage.The listTables method runs
well.

Here is my code:
   /**
 * Ignite config file.
 */
private static final String CONFIG = "example-ignite.xml";

/**
 * Test cache name.
 */
private static final String CACHE_NAME = "testCache";

/**
 * @throws IOException  */
public static void main(String args[]) throws AnalysisException,
IOException {

//setupServerAndData();
Ignite ignite = Ignition.start(CONFIG);
CacheConfiguration ccfg = new
CacheConfiguration<>(CACHE_NAME).setSqlSchema("PUBLIC");

IgniteCache cache = ignite.getOrCreateCache(ccfg);

//Creating Ignite-specific implementation of Spark session.
IgniteSparkSession igniteSession = IgniteSparkSession.builder()
.appName("Spark Ignite catalog example")
.master("spark://my_computer_ip:7077")
//.master("local[*]")
.config("spark.executor.instances", "2")
.config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.adaptive.enabled",true)
.config("spark.sql.cbo.enabled",true)
.igniteConfig(CONFIG)
.getOrCreate();

//Adjust the logger to exclude the logs of no interest.
Logger.getRootLogger().setLevel(Level.ERROR);
Logger.getLogger("org.apache.ignite").setLevel(Level.INFO);

System.out.println("List of available tables:");

//Showing existing tables.
igniteSession.catalog().listTables().show();

//Selecting data through Spark SQL engine.
Dataset df = igniteSession.sql("select count(*) from my 
test_table");
System.out.println(df.count());
Ignition.stop(false);
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSparkSession exception:Ignite instance with provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite instance? [name=null]

[aealexsandrov]

Hi,

Can you provide the code and configuration? Next one works okay for me:

//start the server
Ignite ignite = Ignition.start("server_config.xml");

//activate the cluster
ignite.cluster().active();

//create the spark session with daemon client node inside
IgniteSparkSession igniteSparkSession = IgniteSparkSession.builder()
.appName("example")
.master("local")
.config("spark.executor.instances", "2")
.igniteConfig("client_config.xml")
.getOrCreate();

System.out.println("spark start--");

igniteSparkSession.sql("SELECT * FROM DdaRecord");

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/