Re: set ExpiryPolicy does not work in C# client code

2016-07-12 Thread Valentin Kulichenko 
WithExpiryPolicy() returns another instance of ICache with overridden
expiry policy and you should use this new instance to do the Put. So the
code should look like this:

cache.WithExpiryPolicy(new ExpiryPolicy(TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(100))).Put("key1", "value1");

In the XML configuration you can specify the expiry policy like this:


















-Val

On Tue, Jul 12, 2016 at 5:50 AM, 右边口袋  wrote:

> Guys,
>
> I test below code and found the key(key1) still exists after 100
> milliseconds(even longer), seems cache.WithExpiryPolicy does not work?
>
> Can someone tell how to set ExpiryPolicy in config.xml? Thanks much!
>
> Code snippet:
> public static void WriteTestCache(IIgnite ignite)
> {
> try
> {
> cache = ignite.GetOrCreateCache string>("cache_test");
> cache.WithExpiryPolicy(new
> ExpiryPolicy(TimeSpan.FromMilliseconds(100),
> TimeSpan.FromMilliseconds(100),
> TimeSpan.FromMilliseconds(100)));
>
> cache.Put("key1", "value1");
> }
> catch (Exception ex)
> {
> Console.WriteLine("IgniteHttpTest:WriteTestCache, ex {0}",
> ex);
> }
> }
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/set-ExpiryPolicy-does-not-work-in-C-client-code-tp9833.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

Re: IGNITE-3055

2016-07-12 Thread Dmitriy Setrakyan 
On Tue, Jul 12, 2016 at 2:40 PM, Denis Magda  wrote:

> Personally, I don’t get why we need to have specific timeout related
> exceptions. My preferences is to stop this practice starting from
> IgniteDataStreamer and create generic TimeoutException that can be used by
> the new features later.
>

Disagree. We should either fix it everywhere, or remain consistent
everywhere. From the API standpoint, it will be very confusing if we don’t.

One way to fix it everywhere, and to preserve backward compatibility, we
could have all specific TimeoutExceptions inherit from the generic
TimeoutException and then deprecate them.


>
> —
> Denis
>
> > On Jul 12, 2016, at 2:35 PM, Dmitriy Setrakyan 
> wrote:
> >
> > If other Timeout Exceptions are specific, shouldn’t this one be specific
> to
> > data streamer as well, for consistency reasons, e.g.
> > IgniteDataStreamerTimeoutExcepiton?
> >
> > On Tue, Jul 12, 2016 at 2:20 PM, Vladislav Pyatkov <
> vpyat...@gridgain.com>
> > wrote:
> >
> >> Dmitriy,
> >>
> >> I discussed that with Denis and we have decided create new Exception
> type
> >> by some reasons:
> >> 1) The exception need to be unchecked (the behavior means serious
> problem
> >> in grid) hence, TimeoutException will implement IgniteException (as
> >> RintimeException).
> >>
> >> 2) Other timeout exception, which follow the logic, are specific (for
> >> example TransactionTimeoutException).
> >>
> >> So TimeoutException will be RuntimeException, will not be specific for
> >> DataStreamer and can be reuse.
> >>
> >> On Tue, Jul 12, 2016 at 1:54 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> >> wrote:
> >>
> >>> Thanks Vlad.
> >>>
> >>> At a high level, the changes look OK. However, I am not sure about
> >>> TimeoutException. Don’t we already have other timeout exceptions in
> >> Ignite?
> >>> Can we reuse them?
> >>>
> >>> D.
> >>>
> >>> On Tue, Jul 12, 2016 at 1:51 PM, Vladislav Pyatkov <
> >> vpyat...@gridgain.com>
> >>> wrote:
> >>>
>  Dmitriy,
> 
>  I have added the description of changes in the JIRA ticket.
> 
>  On Tue, Jul 12, 2016 at 1:28 PM, Dmitriy Setrakyan <
> >>> dsetrak...@apache.org>
>  wrote:
> 
> > Hi Vlad,
> >
> > Can you please list the API changes in the ticket, so others can
> >> review
> > without digging in code?
> >
> > Thanks,
> > D.
> >
> > On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov <
>  vpyat...@gridgain.com>
> > wrote:
> >
> >> Igniters,
> >>
> >> I have implemented timeout in DataStreamer by issue IGNITE-3055
> >>  > (*IgniteDataStreamer
> >> can't be timed out*).
> >>
> >> That lead the change public API:
> >>
> >> 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
> >> IgniteDataStreamer.timeout()) for establish. Default value is -1,
> >>> means
> >> unlimited time.
> >>
> >> 2) Added new TimeoutException (inherited IgniteException). The
>  exception
> >> will thrown when timeout will be reached (may be take place in
> >> IgniteDataStreamer.addData, IgniteDataStreamer.close and
> >> IgniteDataStreamer.flash operations).
> >>
> >> Dmitry, colleagues, please comment or approve.
> >>
> >>
> >
> 
> >>>
> >>
>
>

set ExpiryPolicy does not work in C# client code

2016-07-12 Thread 右边口袋 
Guys,

I test below code and found the key(key1) still exists after 100
milliseconds(even longer), seems cache.WithExpiryPolicy does not work? 

Can someone tell how to set ExpiryPolicy in config.xml? Thanks much!

Code snippet:
public static void WriteTestCache(IIgnite ignite)
{
try
{
cache = ignite.GetOrCreateCache("cache_test");
cache.WithExpiryPolicy(new
ExpiryPolicy(TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(100)));

cache.Put("key1", "value1");
}
catch (Exception ex)
{
Console.WriteLine("IgniteHttpTest:WriteTestCache, ex {0}",
ex);
}
}



--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/set-ExpiryPolicy-does-not-work-in-C-client-code-tp9833.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: kick off a discussion

2016-07-12 Thread Dmitriy Setrakyan 
On Tue, Jul 12, 2016 at 6:44 PM, Konstantin Boudnik  wrote:

> On Mon, Jul 11, 2016 at 08:15AM, Dmitriy Setrakyan wrote:
> > >

> I think you mean data center replication here. It is not an easy feature
> to
> > implement, and so far has been handled by commercial vendors of Ignite,
> > e.g. GridGain.
>
> Apache Geode (incubating) has added the DC replication a couple months
> ago, so
> I don't see why Ignite shouldn't?
>
> Cos


Thanks, Cos! I didn’t know that. I will look into it.

[GitHub] ignite pull request #867: Ignite 1849

2016-07-12 Thread alexpaschenko 
GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/867

Ignite 1849



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-1849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/867.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #867


commit 1c04005c08ae7ca2006d6f2f687a95c2638c73fc
Author: nikolay_tikhonov 
Date:   2016-03-25T12:22:37Z

IGNITE-GG-11048 Added tests.

commit 53a87290fe028e12a81058fc4b2f2f2341395f9a
Author: Tikhonov Nikolay 
Date:   2016-03-25T16:49:31Z

Fixed tests (renamed test methods).

commit ce56625099b5253f8baf27bfbcff4682787a5ee1
Author: Anton Vinogradov 
Date:   2016-03-28T11:13:20Z

Query Suite separation : 1,2,3

commit 08f5981509711e65c0d4c6fc1209068f8958eb06
Author: vozerov-gridgain 
Date:   2016-03-28T09:24:16Z

Support optional IO policy resolver in DataStreamer.

commit 60e74618b310dafc28eb5f935859fed4ec87584c
Author: vozerov-gridgain 
Date:   2016-03-28T11:52:13Z

Merge remote-tracking branch 'upstream/gridgain-7.5.10' into gridgain-7.5.10

commit 9e84e508a394b790ef046cdcf2a015bcf853c064
Author: nikolay_tikhonov 
Date:   2016-03-28T13:59:56Z

Divide test suite.

commit d36a2e51e3aa3105dff73839c84e52a531fbd918
Author: Denis Magda 
Date:   2016-03-24T17:07:48Z

IGNITE-2849: BinaryObjectBuilder doesn't properly check metadata

commit 9e62b667e73396d3affbc4d1e62449d07d08ec8b
Author: Anton Vinogradov 
Date:   2016-03-29T11:56:21Z

 IGNITE-2801 Coordinator floods network with partitions full map exchange 
messages

commit 85a675b7bfbc54745ee9ce95d5786b5bae5be963
Author: iveselovskiy 
Date:   2016-01-18T14:28:42Z

IGNITE-1788: Removed duplicate check of a single invariant from 
IgfsProcessor.  This closes #221.

commit 0e753c38a986cbd46eb24845808ec1d0fc7d1dea
Author: dkarachentsev 
Date:   2016-02-10T09:38:43Z

IGNITE-2575: Added validation of IGFS endpoint port value. This closes #469.

commit 99028b509c736b79faac7fb8104b1bc3cfe30720
Author: vozerov-gridgain 
Date:   2016-03-03T09:56:55Z

IGFS: Minor refactoring.

commit 219238f2793a3f3f9f5705a065c67510c286df1c
Author: vozerov-gridgain 
Date:   2016-03-03T10:25:43Z

IGNITE-2754: IGFS: Created separate processor for listing remove operation.

commit ff5b68ca69050817794ef4b142c955a186e03de9
Author: vozerov-gridgain 
Date:   2016-03-14T07:19:23Z

IGNITE-2781: IGFS: Force "copyOnRead=false" for meta and data caches.

commit 37c4d508c9e40dcba274ae1625d7bf59bfeef144
Author: vozerov-gridgain 
Date:   2016-03-14T08:49:03Z

IGNITE-2810: IGFS: Striped trash directory to reduce contention during 
removals.

commit 54e6991cb1d0b68c4490dede603c9e3ba7cc3b9e
Author: vozerov-gridgain 
Date:   2016-03-14T09:05:39Z

IGNITE-2810: IGFS: Minor correction to IgfsUtils.isRootOrTrashId() method.

commit 8083391be726c2bbc27e018983ca713e4b25e2a2
Author: vozerov-gridgain 
Date:   2016-03-14T10:17:58Z

IGNITE-2814: IGFS: File lock/unlock/reserve operations are no longer 
require put/replace on cache. Thin entry processors are used instead.

commit a7c1f44420ae96f183abc2e17125f0c9aa0775d5
Author: vozerov-gridgain 
Date:   2016-03-14T12:57:28Z

IGNITE-2828: IGFS: Introduced processor for "updateTimes" operation.

commit ba30ddbc599d67f398ffba1263d174f5b58b4b7d
Author: thatcoach 
Date:   2016-03-15T17:46:13Z

IGNITE-2838: IGFS: Opimized format of IgfsListingEntry. Now it contains 
only file ID and boolean flag endicating whether this a directory or file.

commit 4f7e3c1c2e82596a26cec3b3587991ae18078b64
Author: vozerov-gridgain 
Date:   2016-03-16T06:14:49Z

IGNITE-2817: IGFS: Optimized "updateProperties" and several other cache 
operations. Reafactored IgfsMetaManager a bit to simplify work with cache.

commit 3e59321ef0ae1d936d94f8f804db45ceeff55844
Author: vozerov-gridgain 
Date:   2016-03-16T09:31:37Z

IGNITE-2842: IGFS: Optimized create/mkdirs operations with help of entry 
processors.

commit f57f3657b1da33abf28f885cd405780dabfd57e3
Author: vozerov-gridgain 
Date:   2016-03-16T10:22:07Z

IGNITE-2846: IGFS: Reworked IgfsMetaManager.updateInfo() operation to use 
"invoke" instead of "put".

commit 8a93c3bf1687e6f2de1a4391d95366d733a44a7d
Author: vozerov-gridgain 
Date:   2016-03-18T13:38:45Z

[jira] [Created] (IGNITE-3471) Do not send previous value to client node for invoke() when possible

2016-07-12 Thread Alexey Goncharuk (JIRA) 
Alexey Goncharuk created IGNITE-3471:


 Summary: Do not send previous value to client node for invoke() 
when possible
 Key: IGNITE-3471
 URL: https://issues.apache.org/jira/browse/IGNITE-3471
 Project: Ignite
  Issue Type: Task
  Components: cache
Affects Versions: 1.4
Reporter: Alexey Goncharuk


Currently for invoke() or invokeAll() methods we send previous cache value to 
near node and apply EntryProcessor locally to get a return value. This can 
induce a significant overhead when cache value is much larger than entry 
processor result.

For many cases this can be avoided, e.g.
{code}
try (tx = txStart()) {
cache.invoke(key, EP); // No need to send previous value to client in this 
case.
tx.commit();
}
{code}

Note that we need to add additional handling of such a case:
{code}
try (tx = txStart()) {
cache.invoke(key, EP); // No need to send previous value to client in this 
case.

cache.get(key); // This should actually get the current cache value from 
primary node and apply an entry processor locally to get the updated value.
tx.commit();
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3469) Get rid of deprecated APIs and entities

2016-07-12 Thread Alexey Goncharuk (JIRA) 
Alexey Goncharuk created IGNITE-3469:


 Summary: Get rid of deprecated APIs and entities
 Key: IGNITE-3469
 URL: https://issues.apache.org/jira/browse/IGNITE-3469
 Project: Ignite
  Issue Type: Task
Reporter: Alexey Goncharuk
 Fix For: 2.0


There are more than 220 deprecated elements in Ignite code kept for the sake of 
backward compatibility. We need to cleanup the code for Ignite 2.0 release.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ignite - New Release

2016-07-12 Thread Alexandre Boudnik 
Yakov,

Is it possible to include several probably easy-to-fix bugs and
improvements; they are very annoying and they decrease the value of
the product? We're working on Apache Ignite based BI solution
accelerator, and these issues impact us.

I've opened them and I fix one and working on others.

IGNITE-3389 - metadata result set throws NPE when closed - Pull Request #838
IGNITE-3466 - select * causes NoClassDefFoundError with jdbc query tools
IGNITE-3467 - jdbc getTables() returns catalog as null
IGNITE-3468 - Missing Primary Key flag in getColumns()





Take care,
Alexandre "Sasha" Boudnik

call me via Google Voice:
1(405) BUDNIKA
1(405) 283-6452



On Mon, Jul 11, 2016 at 11:46 AM, Yakov Zhdanov  wrote:
> Guys,
>
> We have recently found and fixed several issues in the product along with
> number of smaller fixes and optimizations and I would like to release these
> changes. Issue list follows (complete change list can be obtained via git
> log).
>
> ignite-3428 Fixed message recovery handling on reconnect
> ignite-3418 Avoid unnecessary discovery messages
> IGNITE-2949 - Replaced JCache dependency with Geronimo
> ignite-3372: IgniteDataStreamer: pre-loading starvation if multiple
> streamers preload the same cache
> IGNITE-3305 - Fixed SYNC rebalance mode for dynamically started cache.
> + Lots of changes and fixes for IGFS and Hadoop Accelerator
>
> Does everyone agree this list is worth making new release?
>
> Thanks!
>
> --Yakov

[jira] [Created] (IGNITE-3468) Missing Primary Key flag in getColumns()

2016-07-12 Thread Alexandre Boudnik (JIRA) 
Alexandre Boudnik created IGNITE-3468:
-

 Summary: Missing Primary Key flag in getColumns()
 Key: IGNITE-3468
 URL: https://issues.apache.org/jira/browse/IGNITE-3468
 Project: Ignite
  Issue Type: Improvement
  Components: data structures, odbc
Affects Versions: 1.6
Reporter: Alexandre Boudnik
Assignee: Alexandre Boudnik
 Fix For: 1.7


When implemented it allows BI tools to build more optimal queries



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3467) jdbc getTables() returns catalog as null

2016-07-12 Thread Alexandre Boudnik (JIRA) 
Alexandre Boudnik created IGNITE-3467:
-

 Summary: jdbc getTables() returns catalog as null
 Key: IGNITE-3467
 URL: https://issues.apache.org/jira/browse/IGNITE-3467
 Project: Ignite
  Issue Type: Bug
Affects Versions: 1.6
Reporter: Alexandre Boudnik
Assignee: Alexandre Boudnik
Priority: Critical
 Fix For: 1.7


Then some jdbc query tool uses null values as catalog name. An H2 database 
returns word "DATABASE" in CATALOG column, and then correctly resolves a 
fully-qualified name. I would recommend to do the same for these metadata.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3466) select * causes NoClassDefFoundError with jdbc query tools

2016-07-12 Thread Alexandre Boudnik (JIRA) 
Alexandre Boudnik created IGNITE-3466:
-

 Summary: select * causes NoClassDefFoundError with jdbc query tools
 Key: IGNITE-3466
 URL: https://issues.apache.org/jira/browse/IGNITE-3466
 Project: Ignite
  Issue Type: Bug
Reporter: Alexandre Boudnik
Priority: Critical


it happens when _KEY and _VALUE fields have types other then primitive. I would 
recommend exclude these from "SELECT STAR" list



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kick off a discussion

2016-07-12 Thread Alexandre Boudnik 
Dmitriy, thank you for your time and questions, which helped me to
realize what I forget to mentioned!
See my answers inline; later I'll combine everything together to help
to the next readers :)

I put together some implementation ideas in Apache Ignite JIRA, as
promised: https://issues.apache.org/jira/browse/IGNITE-3457. I see
this facility as another CacheStore implementation, so it wouldn't
interfere with base principals of Ignite platform.


On Mon, Jul 11, 2016 at 1:15 AM, Dmitriy Setrakyan
 wrote:
> My answers are inline…
>
> On Sat, Jul 9, 2016 at 3:04 AM, Dmitriy Setrakyan 
> wrote:
>
>> Thanks Sasha!
>>
>> Resending to the dev list.
>>
>> D.
>>
>> On Fri, Jul 8, 2016 at 2:02 PM, Alexandre Boudnik 
>> wrote:
>>
>>> Apache Ignite a great platform but it lacks of certain capabilities,
>>> which are common in RDMS world, such as:
>>> - Consistent on-line backup for data on entire cluster (or for
>>> specified set of caches)
>>>
>>
> I think you mean data center replication here. It is not an easy feature to
> implement, and so far has been handled by commercial vendors of Ignite,
> e.g. GridGain.
>
Actually not. Right here I meant exactly what I said: full or
incremental backup of all/selected caches in consistent state so it
can be used for the purpose of being able to restore them in case of
data loss or data corruption. One of important use cases is the OLAP
systems (let's say for banking), which has been built on Apache Ignite
platform.

And you right, data center replication can be easily implemented based
on log/snapshot shipment.

>
>> - Hierarchal snapshots for specified set caches
>>>
>>
> What do you mean by hierarchical?
>
In this particular case the notion of hierarchical snapshots is very
similar to the same notion used in SAN appliances or by Virtual Box or
vmware. Using concept of snapshots we can do all this amazing things:
- full and incremental backup
- restore
- rollback to checkpoint
- roll forward
much easier, with minimal memory and I/O overhead.

>
>> - Transaction log
>>>
>>
> Why does Ignite need it for in-memory transactions?
>
At least it is required to provide roll-forward functionality, when
you restores the state of the cache from checkpoint (the cache state
before snapshot has been made) and then reapply transactions one by
one.

>
>> - Restore cluster state as of certain point in time
>>>
>>
> Given that such restorability may introduce lots of memory overhead, does
> it really make sense  for an in-memory cache?
>
Actually, it will not consume any memory. It will use external memory,
such as HDD/SSD space instead. And yes, I think that this
functionality makes complete sense for our users IRL, who will love
it.

>
>> - Rolling forward from snapshot with ability to filter/modify transactions
>>>
>>
> Same as above
>
The same as above: my customers in trenches are begging for that feature.

>
>> - Asynchronous replication based either on log shipment or snapshot
>>> shipment
>>> -- Between clusters
>>>
>>
> This is the same as data center replication, no?
Including but not limited to: log shipment or snapshot shipment also
could be used to implement so called "better-than-lambda-architecture"
for BI and OLAP, when data replicated to a query-able datasource let's
say Oracle as soon as they are produced by OLTP system. We can use
RDBMS API such as Oracle Streams (going to be discontinued - sad) or
Golden Gate to filter changes from logs/snapshots and then apply them.
That approach allows to save a tons of legacy reports and BI
dashboards.

>
>
>> -- Continues data export to let’s say RDMS
>>>
>>
> Don’t we already support it with our write-through feature to a database?
>
When write-through used for non-local caches it may cause the data
corruption in RDBMS: I have opened this issue a few weeks ago:
https://issues.apache.org/jira/browse/IGNITE-3321

>
>> It is also a necessity to reduce cold start time for huge clusters
>>> with strict SLAs.
>>>
>>
> What part are you trying to speed up here? Are you talking about loading
> data from databases?
>
I'm talking about the initial load from Persistent Store when cluster
has been cold-started (like from GridGain's Local Recoverable Store).

>
>>
>>> I'll put some implementation ideas in JIRA later on. I believe that
>>> this list is far from being complete, but I want the community to
>>> discuss these abovementioned use cases.
>>>
>>> --Sasha
>>>
>>
>>

Re: Apache Ignite - New Release

2016-07-12 Thread Konstantin Boudnik 
Yup, looks like having a minor bump is totally warranted.

On Mon, Jul 11, 2016 at 10:11PM, Sergi Vladykin wrote:
> Yakov,
> 
> Good idea.
> 
> Alexey,
> 
> Why not just bump it to 1.7 as we do usually?
> 
> Sergi
> 
> 
> 
> On Mon, Jul 11, 2016 at 9:21 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
> 
> > There were also numerous contributions merged with small API improvements.
> > +1 for making a new release.
> >
> > Since the changes are mostly bugfixes, should we make it a point release?​
> >

Re: kick off a discussion

2016-07-12 Thread Konstantin Boudnik 
On Mon, Jul 11, 2016 at 08:15AM, Dmitriy Setrakyan wrote:
> My answers are inline…
> 
> On Sat, Jul 9, 2016 at 3:04 AM, Dmitriy Setrakyan 
> wrote:
> 
> > Thanks Sasha!
> >
> > Resending to the dev list.
> >
> > D.
> >
> > On Fri, Jul 8, 2016 at 2:02 PM, Alexandre Boudnik 
> > wrote:
> >
> >> Apache Ignite a great platform but it lacks of certain capabilities,
> >> which are common in RDMS world, such as:
> >> - Consistent on-line backup for data on entire cluster (or for
> >> specified set of caches)
> >>
> >
> I think you mean data center replication here. It is not an easy feature to
> implement, and so far has been handled by commercial vendors of Ignite,
> e.g. GridGain.

Apache Geode (incubating) has added the DC replication a couple months ago, so
I don't see why Ignite shouldn't?

Cos

> > - Hierarchal snapshots for specified set caches
> >>
> >
> What do you mean by hierarchical?
> 
> 
> > - Transaction log
> >>
> >
> Why does Ignite need it for in-memory transactions?
> 
> 
> > - Restore cluster state as of certain point in time
> >>
> >
> Given that such restorability may introduce lots of memory overhead, does
> it really make sense  for an in-memory cache?
> 
> 
> > - Rolling forward from snapshot with ability to filter/modify transactions
> >>
> >
> Same as above
> 
> 
> > - Asynchronous replication based either on log shipment or snapshot
> >> shipment
> >> -- Between clusters
> >>
> >
> This is the same as data center replication, no?
> 
> 
> > -- Continues data export to let’s say RDMS
> >>
> >
> Don’t we already support it with our write-through feature to a database?
> 
> 
> > It is also a necessity to reduce cold start time for huge clusters
> >> with strict SLAs.
> >>
> >
> What part are you trying to speed up here? Are you talking about loading
> data from databases?
> 
> 
> >
> >> I'll put some implementation ideas in JIRA later on. I believe that
> >> this list is far from being complete, but I want the community to
> >> discuss these abovementioned use cases.
> >>
> >> --Sasha
> >>
> >
> >

[jira] [Created] (IGNITE-3465) Java crash: problematic frame org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()

2016-07-12 Thread Ksenia Rybakova (JIRA) 
Ksenia Rybakova created IGNITE-3465:
---

 Summary: Java crash: problematic frame 
org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()
 Key: IGNITE-3465
 URL: https://issues.apache.org/jira/browse/IGNITE-3465
 Project: Ignite
  Issue Type: Bug
Reporter: Ksenia Rybakova


During streaming load test some of JVMs crashed while logging info about 
"Failed to wait for partition map exchange" problem
{noformat}
[01:05:23,949][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Failed to wait 
for partition map exchange [topVer=AffinityTopologyVersion [topVer=61, 
minorTopVer=0], node=4f7b195a-62e3-4c32-b496-4a98de50b277]. Dumping
[01:05:23,949][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Ready affinity 
version: AffinityTopologyVersion [topVer=60, minorTopVer=0]
[01:05:24,107][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last exchange 
future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[01:05:24,108][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending 
exchange futures:
[01:05:24,108][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last 10 
exchange futures (total: 48):
[01:05:24,109][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=7291b03d-ce91
[01:05:24,110][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=739ab624-7bce
[01:05:24,111][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=93572383-07e8
[01:05:24,111][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=3c24caa1-d4b2
[01:05:24,112][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=e762044c-8ccc
[01:05:24,118][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=5ffbe144-f843
[01:05:24,119][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=28199123-fa5b
[01:05:24,119][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=14d27ae8-ee26
[01:05:24,120][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=fea5c90e-0467
[01:05:24,120][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, 
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=a5370f23-42bc
[01:05:24,122][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending 
transactions:
[01:05:24,767][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtTxRemote [nearNodeId=fea5c90e-0467-462a-8755-928b539fb278, 
rmtFutId=a52022ed551-938a204f-bf03-451b-8425-be7047673bf6, 
nearXidVer=GridCacheVers
[01:05:24,769][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtTxRemote [nearNodeId=739ab624-7bce-4e4b-81cc-3696f65247bd, 
rmtFutId=ddfe12ed551-15e1d48c-8146-44ac-8d7b-370ea34fa44b, 
nearXidVer=GridCacheVers
[01:05:24,771][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>> 
GridDhtTxRemote [nearNodeId=5ffbe144-f843-4ad1-9ade-688a18f0f872, 
rmtFutId=401f12ed551-62727d5b-0dd3-48a0-bb04-72f7363ec19b, 
nearXidVer=GridCacheVers
[01:05:24,771][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending 
explicit locks:
[01:05:24,771][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending cache 
futures:
[01:05:24,771][WARN 
][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending atomic 
cache futures:

Re: IGNITE-3055

2016-07-12 Thread Denis Magda 
Personally, I don’t get why we need to have specific timeout related 
exceptions. My preferences is to stop this practice starting from 
IgniteDataStreamer and create generic TimeoutException that can be used by the 
new features later.

—
Denis

> On Jul 12, 2016, at 2:35 PM, Dmitriy Setrakyan  wrote:
> 
> If other Timeout Exceptions are specific, shouldn’t this one be specific to
> data streamer as well, for consistency reasons, e.g.
> IgniteDataStreamerTimeoutExcepiton?
> 
> On Tue, Jul 12, 2016 at 2:20 PM, Vladislav Pyatkov 
> wrote:
> 
>> Dmitriy,
>> 
>> I discussed that with Denis and we have decided create new Exception type
>> by some reasons:
>> 1) The exception need to be unchecked (the behavior means serious problem
>> in grid) hence, TimeoutException will implement IgniteException (as
>> RintimeException).
>> 
>> 2) Other timeout exception, which follow the logic, are specific (for
>> example TransactionTimeoutException).
>> 
>> So TimeoutException will be RuntimeException, will not be specific for
>> DataStreamer and can be reuse.
>> 
>> On Tue, Jul 12, 2016 at 1:54 PM, Dmitriy Setrakyan 
>> wrote:
>> 
>>> Thanks Vlad.
>>> 
>>> At a high level, the changes look OK. However, I am not sure about
>>> TimeoutException. Don’t we already have other timeout exceptions in
>> Ignite?
>>> Can we reuse them?
>>> 
>>> D.
>>> 
>>> On Tue, Jul 12, 2016 at 1:51 PM, Vladislav Pyatkov <
>> vpyat...@gridgain.com>
>>> wrote:
>>> 
 Dmitriy,
 
 I have added the description of changes in the JIRA ticket.
 
 On Tue, Jul 12, 2016 at 1:28 PM, Dmitriy Setrakyan <
>>> dsetrak...@apache.org>
 wrote:
 
> Hi Vlad,
> 
> Can you please list the API changes in the ticket, so others can
>> review
> without digging in code?
> 
> Thanks,
> D.
> 
> On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov <
 vpyat...@gridgain.com>
> wrote:
> 
>> Igniters,
>> 
>> I have implemented timeout in DataStreamer by issue IGNITE-3055
>>  (*IgniteDataStreamer
>> can't be timed out*).
>> 
>> That lead the change public API:
>> 
>> 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
>> IgniteDataStreamer.timeout()) for establish. Default value is -1,
>>> means
>> unlimited time.
>> 
>> 2) Added new TimeoutException (inherited IgniteException). The
 exception
>> will thrown when timeout will be reached (may be take place in
>> IgniteDataStreamer.addData, IgniteDataStreamer.close and
>> IgniteDataStreamer.flash operations).
>> 
>> Dmitry, colleagues, please comment or approve.
>> 
>> 
> 
 
>>> 
>>

Re: IGNITE-3055

2016-07-12 Thread Vladislav Pyatkov 
Dmitriy,

I discussed that with Denis and we have decided create new Exception type
by some reasons:
1) The exception need to be unchecked (the behavior means serious problem
in grid) hence, TimeoutException will implement IgniteException (as
RintimeException).

2) Other timeout exception, which follow the logic, are specific (for
example TransactionTimeoutException).

So TimeoutException will be RuntimeException, will not be specific for
DataStreamer and can be reuse.

On Tue, Jul 12, 2016 at 1:54 PM, Dmitriy Setrakyan 
wrote:

> Thanks Vlad.
>
> At a high level, the changes look OK. However, I am not sure about
> TimeoutException. Don’t we already have other timeout exceptions in Ignite?
> Can we reuse them?
>
> D.
>
> On Tue, Jul 12, 2016 at 1:51 PM, Vladislav Pyatkov 
> wrote:
>
> > Dmitriy,
> >
> > I have added the description of changes in the JIRA ticket.
> >
> > On Tue, Jul 12, 2016 at 1:28 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Hi Vlad,
> > >
> > > Can you please list the API changes in the ticket, so others can review
> > > without digging in code?
> > >
> > > Thanks,
> > > D.
> > >
> > > On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov <
> > vpyat...@gridgain.com>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I have implemented timeout in DataStreamer by issue IGNITE-3055
> > > >  > >(*IgniteDataStreamer
> > > > can't be timed out*).
> > > >
> > > > That lead the change public API:
> > > >
> > > > 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
> > > > IgniteDataStreamer.timeout()) for establish. Default value is -1,
> means
> > > > unlimited time.
> > > >
> > > > 2) Added new TimeoutException (inherited IgniteException). The
> > exception
> > > > will thrown when timeout will be reached (may be take place in
> > > > IgniteDataStreamer.addData, IgniteDataStreamer.close and
> > > > IgniteDataStreamer.flash operations).
> > > >
> > > > Dmitry, colleagues, please comment or approve.
> > > >
> > > >
> > >
> >
>

Re: IGNITE-3055

2016-07-12 Thread Dmitriy Setrakyan 
Thanks Vlad.

At a high level, the changes look OK. However, I am not sure about
TimeoutException. Don’t we already have other timeout exceptions in Ignite?
Can we reuse them?

D.

On Tue, Jul 12, 2016 at 1:51 PM, Vladislav Pyatkov 
wrote:

> Dmitriy,
>
> I have added the description of changes in the JIRA ticket.
>
> On Tue, Jul 12, 2016 at 1:28 PM, Dmitriy Setrakyan 
> wrote:
>
> > Hi Vlad,
> >
> > Can you please list the API changes in the ticket, so others can review
> > without digging in code?
> >
> > Thanks,
> > D.
> >
> > On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov <
> vpyat...@gridgain.com>
> > wrote:
> >
> > > Igniters,
> > >
> > > I have implemented timeout in DataStreamer by issue IGNITE-3055
> > >  >(*IgniteDataStreamer
> > > can't be timed out*).
> > >
> > > That lead the change public API:
> > >
> > > 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
> > > IgniteDataStreamer.timeout()) for establish. Default value is -1, means
> > > unlimited time.
> > >
> > > 2) Added new TimeoutException (inherited IgniteException). The
> exception
> > > will thrown when timeout will be reached (may be take place in
> > > IgniteDataStreamer.addData, IgniteDataStreamer.close and
> > > IgniteDataStreamer.flash operations).
> > >
> > > Dmitry, colleagues, please comment or approve.
> > >
> > >
> >
>

Re: IGNITE-3055

2016-07-12 Thread Vladislav Pyatkov 
Dmitriy,

I have added the description of changes in the JIRA ticket.

On Tue, Jul 12, 2016 at 1:28 PM, Dmitriy Setrakyan 
wrote:

> Hi Vlad,
>
> Can you please list the API changes in the ticket, so others can review
> without digging in code?
>
> Thanks,
> D.
>
> On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov 
> wrote:
>
> > Igniters,
> >
> > I have implemented timeout in DataStreamer by issue IGNITE-3055
> > (*IgniteDataStreamer
> > can't be timed out*).
> >
> > That lead the change public API:
> >
> > 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
> > IgniteDataStreamer.timeout()) for establish. Default value is -1, means
> > unlimited time.
> >
> > 2) Added new TimeoutException (inherited IgniteException). The exception
> > will thrown when timeout will be reached (may be take place in
> > IgniteDataStreamer.addData, IgniteDataStreamer.close and
> > IgniteDataStreamer.flash operations).
> >
> > Dmitry, colleagues, please comment or approve.
> >
> >
>

Re: IGNITE-3055

2016-07-12 Thread Dmitriy Setrakyan 
Hi Vlad,

Can you please list the API changes in the ticket, so others can review
without digging in code?

Thanks,
D.

On Mon, Jul 11, 2016 at 1:14 PM, Vladislav Pyatkov 
wrote:

> Igniters,
>
> I have implemented timeout in DataStreamer by issue IGNITE-3055
> (*IgniteDataStreamer
> can't be timed out*).
>
> That lead the change public API:
>
> 1) Added set and get methods (IgniteDataStreamer.timeout(long) and
> IgniteDataStreamer.timeout()) for establish. Default value is -1, means
> unlimited time.
>
> 2) Added new TimeoutException (inherited IgniteException). The exception
> will thrown when timeout will be reached (may be take place in
> IgniteDataStreamer.addData, IgniteDataStreamer.close and
> IgniteDataStreamer.flash operations).
>
> Dmitry, colleagues, please comment or approve.
>
>

Re: Cache unhandled exception

2016-07-12 Thread Denis Magda 
I would add new event like UnhandledExceptioEvent. Presently it will be used 
for your scenario however in the future it can be reused for other cases.

Other thoughts?

—
Denis

> On Jul 11, 2016, at 7:26 PM, AndreyVel  wrote:
> 
> I have question about https://issues.apache.org/jira/browse/IGNITE-2079
> 
> Sametime need send notification for client listener about unhandled 
> exception,  what do you think if to add new class CacheExceptionEvent ?
> Maybe there is more convenient event type/name ?
> 
> Server code
>CacheExceptionEvent evt = new CacheExceptionEvent(node, shortMsg, ex, 
> EVT_UNHANDLED_EXCEPTION);
>ctx.event().record(evt);
> 
> Client code example
> 
> grid(0).events().localListen(new IgnitePredicate() {
>@Override public boolean apply(Event event) {
>. . .  return true;
>}
> }, EventType.EVT_UNHANDLED_EXCEPTION);
>

Re: Cache performance for LOCAL mode + setIndexedTypes

2016-07-12 Thread Dmitriy Setrakyan 
Andrey, the values clearly don’t make sense, which means that the test was
conducted incorrectly. I would rerun it.


On Tue, Jul 12, 2016 at 10:38 AM, Andrey Velichko 
wrote:

> Agree, indexes consume CPU, but as shown in the table,
> performance for index and  CacheMode=LOCAL more then 2x.
>
> CacheMode.CacheMode + TRANSACTIONAL = 45 K op./sec
> CacheMode.CacheMode + TRANSACTIONAL + setIndexedTypes = 107 K op./sec
>
> CacheMode.CacheMode + ATOMIC= 340 K op./sec
> CacheMode.CacheMode + ATOMIC+ setIndexedTypes = 581 K op./sec
>
>
> 12.07.2016 0:15, Sergi Vladykin пишет:
>
> I guess updating indexes is not free.
>>
>> Sergi
>>
>> On Mon, Jul 11, 2016 at 11:48 PM, AndreyVel  wrote:
>>
>> Hello, I have created simple test for testing performance put operations
>>> on single computer,
>>> can anybody explain why performance for LOCAL mode different for Indexed
>>> and non Indexed mode?
>>>
>>>
>>>   Cache performance put(Integer, String)/sec: one client and 2
>>>   remote servers.
>>>
>>> CacheMode   TRANSACTIONAL   TRANSACTIONAL
>>> setIndexedTypes(Integer, String)ATOMIC  ATOMIC
>>> setIndexedTypes(Integer, String)
>>> LOCAL   45154   107306  340627  581250
>>> REPLICATED  3270286572725694
>>> PARTITIONED 6773540385586714
>>>
>>>
>>> VM options -Xms1g -Xmx1g
>>> Topology snapshot [ver=5, servers=2, clients=1, CPUs=8, heap=3.0GB]
>>>
>>>
>>> public class CachePerformance {
>>>
>>>  private static int CACHE_SIZE =100 * 1000;
>>>  private static StringCACHE_NAME ="cachePerf";
>>>
>>>  public static void main(String[] args) {
>>>  try {
>>>  Ignition.setClientMode(true);
>>>  IgniteConfiguration igniteCfg =new IgniteConfiguration();
>>>  igniteCfg.setPeerClassLoadingEnabled(true);
>>>
>>>  Ignite ignite = Ignition.start(igniteCfg);
>>>  CachePerformance.execute(ignite);
>>>  }catch (Throwable ex) {
>>>  ex.printStackTrace();
>>>  }
>>>  }
>>>
>>> public static void execute(Ignite ignite)throws Exception {
>>>
>>>  Random rand =new Random();
>>>
>>>  for (CacheMode cacheMode : CacheMode.values()) {
>>>
>>>  for (CacheAtomicityMode atomMode : CacheAtomicityMode.values())
>>> {
>>>  for (int indexMode =0; indexMode <=1; indexMode++) {
>>>
>>>  CacheConfiguration cacheCfg =new
>>> CacheConfiguration<>(CACHE_NAME);
>>>  if (indexMode ==1)
>>>  cacheCfg.setIndexedTypes(Integer.class,
>>> String.class);
>>>
>>>  cacheCfg.setAtomicityMode(atomMode);
>>>  cacheCfg.setCacheMode(cacheMode);
>>>
>>>  ignite.destroyCache(CACHE_NAME);
>>>  IgniteCache cache =
>>> ignite.getOrCreateCache(cacheCfg);
>>>  String cacheDesc ="cacheMode=" + cacheMode +",
>>> atomMode="
>>> + atomMode +", indexMode=" + indexMode;
>>>
>>>  long time = System.nanoTime();
>>>  for (int ind =0; ind >>  Integer key = rand.nextInt();
>>>  cache.put(key,"=" + ind);
>>>  }
>>>
>>>  long deltaMs = (System.nanoTime() - time) /100;
>>>  long perfSec = (deltaMs ==0) ?0 :1000 *CACHE_SIZE /
>>> deltaMs;
>>>  System.out.println(cacheDesc +", Perf/Sec = " +
>>> perfSec);
>>>  }
>>>  }
>>>  }
>>>
>>>
>

Re: Cache performance for LOCAL mode + setIndexedTypes

2016-07-12 Thread Andrey Velichko 

Agree, indexes consume CPU, but as shown in the table,
performance for index and  CacheMode=LOCAL more then 2x.

CacheMode.CacheMode + TRANSACTIONAL = 45 K op./sec
CacheMode.CacheMode + TRANSACTIONAL + setIndexedTypes = 107 K op./sec

CacheMode.CacheMode + ATOMIC= 340 K op./sec
CacheMode.CacheMode + ATOMIC+ setIndexedTypes = 581 K op./sec


12.07.2016 0:15, Sergi Vladykin пишет:

I guess updating indexes is not free.

Sergi

On Mon, Jul 11, 2016 at 11:48 PM, AndreyVel  wrote:


Hello, I have created simple test for testing performance put operations
on single computer,
can anybody explain why performance for LOCAL mode different for Indexed
and non Indexed mode?


  Cache performance put(Integer, String)/sec: one client and 2
  remote servers.

CacheMode   TRANSACTIONAL   TRANSACTIONAL
setIndexedTypes(Integer, String)ATOMIC  ATOMIC
setIndexedTypes(Integer, String)
LOCAL   45154   107306  340627  581250
REPLICATED  3270286572725694
PARTITIONED 6773540385586714


VM options -Xms1g -Xmx1g
Topology snapshot [ver=5, servers=2, clients=1, CPUs=8, heap=3.0GB]


public class CachePerformance {

 private static int CACHE_SIZE =100 * 1000;
 private static StringCACHE_NAME ="cachePerf";

 public static void main(String[] args) {
 try {
 Ignition.setClientMode(true);
 IgniteConfiguration igniteCfg =new IgniteConfiguration();
 igniteCfg.setPeerClassLoadingEnabled(true);

 Ignite ignite = Ignition.start(igniteCfg);
 CachePerformance.execute(ignite);
 }catch (Throwable ex) {
 ex.printStackTrace();
 }
 }

public static void execute(Ignite ignite)throws Exception {

 Random rand =new Random();

 for (CacheMode cacheMode : CacheMode.values()) {

 for (CacheAtomicityMode atomMode : CacheAtomicityMode.values()) {
 for (int indexMode =0; indexMode <=1; indexMode++) {

 CacheConfiguration cacheCfg =new
CacheConfiguration<>(CACHE_NAME);
 if (indexMode ==1)
 cacheCfg.setIndexedTypes(Integer.class, String.class);

 cacheCfg.setAtomicityMode(atomMode);
 cacheCfg.setCacheMode(cacheMode);

 ignite.destroyCache(CACHE_NAME);
 IgniteCache cache =
ignite.getOrCreateCache(cacheCfg);
 String cacheDesc ="cacheMode=" + cacheMode +", atomMode="
+ atomMode +", indexMode=" + indexMode;

 long time = System.nanoTime();
 for (int ind =0; ind 
    
    
23 matches