Multi-cache transactions & persistent store

[Igor Rudyak]

Hi guys,

Playing with Ignite multi-cache transactions(transactions involving
multiple caches) we run into the lack of proper/easy mechanism to support
such transactions on persistent store side (while implementing particular
*CacheStore*).

The problem is there is no easy way for *CacheStore *implementation to get
information about caches involved in the transaction.

Here are more details:

When committing multi-cache transaction, Ignite produces multiple
*sessionEnd(...)* calls (in *CacheStore *implementation) - one for each
cache. Thus to support multi-cache transaction in persistent store, we need
to accumulate all the changes made to multiple caches (by
*write/writeAll/delete/deleteAll* operations) and apply them as one
transaction/batch to a persistent store in the final(last) *sessionEnd *call.
To do this we need to know exact number of caches participating in the
transaction. Having this number we can skip all *sessionEnd *calls except
the last - which we will use to commit/persist all the changes we
previously accumulated.

As for now, the only way to get the number of caches participating in a
transaction is to register *CacheStoreSessionListener *and count number of
caches in its "*onSessionStart*" method. Such approach doesn't look very
elegant cause we always need to keep in mind that specifying "
*cacheStoreFactory*" in cache configuration is not enough. In addition to
this we also need to specify appropriate "
*cacheStoreSessionListenerFactories*" property - otherwise it will not work
for multi-cache transactions.

Here is an example chunk from cache configuration:

* *
* *
* *
* *
* *
* *
* *
* *
* *
* *
* *

We see that, instead of specifying only one property "*cacheStoreFactory*"
for a cache we should always specify two properties in case we need
multi-cache transactions support in persistent store. Conceptually it
doesn't look good, cause users thinking about *CacheStore *implementation
like something self-contained - once it specified in cache configuration
everything should be smooth. But in case of multi-cache transactions
support we always need to remember about registering some strange
"listener".

In Ignite we already have such concept as "
*org.apache.ignite.transactions.Transaction*" which defines our
transaction. It looks logical if it will also provide information about
transaction boundaries (caches involved into transaction). Such approach
can simplify persistent store implementation for multi-cache transactions.

Any thoughts?


Igor Rudyak

Re: Batch support in Cassandra store

[Igor Rudyak]

Hi Valentin,

Sounds reasonable. I'll create a ticket to add Cassandra logged batches and
will try to prepare some load tests to investigate if unlogged batches can
provide better performance. Will also add ticket for RAMP as a long term
enhancement.

Igor Rudyak

On Fri, Jul 29, 2016 at 5:45 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Igor,
>
> 1) Yes, I'm talking about splitting the entry set into per-partition (or
> per-node) batches. Having entries that are stores on different nodes in the
> same batch doesn't make much sense, of course.
>
> 2) RAMP looks interesting, but it seems to be a pretty complicated task.
> How about adding the support for built-in logged batches (this should be
> fairly easy to implement) and then improve the atomicity as a second phase?
>
> -Val
>
> On Fri, Jul 29, 2016 at 5:19 PM, Igor Rudyak  wrote:
>
>> Hi Valentin,
>>
>> 1) According unlogged batches I think it doesn't make sense to support
>> them, cause:
>> - They are deprecated starting from Cassandra 3.0 (which we are currently
>> using in Cassandra module)
>> - According to Cassandra documentation (
>> http://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html)
>> "Batches are often mistakenly used in an attempt to optimize performance".
>> Cassandra guys saying that no batches (
>> https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-keyword-40f00e35e23e#.rxkmfe209)
>> is the fastest way to load data. I checked it with the batches having
>> records with different partition keys and it's definitely true. For small
>> batch of records having all the same partition key (affinity in Ignite)
>> they could provide better performance, but I didn't investigated this case
>> deeply (what is the optimal size of a batch, how significantly is the
>> performance benefits and etc.) Can try to do some load tests to have better
>> understanding of this.
>>
>> 2) Regarding logged batches I think that it makes sense to support them
>> in Cassandra module for transactional caches. The bad thing is that they
>> don't provide isolation, the good thing is they guaranty that all your
>> changes will be eventually committed and visible to clients. Thus it's
>> still better than nothing... However there is a better approach for this.
>> We can implement transactional protocol on top of Cassandra, which will
>> give us atomic read isolation - you'll either see all the changes made by
>> transaction or none of them. For example we can implement RAMP transactions(
>> http://www.bailis.org/papers/ramp-sigmod2014.pdf) cause it provides
>> rather low overhead.
>>
>> Igor Rudyak
>>
>> On Thu, Jul 28, 2016 at 11:00 PM, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>>> Hi Igor,
>>>
>>> I'm not a big Cassandra expert, but here are my thoughts.
>>>
>>> 1. Sending updates in a batch is always better than sending them one by
>>> one. For example, if you do putAll in Ignite with 100 entries, and these
>>> entries are split across 5 nodes, the client will send 5 requests instead
>>> of 100. This provides significant performance improvement. Is there a way
>>> to use similar approach in Cassandra?
>>> 2. As for logged batches, I can easily believe that this is a rarely
>>> used feature, but since it exists in Cassandra, I can't find a single
>>> reason why not to support it in our store as an option. Users that come
>>> across those rare cases, will only say thank you to us :)
>>>
>>> What do you think?
>>>
>>> -Val
>>>
>>> On Thu, Jul 28, 2016 at 10:41 PM, Igor Rudyak  wrote:
>>>
 There are actually some cases when atomic read isolation in Cassandra
 could
 be important. Lets assume batch was persisted in Cassandra, but not
 finalized yet - read operation from Cassandra returns us only partially
 committed data of the batch. In the such situation we have problems
 when:

 1) Some of the batch records already expired from Ignite cache and we
 reading them from persistent store (Cassandra in our case).

 2) All Ignite nodes storing the batch records (or subset records) died
 (or
 for example became unavailable for 10sec because of network problem).
 While
 reading such records from Ignite cache we will be redirected to
 persistent
 store.

 3) Network separation occurred such a way that we now have two Ignite
 cluster, but all the replicas of the batch data are located only in one
 of
 these clusters. Again while reading such records from Ignite cache on
 the
 second cluster we will be redirected to persistent store.

 In all mentioned cases, if Cassandra batch isn't finalized yet - we will
 read partially committed transaction data.


 On Thu, Jul 28, 2016 at 6:52 AM, Luiz Felipe Trevisan <
 luizfelipe.trevi...@gmail.com> wrote:

 > I totally agree with you regarding the guarantees we have with logged
 > batches and 

Re: Batch support in Cassandra store

[Valentin Kulichenko]

Hi Igor,

1) Yes, I'm talking about splitting the entry set into per-partition (or
per-node) batches. Having entries that are stores on different nodes in the
same batch doesn't make much sense, of course.

2) RAMP looks interesting, but it seems to be a pretty complicated task.
How about adding the support for built-in logged batches (this should be
fairly easy to implement) and then improve the atomicity as a second phase?

-Val

On Fri, Jul 29, 2016 at 5:19 PM, Igor Rudyak  wrote:

> Hi Valentin,
>
> 1) According unlogged batches I think it doesn't make sense to support
> them, cause:
> - They are deprecated starting from Cassandra 3.0 (which we are currently
> using in Cassandra module)
> - According to Cassandra documentation (
> http://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html) "Batches
> are often mistakenly used in an attempt to optimize performance". Cassandra
> guys saying that no batches (
> https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-keyword-40f00e35e23e#.rxkmfe209)
> is the fastest way to load data. I checked it with the batches having
> records with different partition keys and it's definitely true. For small
> batch of records having all the same partition key (affinity in Ignite)
> they could provide better performance, but I didn't investigated this case
> deeply (what is the optimal size of a batch, how significantly is the
> performance benefits and etc.) Can try to do some load tests to have better
> understanding of this.
>
> 2) Regarding logged batches I think that it makes sense to support them in
> Cassandra module for transactional caches. The bad thing is that they don't
> provide isolation, the good thing is they guaranty that all your changes
> will be eventually committed and visible to clients. Thus it's still better
> than nothing... However there is a better approach for this. We can
> implement transactional protocol on top of Cassandra, which will give us
> atomic read isolation - you'll either see all the changes made by
> transaction or none of them. For example we can implement RAMP transactions(
> http://www.bailis.org/papers/ramp-sigmod2014.pdf) cause it provides
> rather low overhead.
>
> Igor Rudyak
>
> On Thu, Jul 28, 2016 at 11:00 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Hi Igor,
>>
>> I'm not a big Cassandra expert, but here are my thoughts.
>>
>> 1. Sending updates in a batch is always better than sending them one by
>> one. For example, if you do putAll in Ignite with 100 entries, and these
>> entries are split across 5 nodes, the client will send 5 requests instead
>> of 100. This provides significant performance improvement. Is there a way
>> to use similar approach in Cassandra?
>> 2. As for logged batches, I can easily believe that this is a rarely used
>> feature, but since it exists in Cassandra, I can't find a single reason why
>> not to support it in our store as an option. Users that come across those
>> rare cases, will only say thank you to us :)
>>
>> What do you think?
>>
>> -Val
>>
>> On Thu, Jul 28, 2016 at 10:41 PM, Igor Rudyak  wrote:
>>
>>> There are actually some cases when atomic read isolation in Cassandra
>>> could
>>> be important. Lets assume batch was persisted in Cassandra, but not
>>> finalized yet - read operation from Cassandra returns us only partially
>>> committed data of the batch. In the such situation we have problems when:
>>>
>>> 1) Some of the batch records already expired from Ignite cache and we
>>> reading them from persistent store (Cassandra in our case).
>>>
>>> 2) All Ignite nodes storing the batch records (or subset records) died
>>> (or
>>> for example became unavailable for 10sec because of network problem).
>>> While
>>> reading such records from Ignite cache we will be redirected to
>>> persistent
>>> store.
>>>
>>> 3) Network separation occurred such a way that we now have two Ignite
>>> cluster, but all the replicas of the batch data are located only in one
>>> of
>>> these clusters. Again while reading such records from Ignite cache on the
>>> second cluster we will be redirected to persistent store.
>>>
>>> In all mentioned cases, if Cassandra batch isn't finalized yet - we will
>>> read partially committed transaction data.
>>>
>>>
>>> On Thu, Jul 28, 2016 at 6:52 AM, Luiz Felipe Trevisan <
>>> luizfelipe.trevi...@gmail.com> wrote:
>>>
>>> > I totally agree with you regarding the guarantees we have with logged
>>> > batches and I'm also pretty much aware of the performance penalty
>>> involved
>>> > using this solution.
>>> >
>>> > But since all read operations are executed via ignite it means that
>>> > isolation in the Cassandra level is not really important. I think the
>>> only
>>> > guarantee really needed is that we don't end up with a partial insert
>>> in
>>> > Cassandra in case we have a failure in ignite and we loose the node
>>> that
>>> > was responsible for this write operation.
>>> >

Re: Batch support in Cassandra store

[Igor Rudyak]

Hi Valentin,

1) According unlogged batches I think it doesn't make sense to support
them, cause:
- They are deprecated starting from Cassandra 3.0 (which we are currently
using in Cassandra module)
- According to Cassandra documentation (
http://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html) "Batches
are often mistakenly used in an attempt to optimize performance". Cassandra
guys saying that no batches (
https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-keyword-40f00e35e23e#.rxkmfe209)
is the fastest way to load data. I checked it with the batches having
records with different partition keys and it's definitely true. For small
batch of records having all the same partition key (affinity in Ignite)
they could provide better performance, but I didn't investigated this case
deeply (what is the optimal size of a batch, how significantly is the
performance benefits and etc.) Can try to do some load tests to have better
understanding of this.

2) Regarding logged batches I think that it makes sense to support them in
Cassandra module for transactional caches. The bad thing is that they don't
provide isolation, the good thing is they guaranty that all your changes
will be eventually committed and visible to clients. Thus it's still better
than nothing... However there is a better approach for this. We can
implement transactional protocol on top of Cassandra, which will give us
atomic read isolation - you'll either see all the changes made by
transaction or none of them. For example we can implement RAMP transactions(
http://www.bailis.org/papers/ramp-sigmod2014.pdf) cause it provides rather
low overhead.

Igor Rudyak

On Thu, Jul 28, 2016 at 11:00 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Igor,
>
> I'm not a big Cassandra expert, but here are my thoughts.
>
> 1. Sending updates in a batch is always better than sending them one by
> one. For example, if you do putAll in Ignite with 100 entries, and these
> entries are split across 5 nodes, the client will send 5 requests instead
> of 100. This provides significant performance improvement. Is there a way
> to use similar approach in Cassandra?
> 2. As for logged batches, I can easily believe that this is a rarely used
> feature, but since it exists in Cassandra, I can't find a single reason why
> not to support it in our store as an option. Users that come across those
> rare cases, will only say thank you to us :)
>
> What do you think?
>
> -Val
>
> On Thu, Jul 28, 2016 at 10:41 PM, Igor Rudyak  wrote:
>
>> There are actually some cases when atomic read isolation in Cassandra
>> could
>> be important. Lets assume batch was persisted in Cassandra, but not
>> finalized yet - read operation from Cassandra returns us only partially
>> committed data of the batch. In the such situation we have problems when:
>>
>> 1) Some of the batch records already expired from Ignite cache and we
>> reading them from persistent store (Cassandra in our case).
>>
>> 2) All Ignite nodes storing the batch records (or subset records) died (or
>> for example became unavailable for 10sec because of network problem).
>> While
>> reading such records from Ignite cache we will be redirected to persistent
>> store.
>>
>> 3) Network separation occurred such a way that we now have two Ignite
>> cluster, but all the replicas of the batch data are located only in one of
>> these clusters. Again while reading such records from Ignite cache on the
>> second cluster we will be redirected to persistent store.
>>
>> In all mentioned cases, if Cassandra batch isn't finalized yet - we will
>> read partially committed transaction data.
>>
>>
>> On Thu, Jul 28, 2016 at 6:52 AM, Luiz Felipe Trevisan <
>> luizfelipe.trevi...@gmail.com> wrote:
>>
>> > I totally agree with you regarding the guarantees we have with logged
>> > batches and I'm also pretty much aware of the performance penalty
>> involved
>> > using this solution.
>> >
>> > But since all read operations are executed via ignite it means that
>> > isolation in the Cassandra level is not really important. I think the
>> only
>> > guarantee really needed is that we don't end up with a partial insert in
>> > Cassandra in case we have a failure in ignite and we loose the node that
>> > was responsible for this write operation.
>> >
>> > My other assumption is that the write operation needs to finish before
>> an
>> > eviction happens for this entry and we loose the data in cache (since
>> batch
>> > doesn't guarantee isolation). However if we cannot achieve this I don't
>> see
>> > why use ignite as a cache store.
>> >
>> > Luiz
>> >
>> > --
>> > Luiz Felipe Trevisan
>> >
>> > On Wed, Jul 27, 2016 at 4:55 PM, Igor Rudyak  wrote:
>> >
>> >> Hi Luiz,
>> >>
>> >> Logged batches is not the solution to achieve atomic view of your
>> Ignite
>> >> transaction changes in Cassandra.
>> >>
>> >> The problem with logged batches(aka atomic) is they guarantees that if

Re: JetBrains Annotaions usage.

[Anton Vinogradov]

Alexey, thanks for tips.
Seems maven dependency's "optional" tag can handle this.
See pullrequest  for details.
P.s. I'm not pretty sure this change will not brake something, need to
recheck it twice.

On Thu, Jul 28, 2016 at 5:15 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> According to the JLS [1], adding or removing annotations has no effect on
> the correct linkage of the binary representations of programs in the Java
> programming language. Even if these annotations were RUNTIME, a user could
> successfully use Ignite unless he explicitly uses those classes in runtime.
> See also [2]
>
> [1]
> https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html#jls-13.5.7
> [2] http://bugs.java.com/view_bug.do?bug_id=6322301
>
> 2016-07-28 16:29 GMT+03:00 Anton Vinogradov :
>
> > Dmitriy,
> >
> > Annotations have @Retention(RetentionPolicy.CLASS)
> > which means
> >  * Annotations are to be recorded in the class file by the compiler
> >  * but need not be retained by the VM at run time.  This is the
> default
> >  * behavior.
> >
> > So, from what I understand, everyone using ignite API with such
> annotations
> > still require them, correct?
> >
> > On Thu, Jul 28, 2016 at 4:16 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Anton, compile-time annotations should mean that Ignite should not
> > require
> > > these libraries at runtime. Did you try it already and run into issues?
> > >
> > > On Thu, Jul 28, 2016 at 4:34 AM, Anton Vinogradov <
> > > avinogra...@gridgain.com>
> > > wrote:
> > >
> > > > Possible, what's the solution in this case?
> > > >
> > > > On Thu, Jul 28, 2016 at 11:17 AM, Sergi Vladykin <
> > > sergi.vlady...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I don't think that JB annotations is a runtime dependency, we
> should
> > > need
> > > > > them only at compile time, no?
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2016-07-28 11:09 GMT+03:00 Anton Vinogradov :
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > As you may know we had only 2 dependencies at ignite-core: jcache
> > and
> > > > > > ignite-shmen.
> > > > > >
> > > > > > IGNITE-3323 Get rid of copypasted JB annotations, use dependency
> > > > instead.
> > > > > >  brings one
> > more
> > > > > > dependency to org.jetbrains.annotations.
> > > > > > This solves problems for people using JB annotations at their
> > > projects.
> > > > > but
> > > > > > makes ignite-core less dependency-lightweight.
> > > > > >
> > > > > > JB annotaions are used to highlight null problems in IDE and
> > provide
> > > > some
> > > > > > understanding to users on whether null is allowed/possible in
> > > concrete
> > > > > > place. I doubt other products/frameworks use it much.
> > > > > >
> > > > > > But, do we really need them? As for me, we can replace all these
> > > > > > annotations by asserts.
> > > > > >
> > > > > > I propose to discontinue usage of jb annotation and replace them
> by
> > > > > > asserts.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > >
> > > >
> > >
> >
>

[GitHub] ignite pull request #909: JB Annotatins required only at Ignite sources comp...

[avinogradovgg]

GitHub user avinogradovgg opened a pull request:

https://github.com/apache/ignite/pull/909

JB Annotatins required only at Ignite sources compilation time.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/avinogradovgg/ignite ignite-jb

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #909


commit fecffe63d059e157d05d685fbbd53b7dedc78eee
Author: Anton Vinogradov 
Date:   2016-07-29T21:07:12Z

JB Annotatins required only at Ignite sources compilation time.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apache Ignite - New Release

[Dmitriy Setrakyan]

On Fri, Jul 29, 2016 at 9:17 AM, Pavel Tupitsyn 
wrote:

> Guys, I can handle upcoming 1.7 release and send the vote next week, if
> there are no objections.
>

Thanks Pavel! I don't think anyone will have any objections.

Given that you will be a manager for this release, can you please take
charge of release notes and make sure that all the required functionality
is merged?


>
> Pavel.
>
> On Thu, Jul 28, 2016 at 12:36 AM, Dmitriy Setrakyan  >
> wrote:
>
> > On Wed, Jul 27, 2016 at 4:00 AM, Semyon Boikov 
> > wrote:
> >
> > > Dmitry, all Ignite features are always thoroughly tested =)
> > >
> >
> > True, completely forgot about it :)
> >
> >
> > > I created branch ignite-1.7.0.
> > >
> > > On Wed, Jul 27, 2016 at 9:32 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > On Wed, Jul 27, 2016 at 2:29 AM, Semyon Boikov  >
> > > > wrote:
> > > >
> > > > > Sure Dmitry, we will run tests with large data set.
> > > > >
> > > > > Regarding 1.7. release: If there are no objections I'm going to cut
> > off
> > > > 1.7
> > > > > branch and start prepare it for release.
> > > > >
> > > >
> > > > In my opinion this is a great addition to Ignite and is definitely
> > worth
> > > a
> > > > release, again, assuming that we thoroughly test it.
> > > >
> > > >
> > > > >
> > > > >
> > > > > On Wed, Jul 27, 2016 at 9:15 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > On Wed, Jul 27, 2016 at 2:09 AM, Semyon Boikov <
> > sboi...@gridgain.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Regarding distributed join testsing: we added tests verifying
> > > correct
> > > > > > join
> > > > > > > behavior and correct execution plan generation for various SQL
> > > > queries,
> > > > > > > tests for joins for various cache types (different number of
> > > backups,
> > > > > > > partitioned/replicated), there are tests verifying correct
> > > > distributed
> > > > > > > joins results on changing topology with nodes restarts. Also we
> > > added
> > > > > > > benchmarks which will be used to verify that there are no
> > > performance
> > > > > > > degradation in this functionality between releases. These
> > > benchmarks
> > > > > were
> > > > > > > executed on real clusters, and in next few days we are going to
> > run
> > > > > more
> > > > > > > load tests.
> > > > > > >
> > > > > >
> > > > > > Thanks Semyon, sounds great! I would also test it on larger data
> > sets
> > > > to
> > > > > > see how a join query will take, say, on 10GB of data. Is it
> > possible?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 22, 2016 at 5:39 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Jul 22, 2016 at 7:10 AM, Semyon Boikov <
> > > > sboi...@gridgain.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Today I merged into master 'distributed join'
> implementation
> > -
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1232 (thanks
> to
> > > > > Sergi,
> > > > > > he
> > > > > > > > > implemented this feature). I think this together with
> recent
> > > > > bugfixes
> > > > > > > > worth
> > > > > > > > > 1.7 release. Do you think we can cut off 1.7 release branch
> > > from
> > > > > > > master?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Great news. Can you describe that amount of testing we did
> for
> > > this
> > > > > > > > feature?
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 21, 2016 at 7:47 PM, Alexandre Boudnik <
> > > > > > > > > alexander.boud...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Sorry, I missed the typo:
> > > > > > > > > > To support the point, I would add that PosgreSQL has
> > > > demonstrated
> > > > > > the
> > > > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > > > - they do not appear in select * list unless they are
> > > specified
> > > > > > > > > explicitly
> > > > > > > > > > Take care,
> > > > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > > > >
> > > > > > > > > > call me via Google Voice:
> > > > > > > > > > 1(405) BUDNIKA
> > > > > > > > > > 1(405) 283-6452
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 21, 2016 at 12:46 PM, Alexandre Boudnik
> > > > > > > > > >  wrote:
> > > > > > > > > > > To support the point, I would add that PosgreSQL has
> > > > > demonstrated
> > > > > > > the
> > > > > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > > > > - they appear in select * 

[jira] [Created] (IGNITE-3608) QuerySqlFunction methods with Object type var args do not work

[Edward Kaganovich (JIRA)]

Edward Kaganovich created IGNITE-3608:
-

 Summary: QuerySqlFunction methods with Object type var args do not 
work
 Key: IGNITE-3608
 URL: https://issues.apache.org/jira/browse/IGNITE-3608
 Project: Ignite
  Issue Type: Bug
  Components: cache, SQL
Affects Versions: 1.6, 1.5.0.final
Reporter: Edward Kaganovich


We often use interactive sql consoles to query caches.  Several field values as 
well as _key in caches are stored as objects.  To support queries by these 
fields we have registered a generic UDF that should let us instantiate certain 
types:

{code}   @QuerySqlFunction Object t(String clz, Object... ctorArgs) throws
Exception {
Class c = Class.forName(keyClz); 
Class[] argTypes = new Class[ctorArgs.length]; 
for (int i=0; i < ctorArgs.length; i++) { 
argTypes[i] = ctorArgs[i].getClass(); 
} 
Constructor ctor = c.getConstructor(argTypes); 
return ctor.newInstance(ctorArgs); 
}
{code}
Unfortunately Ignite fails to find and execute this function for the following 
SQL:
{color:blue}select t('java.lang.String', 'key20')){color}{color:red} <- 
Fails{color}

However, there is no problems with this function:
{code}   @QuerySqlFunction 
public static Object t1(String clz, String... ctorArgs) throws Exception { 
return t(clz, ctorArgs);
}
{code}
{color:blue}select t1('java.lang.String', 'key20')){color}{color:green} <- 
Works as expected{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ignite - New Release

[Pavel Tupitsyn]

Guys, I can handle upcoming 1.7 release and send the vote next week, if
there are no objections.

Pavel.

On Thu, Jul 28, 2016 at 12:36 AM, Dmitriy Setrakyan 
wrote:

> On Wed, Jul 27, 2016 at 4:00 AM, Semyon Boikov 
> wrote:
>
> > Dmitry, all Ignite features are always thoroughly tested =)
> >
>
> True, completely forgot about it :)
>
>
> > I created branch ignite-1.7.0.
> >
> > On Wed, Jul 27, 2016 at 9:32 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Wed, Jul 27, 2016 at 2:29 AM, Semyon Boikov 
> > > wrote:
> > >
> > > > Sure Dmitry, we will run tests with large data set.
> > > >
> > > > Regarding 1.7. release: If there are no objections I'm going to cut
> off
> > > 1.7
> > > > branch and start prepare it for release.
> > > >
> > >
> > > In my opinion this is a great addition to Ignite and is definitely
> worth
> > a
> > > release, again, assuming that we thoroughly test it.
> > >
> > >
> > > >
> > > >
> > > > On Wed, Jul 27, 2016 at 9:15 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > On Wed, Jul 27, 2016 at 2:09 AM, Semyon Boikov <
> sboi...@gridgain.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Regarding distributed join testsing: we added tests verifying
> > correct
> > > > > join
> > > > > > behavior and correct execution plan generation for various SQL
> > > queries,
> > > > > > tests for joins for various cache types (different number of
> > backups,
> > > > > > partitioned/replicated), there are tests verifying correct
> > > distributed
> > > > > > joins results on changing topology with nodes restarts. Also we
> > added
> > > > > > benchmarks which will be used to verify that there are no
> > performance
> > > > > > degradation in this functionality between releases. These
> > benchmarks
> > > > were
> > > > > > executed on real clusters, and in next few days we are going to
> run
> > > > more
> > > > > > load tests.
> > > > > >
> > > > >
> > > > > Thanks Semyon, sounds great! I would also test it on larger data
> sets
> > > to
> > > > > see how a join query will take, say, on 10GB of data. Is it
> possible?
> > > > >
> > > > >
> > > > > >
> > > > > > On Fri, Jul 22, 2016 at 5:39 PM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > On Fri, Jul 22, 2016 at 7:10 AM, Semyon Boikov <
> > > sboi...@gridgain.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Today I merged into master 'distributed join' implementation
> -
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1232 (thanks to
> > > > Sergi,
> > > > > he
> > > > > > > > implemented this feature). I think this together with recent
> > > > bugfixes
> > > > > > > worth
> > > > > > > > 1.7 release. Do you think we can cut off 1.7 release branch
> > from
> > > > > > master?
> > > > > > > >
> > > > > > >
> > > > > > > Great news. Can you describe that amount of testing we did for
> > this
> > > > > > > feature?
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 21, 2016 at 7:47 PM, Alexandre Boudnik <
> > > > > > > > alexander.boud...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Sorry, I missed the typo:
> > > > > > > > > To support the point, I would add that PosgreSQL has
> > > demonstrated
> > > > > the
> > > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > > - they do not appear in select * list unless they are
> > specified
> > > > > > > > explicitly
> > > > > > > > > Take care,
> > > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > > >
> > > > > > > > > call me via Google Voice:
> > > > > > > > > 1(405) BUDNIKA
> > > > > > > > > 1(405) 283-6452
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 21, 2016 at 12:46 PM, Alexandre Boudnik
> > > > > > > > >  wrote:
> > > > > > > > > > To support the point, I would add that PosgreSQL has
> > > > demonstrated
> > > > > > the
> > > > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > > > - they appear in select * list unless they are specified
> > > > > explicitly
> > > > > > > > > >
> > > > > > > > > > I'll add some notices to the ticket
> > > > > > > > > > Take care,
> > > > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > > > >
> > > > > > > > > > call me via Google Voice:
> > > > > > > > > > 1(405) BUDNIKA
> > > > > > > > > > 1(405) 283-6452
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 15, 2016 at 6:37 PM, Valentin Kulichenko
> > > > > > > > > >  wrote:
> > > > > > > > > >> Agree.
> > > > > > > > > >>
> > > > > > > > 

[jira] [Created] (IGNITE-3607) Send non-filtered continuous query notifications in one message per client

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-3607:


 Summary: Send non-filtered continuous query notifications in one 
message per client
 Key: IGNITE-3607
 URL: https://issues.apache.org/jira/browse/IGNITE-3607
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 1.4
Reporter: Alexey Goncharuk


If a client subscribes N continuous queries to a cache, Ignite will send 
individual notifications for each of the updates. We can send notifications for 
all queries with null filter in one message per client.

See 
http://apache-ignite-users.70518.x6.nabble.com/Deadlock-during-Client-Continuous-Query-deserialization-td6565.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Deadlock during Client Continuous Query deserialization

[Semyon Boikov]

Here is JIRA issue: https://issues.apache.org/jira/browse/IGNITE-3606

On Fri, Jul 29, 2016 at 5:48 PM, Semyon Boikov  wrote:

> Hi,
>
> I reproduced this issue, thank you for test! After quick debugging It
> seems that this is some problem with Ignite backpressure mechanism in
> communication SPI, I'll create JIRA issue with more details.
>
> As a workaround could you try to disable backpressure:
> tcpCommunicationSpi.setMessageQueueLimit(0);
>
> Thanks!
>
> On Fri, Jul 29, 2016 at 1:15 PM, ross.anderson 
> wrote:
>
>> Hi, has anyone had a chance to take a look at this?
>> I'd imagine it's a critical issue if a client can cause a cluster to
>> freeze
>> indefinitely based solely on a smallish number of queries running?
>> If there's anything else I can provide to assist please do let me know, as
>> this is a blocker for us using Ignite.
>> Thanks, Ross
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Deadlock-during-Client-Continuous-Query-deserialization-tp6565p6616.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

[jira] [Created] (IGNITE-3606) Node sometimes fails to detect broken connection

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-3606:


 Summary: Node sometimes fails to detect broken connection
 Key: IGNITE-3606
 URL: https://issues.apache.org/jira/browse/IGNITE-3606
 Project: Ignite
  Issue Type: Bug
  Components: general
Reporter: Semen Boikov
Priority: Critical
 Fix For: 1.8


Here is test reproducing issue https://github.com/rossdanderson/IgniteDeadlock.

When I run this test observe this sequence:
- server starts
- client starts
- server sends 2000 messages to client, on client node communication 
backpressure pauses reads
- server gets write timeout and closes socket
- for some reason client does not detect that existing connection was broken 
and thinks that connection is still established (most probably because reads 
are paused and node does not try to access connection)
- when server tries to re-connect then client sees that connection already 
established and rejects connection, so server constantly tries to reconnect and 
does not exist from reconnect loop:
{noformat}
"main" prio=6 tid=0x01f4a000 nid=0x3588 waiting on condition 
[0x021ed000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at 
org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7414)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2055)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1970)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1936)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1304)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1540)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3605) Commits\rollbacks count is ZERO for transactional cache in case if the load generated by client node

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-3605:
--

 Summary: Commits\rollbacks count is ZERO for transactional cache 
in case if the load generated by client node 
 Key: IGNITE-3605
 URL: https://issues.apache.org/jira/browse/IGNITE-3605
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Priority: Blocker


Start any cluster
Start any load via client node to transactional cache
Take metrics from this cache and verify amount of commits and rollbacks




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3604) .NET: Inconsistent namespace imports in examples

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3604:
--

 Summary: .NET: Inconsistent namespace imports in examples
 Key: IGNITE-3604
 URL: https://issues.apache.org/jira/browse/IGNITE-3604
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 1.5.0.final
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.0


Some files have imports within namespace declaration, some outside, some even 
mix these two.

Both styles are equally popular in the wild, so we should just make examples 
consistent with the rest of the code base (which uses imports within the 
declaration).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Documentation for IGNITE-3399 (Support primitive type names in QueryEntity)

[NoTrueScotsman]

Hi Pavel, yes that's correct re "int". Thanks for the info.

Thanks
Jens

On Thu, Jul 28, 2016 at 1:29 PM, Pavel Tupitsyn  wrote:
> Hi,
>
> Do I understand correctly that we can now replace
> 
> with
> 
> in QueryEntity.fields?
>
> There are Spring XML examples on apacheignite-net.readme.io, these can be
> fixed in 1.7.
>
> But not many people use Spring in .NET, so this change does not affect
> Ignite.NET much.
> When queries are configured in native C# code or app.config, we map .NET
> primitives to Java automatically.
>
> Pavel.
>
> On Thu, Jul 28, 2016 at 1:20 PM, NoTrueScotsman 
> wrote:
>
>> Hi all,
>>
>> I worked on this issue and wondered whether it is worth adding
>> something to the documentation. Candidates:
>> https://apacheignite.readme.io/docs/cache-queries
>> https://apacheignite.readme.io/docs/sql-queries
>>
>> The ticket description also mentioned improved usability for .NET and
>> C++ users, so maybe it's good to mention it too:
>> https://apacheignite-net.readme.io/docs/sql-queries
>>
>> I'm not familiar with .NET so not sure about what the improvements
>> would look like.
>>
>> What do you think?
>>
>> Thanks
>> Jens
>>

Re: Batch support in Cassandra store

[Valentin Kulichenko]

Hi Igor,

I'm not a big Cassandra expert, but here are my thoughts.

1. Sending updates in a batch is always better than sending them one by
one. For example, if you do putAll in Ignite with 100 entries, and these
entries are split across 5 nodes, the client will send 5 requests instead
of 100. This provides significant performance improvement. Is there a way
to use similar approach in Cassandra?
2. As for logged batches, I can easily believe that this is a rarely used
feature, but since it exists in Cassandra, I can't find a single reason why
not to support it in our store as an option. Users that come across those
rare cases, will only say thank you to us :)

What do you think?

-Val

On Thu, Jul 28, 2016 at 10:41 PM, Igor Rudyak  wrote:

> There are actually some cases when atomic read isolation in Cassandra could
> be important. Lets assume batch was persisted in Cassandra, but not
> finalized yet - read operation from Cassandra returns us only partially
> committed data of the batch. In the such situation we have problems when:
>
> 1) Some of the batch records already expired from Ignite cache and we
> reading them from persistent store (Cassandra in our case).
>
> 2) All Ignite nodes storing the batch records (or subset records) died (or
> for example became unavailable for 10sec because of network problem). While
> reading such records from Ignite cache we will be redirected to persistent
> store.
>
> 3) Network separation occurred such a way that we now have two Ignite
> cluster, but all the replicas of the batch data are located only in one of
> these clusters. Again while reading such records from Ignite cache on the
> second cluster we will be redirected to persistent store.
>
> In all mentioned cases, if Cassandra batch isn't finalized yet - we will
> read partially committed transaction data.
>
>
> On Thu, Jul 28, 2016 at 6:52 AM, Luiz Felipe Trevisan <
> luizfelipe.trevi...@gmail.com> wrote:
>
> > I totally agree with you regarding the guarantees we have with logged
> > batches and I'm also pretty much aware of the performance penalty
> involved
> > using this solution.
> >
> > But since all read operations are executed via ignite it means that
> > isolation in the Cassandra level is not really important. I think the
> only
> > guarantee really needed is that we don't end up with a partial insert in
> > Cassandra in case we have a failure in ignite and we loose the node that
> > was responsible for this write operation.
> >
> > My other assumption is that the write operation needs to finish before an
> > eviction happens for this entry and we loose the data in cache (since
> batch
> > doesn't guarantee isolation). However if we cannot achieve this I don't
> see
> > why use ignite as a cache store.
> >
> > Luiz
> >
> > --
> > Luiz Felipe Trevisan
> >
> > On Wed, Jul 27, 2016 at 4:55 PM, Igor Rudyak  wrote:
> >
> >> Hi Luiz,
> >>
> >> Logged batches is not the solution to achieve atomic view of your Ignite
> >> transaction changes in Cassandra.
> >>
> >> The problem with logged batches(aka atomic) is they guarantees that if
> >> any part of the batch succeeds, all of it will, no other transactional
> >> enforcement is done at the batch level. For example, there is no batch
> >> isolation. Clients are able to read the first updated rows from the
> batch,
> >> while other rows are still being updated on the server (in RDBMS
> >> terminology it means *READ-UNCOMMITED* isolation level). Thus Cassandra
> >> mean "atomic" in the database sense that if any part of the batch
> succeeds,
> >> all of it will.
> >>
> >> Probably the best way to archive read atomic isolation for Ignite
> >> transaction persisting data into Cassandra, is to implement RAMP
> >> transactions (http://www.bailis.org/papers/ramp-sigmod2014.pdf) on top
> >> of Cassandra.
> >>
> >> I may create a ticket for this if community would like it.
> >>
> >>
> >> Igor Rudyak
> >>
> >>
> >> On Wed, Jul 27, 2016 at 12:55 PM, Luiz Felipe Trevisan <
> >> luizfelipe.trevi...@gmail.com> wrote:
> >>
> >>> Hi Igor,
> >>>
> >>> Does it make sense for you using logged batches to guarantee atomicity
> >>> in Cassandra in cases we are doing a cross cache transaction operation?
> >>>
> >>> Luiz
> >>>
> >>> --
> >>> Luiz Felipe Trevisan
> >>>
> >>> On Wed, Jul 27, 2016 at 2:05 AM, Dmitriy Setrakyan <
> >>> dsetrak...@apache.org> wrote:
> >>>
>  I am very confused still. Ilya, can you please explain what happens in
>  Cassandra if user calls IgniteCache.putAll(...) method?
> 
>  In Ignite, if putAll(...) is called, Ignite will make the best effort
> to
>  execute the update as a batch, in which case the performance is
> better.
>  What is the analogy in Cassandra?
> 
>  D.
> 
>  On Tue, Jul 26, 2016 at 9:16 PM, Igor Rudyak 
> wrote:
> 
>  > Dmitriy,
>  >
>  > There is absolutely same approach for all async read/write/delete
>  >