date:20170925

Re: Future of Ignite transactions

2017-09-25 Thread Vladimir Ozerov

Dima,

Becuase if you take any pair of our concurrency/isolation modes, you will
not find anything similar in SQL. The main problem is that there is no
"OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic", but
can be converted to "perssimistic" optionally, and all writes are
"pessimistic". Another problem is that some of our pairs cannot be used in
practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and some
duplicates each other (PESSIMISTIC/REPEATABLE_READ -
PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
reason about. If you cannot explain API in 5 mins, it is broken :-)

If we were many years in the past allowed to choose between Ignite and SQL
approaches I would prefer SQL because it is much simpler.

On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan 
wrote:

> Vladimir,
>
> This looks like a complete overhaul of our transactional behavior and I do
> not think we are at a liberty to make such drastic changes. Can you please
> explain why the current behavior would not map to the SQL transactions?
>
> D.
>
> On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov 
> wrote:
>
> > Folks,
> >
> > Sorry for late reply. I had a chat with several Ignite veterans today. We
> > tried to design transactional SQL for Ignite. One of our questions was
> how
> > to align SQL transactions with current Ignite transactions. And we
> failed.
> > And then we came to conclusion that current transaction API is unusable.
> We
> > have 6 pairs of modes from API standpoint and 4 real modes. This is very
> > counterintuitive and cannot be mapped to any transactional framework our
> > users are familiar with.
> >
> > So we thought how new tx API might looks like, and here is the draft.
> >
> > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > current enum) with three standard modes - READ_COMMITTED,
> REPEATABLE_READ,
> > SERIALIZABLE.
> > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > 4) Reads never acuire locks
> > 5) Writes always acquire locks
> > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> will
> > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> > on any update
> > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > SERIALIZABLE isolation level.
> > 9) Define new API methods:
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > TransactionHint... hints)
> > 10) Deprecate old TX start methods
> >
> > As a result we will have simple, clean and extensible API. Which can be
> > explained to users in 5 minutes, instead of current half an hour. And
> which
> > is perfectly aligned with upcoming transactional SQL.
> >
> > Thoughts?
> >
> >
> > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan  >
> > wrote:
> >
> > > Vova,
> > >
> > > Thanks for doing the research. The changes you are suggesting are a bit
> > too
> > > bold, so let's discuss them in some more detail...
> > >
> > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > We are moving towards DBMS system. None of them has a notion of
> > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > > UPDATE)
> > > > 3) Writes do acuire explicit row locks
> > > > 4) Locks are always acquired immediately once statement is executed
> > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > > between transactions. Some vendors throw an exception in case of
> > > conflict -
> > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> Oracle.
> > > > Others do aggressive locking - ones where two-phase locking algorithm
> > is
> > > > used - SQL Server, MySQL.
> > > >
> > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > Instead,
> > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> become
> > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> > (we
> > > > are going in this direction) read-write conflicts are resolved in
> > manner
> > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > >
> > > > That said, I would propose to think on how transactions could look
> like
> > > in
> > > > future Ignite versions (say, 3.0). My rough vision:
> > > >
> > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > > only
> > > >

Re: Warning if no table for BinaryObject

2017-09-25 Thread Yakov Zhdanov

Pavel, we can optionally enforce that, but not by default.

Dmitry, the problem is the following. User configures Ignite cache and
query entity, does several puts than empty results for SQL. The reason is
very simple - typo in class name or simply different classes in
configuration and of actual cache entry. The fix is pretty simple and make
product more usable.

--Yakov

[jira] [Created] (IGNITE-6498) Web console: Query metadata should show table alias when configured

2017-09-25 Thread Vasiliy Sisko (JIRA)

Vasiliy Sisko created IGNITE-6498:
-

 Summary: Web console: Query metadata should show table alias when 
configured
 Key: IGNITE-6498
 URL: https://issues.apache.org/jira/browse/IGNITE-6498
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Affects Versions: 2.1
Reporter: Vasiliy Sisko


On SQL page in cache metadata panel name of stored type is showed.
When type have alias query with original type name is inconsistent.
{code}



{code}

Should be showed alias instead of type name.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2744: IGNITE-6444: Validate that copyOnRead flag is con...

2017-09-25 Thread shroman

GitHub user shroman opened a pull request:

https://github.com/apache/ignite/pull/2744

IGNITE-6444: Validate that copyOnRead flag is configured with on-heap cache 
enabled.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shroman/ignite IGNITE-6444

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2744


commit 6652b70e144cd4627455c69072f5665581f793ca
Author: shroman 
Date:   2017-09-25T03:56:56Z

IGNITE-6444: Validate that copyOnRead flag is configured with on-heap cache 
enabled

commit 3ea14c961dacf9277d23a2e72304043ed8eb2f70
Author: shroman 
Date:   2017-09-26T01:58:51Z

IGNITE-6444: Validate that copyOnRead flag is configured with on-heap cache 
enabled.

commit cd5ee5b367a80d0c780eb5782eb82c85c98e2fb5
Author: shroman 
Date:   2017-09-26T03:31:32Z

IGNITE-6444: Validate that copyOnRead flag is configured with on-heap cache 
enabled.




---

Re: Issues if -Djava.net.preferIPv4Stack=true is not set

2017-09-25 Thread Valentin Kulichenko

Yakov,

Here are couple examples:
- http://apache-ignite-users.70518.x6.nabble.com/How-to-
correctly-shut-down-Ignite-Application-td12548.html
- http://apache-ignite-users.70518.x6.nabble.com/Nodes-
hanging-when-accessing-queue-td12343.html

*Andrey*, it seems that you give this kind of recommendation every now and
then. Can you please provide more detail on this? Is this something that
needs to be fixed in the product?

-Val

On Tue, Sep 19, 2017 at 4:36 AM, Yakov Zhdanov  wrote:

> Val, can you please provide links to threads you meant? I will take a look.
>
> --Yakov
>

Re: Persistence per memory policy configuration

2017-09-25 Thread Dmitriy Setrakyan

Given that we already have a notion of CacheStore which comes from JCache
spec, I think having other stores may get confusing. I like
DurableMemoryConfiguration.

Other opinions?

D.

On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov 
wrote:

> Dima, let's finalize the design first.
>
> As I understand, we are happy with idea to merge MemoryConfiguration
> and PersistentStoreConfiguration
> into something what I called DataConfiguration, and to rename
> MemoryPolicyConfiguration to DataRegionConfiguration.
>
> The only outstanding qurestion is whether DataConfiguration is a good name.
> I am not very happy with it, so let's think of other alternatives. Quick
> ideas:
> 1) StoreConfiguration - looks perfect to me - short and self-describing,
> but clashes a bit with existing CacheStore
> 2) DataStoreConfiguration - same as p.1, but the word "data" is not
> necessary IMO
> 3) PageStoreConfiguration? GIves a hint to our page-based architecture.
> 4) DurableMemoryConfiguration - aligns well with our docs, but I do not
> like it - too long and misleading
>
> Any other ideas?
>
> I would prefer to have either [StoreConfiguration +
> StoreRegionConfiguration] or [PageStoreConfiguration and
> PageStoreRegionConfiguration]. Looks clean and simple.
>
> Vladimir.
>
>
> On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan 
> wrote:
>
> > Vladimir,
> >
> > Can you please add the configuration example in the ticket?
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> > > Guys,
> > >
> > > I suggest we finalize the configuration changes in the original ticket
> > > then: https://issues.apache.org/jira/browse/IGNITE-6030 and proceed
> with
> > > the changes.
> > >
> > > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Can we specify what metrics will look like? I think we should not
> just
> > > > blindly merge them.
> > > >
> > > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > Makes sense. Thanks for catching it!
> > > > >
> > > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda 
> > > wrote:
> > > > >
> > > > > > If we’re taking the consolidation path for Memory and Persistence
> > > > > > configurations then it makes sense to merge MemoryMetrics [1] and
> > > > > > PersistenceMetrics [2] plus their JMX beans.
> > > > > >
> > > > > > Agree?
> > > > > >
> > > > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > apache/ignite/MemoryMetrics.html  > > > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > apache/ignite/
> > > > > > PersistenceMetrics.html
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > I like it.
> > > > > > >
> > > > > > > Alexey G, can you please chime in?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > > > > voze...@gridgain.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Guys,
> > > > > > >>
> > > > > > >> Here is my proposal:
> > > > > > >>
> > > > > > >> 1) MemoryPolicyConfiguration is renamed to
> > > > *DataRegionConfiguration*.
> > > > > > >> 2) PersistenceConfiguration is merged with MemoryConfiguration
> > and
> > > > > > renamed
> > > > > > >> to ... *DataStorageConfiguration*! It has: common memory
> > settings
> > > > > (e.g.
> > > > > > >> default data region), persistence settings (e.g. WAL) and a
> list
> > > of
> > > > > > >> DataRegionConfiguration beans.
> > > > > > >>
> > > > > > >> What we have in the end:
> > > > > > >>
> > > > > > >> 
> > > > > > >>
> > > > > > >>
> > > > > > >> />
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> > value="1_000_000_000"
> > > />
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> > value="1_000_000_000"
> > > />
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> 
> > > > > > >>
> > > > > > >>
> > > > > > >> Makes sense?
> > > > > > >>
> > > > > > >> Vladimir.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Sep 21, 2017 at 7:04 AM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Firstly all, why not call it DataPolicy instead of
> > MemoryPolicy?
> > > > > > >> Secondly,
> > > > > > >>> why not set data policies directly on IgniteConfiguration.
> And
> > > > > lastly,
> >

Re: Future of Ignite transactions

2017-09-25 Thread Dmitriy Setrakyan

Vladimir,

This looks like a complete overhaul of our transactional behavior and I do
not think we are at a liberty to make such drastic changes. Can you please
explain why the current behavior would not map to the SQL transactions?

D.

On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov 
wrote:

> Folks,
>
> Sorry for late reply. I had a chat with several Ignite veterans today. We
> tried to design transactional SQL for Ignite. One of our questions was how
> to align SQL transactions with current Ignite transactions. And we failed.
> And then we came to conclusion that current transaction API is unusable. We
> have 6 pairs of modes from API standpoint and 4 real modes. This is very
> counterintuitive and cannot be mapped to any transactional framework our
> users are familiar with.
>
> So we thought how new tx API might looks like, and here is the draft.
>
> 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
> SERIALIZABLE.
> 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> 4) Reads never acuire locks
> 5) Writes always acquire locks
> 6) *IgniteCache.withReadForUpdate()* will return special facade which will
> obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> on any update
> 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> SERIALIZABLE isolation level.
> 9) Define new API methods:
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> TransactionHint... hints)
> 10) Deprecate old TX start methods
>
> As a result we will have simple, clean and extensible API. Which can be
> explained to users in 5 minutes, instead of current half an hour. And which
> is perfectly aligned with upcoming transactional SQL.
>
> Thoughts?
>
>
> On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan 
> wrote:
>
> > Vova,
> >
> > Thanks for doing the research. The changes you are suggesting are a bit
> too
> > bold, so let's discuss them in some more detail...
> >
> > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Igniters,
> > >
> > > We are moving towards DBMS system. None of them has a notion of
> > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > UPDATE)
> > > 3) Writes do acuire explicit row locks
> > > 4) Locks are always acquired immediately once statement is executed
> > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > between transactions. Some vendors throw an exception in case of
> > conflict -
> > > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > > Others do aggressive locking - ones where two-phase locking algorithm
> is
> > > used - SQL Server, MySQL.
> > >
> > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> Instead,
> > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> (we
> > > are going in this direction) read-write conflicts are resolved in
> manner
> > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > >
> > > That said, I would propose to think on how transactions could look like
> > in
> > > future Ignite versions (say, 3.0). My rough vision:
> > >
> > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > only
> > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> have
> > > good deadlock detector and nice administrative capabilities, this would
> > not
> > > be a problem for us.
> >
> >
> > Hm... The advantage of Optimistic Serialiazable mode is actually
> lock-free
> > transactions. The deadlock is impossible in this case. I doubt any
> deadlock
> > detector would match the performance advantage we get from lock-free
> > transactions.
> >
> >
> > >
> > > 2) Behavior of reads could be controlled through "with" facade:
> > > V val1 = cache.get(key1); // Shared lock or no lock
> > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > >
> >
> > Don't like the API. We are not trying to abandon the data grid use-case
> or
> > API, we are trying to add the database use case.
> >
> >
> > > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> > >
> >
> > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > convenient than the "withForUpdate" API. It almost seems like you are
> >

[GitHub] ignite pull request #2425: IGNITE-5736 Add test of backward-compatibility

2017-09-25 Thread daradurvs

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2425


---

Re: Persistence per memory policy configuration

2017-09-25 Thread Vladimir Ozerov

Dima, let's finalize the design first.

As I understand, we are happy with idea to merge MemoryConfiguration
and PersistentStoreConfiguration
into something what I called DataConfiguration, and to rename
MemoryPolicyConfiguration to DataRegionConfiguration.

The only outstanding qurestion is whether DataConfiguration is a good name.
I am not very happy with it, so let's think of other alternatives. Quick
ideas:
1) StoreConfiguration - looks perfect to me - short and self-describing,
but clashes a bit with existing CacheStore
2) DataStoreConfiguration - same as p.1, but the word "data" is not
necessary IMO
3) PageStoreConfiguration? GIves a hint to our page-based architecture.
4) DurableMemoryConfiguration - aligns well with our docs, but I do not
like it - too long and misleading

Any other ideas?

I would prefer to have either [StoreConfiguration +
StoreRegionConfiguration] or [PageStoreConfiguration and
PageStoreRegionConfiguration]. Looks clean and simple.

Vladimir.


On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan 
wrote:

> Vladimir,
>
> Can you please add the configuration example in the ticket?
>
> D.
>
> On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Guys,
> >
> > I suggest we finalize the configuration changes in the original ticket
> > then: https://issues.apache.org/jira/browse/IGNITE-6030 and proceed with
> > the changes.
> >
> > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Can we specify what metrics will look like? I think we should not just
> > > blindly merge them.
> > >
> > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > Makes sense. Thanks for catching it!
> > > >
> > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda 
> > wrote:
> > > >
> > > > > If we’re taking the consolidation path for Memory and Persistence
> > > > > configurations then it makes sense to merge MemoryMetrics [1] and
> > > > > PersistenceMetrics [2] plus their JMX beans.
> > > > >
> > > > > Agree?
> > > > >
> > > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > apache/ignite/MemoryMetrics.html  > > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > > apache/ignite/
> > > > > PersistenceMetrics.html
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > I like it.
> > > > > >
> > > > > > Alexey G, can you please chime in?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > > > voze...@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Guys,
> > > > > >>
> > > > > >> Here is my proposal:
> > > > > >>
> > > > > >> 1) MemoryPolicyConfiguration is renamed to
> > > *DataRegionConfiguration*.
> > > > > >> 2) PersistenceConfiguration is merged with MemoryConfiguration
> and
> > > > > renamed
> > > > > >> to ... *DataStorageConfiguration*! It has: common memory
> settings
> > > > (e.g.
> > > > > >> default data region), persistence settings (e.g. WAL) and a list
> > of
> > > > > >> DataRegionConfiguration beans.
> > > > > >>
> > > > > >> What we have in the end:
> > > > > >>
> > > > > >> 
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> value="1_000_000_000"
> > />
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> value="1_000_000_000"
> > />
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> 
> > > > > >>
> > > > > >>
> > > > > >> Makes sense?
> > > > > >>
> > > > > >> Vladimir.
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Sep 21, 2017 at 7:04 AM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Firstly all, why not call it DataPolicy instead of
> MemoryPolicy?
> > > > > >> Secondly,
> > > > > >>> why not set data policies directly on IgniteConfiguration. And
> > > > lastly,
> > > > > >> how
> > > > > >>> about we combine memory and disk properties in one bean with
> > clear
> > > > > naming
> > > > > >>> convention?
> > > > > >>>
> > > > > >>> Here is the example. Note that all properties above must start
> > with
> > > > > with
> > > > > >>> "Memory" or "Disk".
> > > > > >>>
> > > > > >>> *IgniteConfiguration cfg = new IgniteConfiguration();*
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > >  *cfg.setDataPolicies(new DataPolicyConfiguration()
> > >

Re: Future of Ignite transactions

2017-09-25 Thread Vladimir Ozerov

Folks,

Sorry for late reply. I had a chat with several Ignite veterans today. We
tried to design transactional SQL for Ignite. One of our questions was how
to align SQL transactions with current Ignite transactions. And we failed.
And then we came to conclusion that current transaction API is unusable. We
have 6 pairs of modes from API standpoint and 4 real modes. This is very
counterintuitive and cannot be mapped to any transactional framework our
users are familiar with.

So we thought how new tx API might looks like, and here is the draft.

1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
SERIALIZABLE.
2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
4) Reads never acuire locks
5) Writes always acquire locks
6) *IgniteCache.withReadForUpdate()* will return special facade which will
obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
on any update
8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
SERIALIZABLE isolation level.
9) Define new API methods:
- IgniteTransactions.txStart(TransactionIsolationLevel isolation)
- IgniteTransactions.txStart(TransactionIsolationLevel isolation,
TransactionHint... hints)
10) Deprecate old TX start methods

As a result we will have simple, clean and extensible API. Which can be
explained to users in 5 minutes, instead of current half an hour. And which
is perfectly aligned with upcoming transactional SQL.

Thoughts?


On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan 
wrote:

> Vova,
>
> Thanks for doing the research. The changes you are suggesting are a bit too
> bold, so let's discuss them in some more detail...
>
> On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > We are moving towards DBMS system. None of them has a notion of
> > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > 1) Reads (SELECT) do not acquire exclusive row locks
> > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > UPDATE)
> > 3) Writes do acuire explicit row locks
> > 4) Locks are always acquired immediately once statement is executed
> > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > so-called *range locks* (or *predicate locks*) to track dependencies
> > between transactions. Some vendors throw an exception in case of
> conflict -
> > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > Others do aggressive locking - ones where two-phase locking algorithm is
> > used - SQL Server, MySQL.
> >
> > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes. Instead,
> > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors (we
> > are going in this direction) read-write conflicts are resolved in manner
> > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> >
> > That said, I would propose to think on how transactions could look like
> in
> > future Ignite versions (say, 3.0). My rough vision:
> >
> > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> only
> > advantage is deadlock-freedom when combined with SERIALIZABLE. If we have
> > good deadlock detector and nice administrative capabilities, this would
> not
> > be a problem for us.
>
>
> Hm... The advantage of Optimistic Serialiazable mode is actually lock-free
> transactions. The deadlock is impossible in this case. I doubt any deadlock
> detector would match the performance advantage we get from lock-free
> transactions.
>
>
> >
> > 2) Behavior of reads could be controlled through "with" facade:
> > V val1 = cache.get(key1); // Shared lock or no lock
> > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> >
>
> Don't like the API. We are not trying to abandon the data grid use-case or
> API, we are trying to add the database use case.
>
>
> > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> >
>
> Well, I would like to preserve the PESSIMISTIC mode. I find it more
> convenient than the "withForUpdate" API. It almost seems like you are
> trying to force the pendulum too far in the opposite direction.
>
>
> > 4) SERIALIZABLE - throw exception in case of write-write and write-read
> > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> doesn't
> > support predicates)
> >
>
> So, no change here? Good :)
>
>
> > 5) Add READ_ONLY isolation mode where updates will not be allowed at all.
> > Such transacrtons would be able to bypass some Ignite internals to
> achieve
> > greater performance, what could be valuable for mostly-read use cases
> (e.g.
> > OLAP).
>

[jira] [Created] (IGNITE-6497) Broken tests in ignite-2.1.5

2017-09-25 Thread Yury Babak (JIRA)

Yury Babak created IGNITE-6497:
--

 Summary: Broken tests in ignite-2.1.5
 Key: IGNITE-6497
 URL: https://issues.apache.org/jira/browse/IGNITE-6497
 Project: Ignite
  Issue Type: Bug
  Components: ml
Affects Versions: 2.1, 2.2
Reporter: Yury Babak
Assignee: Artem Malykh


https://ci.ignite.apache.org/viewLog.html?buildId=852249=buildResultsDiv=Ignite20Tests_IgniteMl





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6496) Client node does not reconnect to server node when the latter is restarted.

2017-09-25 Thread Vyacheslav Koptilin (JIRA)

Vyacheslav Koptilin created IGNITE-6496:
---

 Summary: Client node does not reconnect to server node when the 
latter is restarted.
 Key: IGNITE-6496
 URL: https://issues.apache.org/jira/browse/IGNITE-6496
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 1.9
Reporter: Vyacheslav Koptilin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Integration of Spark and Ignite. Prototype.

2017-09-25 Thread Николай Ижиков

Hello, guys.

Currently, I’m working on integration between Spark and Ignite [1].

For now, I implement following:
* Ignite DataSource implementation(IgniteRelationProvider)
* DataFrame support for Ignite SQL table.
* IgniteCatalog implementation for a transparent resolving of ignites
SQL tables.

Implementation of it can be found in PR [2]
It would be great if someone provides feedback for a prototype.

I made some examples in PR so you can see how API suppose to be used [3].
[4].

I need some advice. Can you help me?

1. How should this PR be tested?

Of course, I need to provide some unit tests. But what about scalability
tests, etc.
Maybe we need some Yardstick benchmark or similar?
What are your thoughts?
Which scenarios should I consider in the first place?

2. Should we provide Spark Catalog implementation inside Ignite codebase?

A current implementation of Spark Catalog based on *internal Spark API*.
Spark community seems not interested in making Catalog API public or
including Ignite Catalog in Spark code base [5], [6].

*Should we include Spark internal API implementation inside Ignite code
base?*

Or should we consider to include Catalog implementation in some external
module?
That will be created and released outside Ignite?(we still can support and
develop it inside Ignite community).

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://github.com/apache/ignite/pull/2742
[3] https://github.com/apache/ignite/pull/2742/files#diff-
f4ff509cef3018e221394474775e0905
[4] https://github.com/apache/ignite/pull/2742/files#diff-
f2b670497d81e780dfd5098c5dd8a89c
[5] http://apache-spark-developers-list.1001551.n3.
nabble.com/Spark-Core-Custom-Catalog-Integration-between-
Apache-Ignite-and-Apache-Spark-td22452.html
[6] https://issues.apache.org/jira/browse/SPARK-17767

--
Nikolay Izhikov
nizhikov@gmail.com

[GitHub] ignite pull request #2743: IGNITE-6445_txLocksInfo

2017-09-25 Thread BiryukovVA

GitHub user BiryukovVA opened a pull request:

https://github.com/apache/ignite/pull/2743

IGNITE-6445_txLocksInfo



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BiryukovVA/ignite IGNITE-6445_txLocksInfo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2743


commit 14611563724432e87ea64683af35c38b99e766ed
Author: Vitaliy Biryukov 
Date:   2017-09-19T15:39:40Z

IGNITE-6442: Bug fixed.

commit 0b6c87d097a180820dcf14fcbd9f48be39b24911
Author: Vitaliy Biryukov 
Date:   2017-09-22T08:08:26Z

IGNITE-6442: Bug fixed in another way.

commit e5433647354ef90571207ca6faef53f768e06fe3
Author: Vitaliy Biryukov 
Date:   2017-09-24T23:22:34Z

IGNITE-6442: Refactoring.

commit 65cb5c17035106a52cec31a08ac6983193bb6adc
Author: Vitaliy Biryukov 
Date:   2017-09-25T15:13:07Z

IGNITE-6442: fixed test.

commit e812122bed19488df3dd3c6345772841b39b754c
Author: Vitaliy Biryukov 
Date:   2017-09-25T16:40:16Z

IGNITE-6445: Bug fixed.

commit 5f50c13ed6a8bf6f2370a51711ea8ab00b2a40b2
Author: Vitaliy Biryukov 
Date:   2017-09-25T16:41:09Z

Merge branch 'IGNITE-6442' into IGNITE-6445_txLocksInfo

commit 6a51e0dd814043991c5d806b3c66503fc37206f9
Author: Vitaliy Biryukov 
Date:   2017-09-25T16:43:02Z

IGNITE-6445: Test.




---

[GitHub] ignite pull request #2742: IGNITE-3084: Prototype of Ignite DataFrame suppor...

2017-09-25 Thread nizhikov

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite/pull/2742

IGNITE-3084: Prototype of Ignite DataFrame support. 

* DataSource and DataFrame implementation for Ignite SQL Tables.
* IgniteCatalog implementation. 
* Some examples provided. 
* There is only placeholer for an Ignite extra optimization and strategies.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite IGNITE-3084

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2742.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2742


commit f54c51e863665b170a6c59d6c57a9dd52caf3a13
Author: Nikolay Izhikov 
Date:   2017-09-05T14:02:46Z

IGNITE-3084: Prototype of Ignite DataFrame support. IgniteCatalog 
implementation. Some examples provided. There is only placeholer for an Ignite 
extra optimization and strategies.




---

[jira] [Created] (IGNITE-6495) performance measurement of decision trees algorithms

2017-09-25 Thread Oleg Ignatenko (JIRA)

Oleg Ignatenko created IGNITE-6495:
--

 Summary: performance measurement of decision trees algorithms
 Key: IGNITE-6495
 URL: https://issues.apache.org/jira/browse/IGNITE-6495
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Reporter: Oleg Ignatenko
Assignee: Oleg Ignatenko


We want to start tracking our performance to avoid performance degradation. 
Also we need some performance comparison with other ml libs.

This task is "extracted" from IGNITE-6123 because implementation of decision 
trees is not yet integrated into master (IGNITE-5218).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2628: IGNITE-6301: Unused variable removed.

2017-09-25 Thread nizhikov

Github user nizhikov closed the pull request at:

https://github.com/apache/ignite/pull/2628


---

[GitHub] ignite pull request #2663: Ignite 3084 - pre alpha to share with AV

2017-09-25 Thread nizhikov

Github user nizhikov closed the pull request at:

https://github.com/apache/ignite/pull/2663


---

[GitHub] ignite pull request #2741: IGNITE-6494 .NET: Fix CacheConfiguration.WriteSyn...

2017-09-25 Thread ptupitsyn

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/2741

IGNITE-6494 .NET: Fix CacheConfiguration.WriteSynchronizationMode default 
value



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-6494

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2741


commit d6ea01c6e8a8878e7afa53b7a732eb4f30cc17c7
Author: Pavel Tupitsyn 
Date:   2017-09-25T14:59:19Z

IGNITE-6494 .NET: Fix CacheConfiguration.WriteSynchronizationMode default 
value




---

[GitHub] ignite pull request #2740: IGNITE-6242

2017-09-25 Thread alexpaschenko

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/2740

IGNITE-6242



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2740.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2740


commit 6ec566a84748f0171ad924d8e52d793bafc34b01
Author: Alexander Paschenko 
Date:   2017-09-21T19:00:16Z

IGNITE-6242 Cache, key and value names for CREATE TABLE

commit 3194d8771c1765a2a682265bc5d1a96b122519ca
Author: Alexander Paschenko 
Date:   2017-09-25T14:48:18Z

IGNITE-6242 Cache, key and value names for CREATE TABLE tests




---

[GitHub] ignite pull request #2668: ignite-5960 race fixed

2017-09-25 Thread voipp

Github user voipp closed the pull request at:

https://github.com/apache/ignite/pull/2668


---

[GitHub] ignite pull request #2641: ignite-5960 continuous query entry update race fi...

2017-09-25 Thread voipp

Github user voipp closed the pull request at:

https://github.com/apache/ignite/pull/2641


---

[jira] [Created] (IGNITE-6494) .NET: CacheConfiguration.WriteSynchronizationMode incorrect default value

2017-09-25 Thread Pavel Tupitsyn (JIRA)

Pavel Tupitsyn created IGNITE-6494:
--

 Summary: .NET: CacheConfiguration.WriteSynchronizationMode 
incorrect default value
 Key: IGNITE-6494
 URL: https://issues.apache.org/jira/browse/IGNITE-6494
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.3


{{CacheConfiguration.WriteSynchronizationMode}} has no explicit default in 
.NET, which causes it to be {{FullSync}} (first enum value). Java-side default 
is {{PRIMARY_SYNC}}.

Write a test to make sure all defaults are the same (start a cache from Spring 
and get configuration).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Warning if no table for BinaryObject

2017-09-25 Thread Dmitriy Setrakyan

On Fri, Sep 22, 2017 at 9:47 AM, Mikhail Cherkasov 
wrote:

> Hi all,
>
> I've seen several times that due wrong cache configuration people can't
> find
> data in cache and blame Ignite that it's buggy and doesn't work.
>
> And it's very difficult to find an error in the code, especially if you
> don't have reach experience with Ignite.
>
> One of the very common case is that user specifys a name to QueryEntity
> value type:
> queryEntity.setValueType("MyCoolName")
> and later put data to cache like:
> ignite.binary.toBinary(value)
>
> The example is simple and the error is obvious when you see this two lines
> close to each other, however, in real life, cache definition and data
> ingestion are separated by tons of code.
>
>
Mikhail, the example may be simple and obvious to some, but not to most
(including me). Can you please explain what it is we are trying to solve?
What is wrong with the above code snippet?

Re: Apache Ignite 2.3 release

2017-09-25 Thread schernolyas

Hi Vladimir!

Could you add to future release by PR
(http://apache-ignite-developers.2346864.n4.nabble.com/GitHub-ignite-pull-request-2737-IGNITE-6286-fix-org-h2-jdbc-JdbcSQLException-td22625.html)?
The PR is very important for project Hibernate OGM .  



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Persistence per memory policy configuration

2017-09-25 Thread Dmitriy Setrakyan

Vladimir,

Can you please add the configuration example in the ticket?

D.

On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Guys,
>
> I suggest we finalize the configuration changes in the original ticket
> then: https://issues.apache.org/jira/browse/IGNITE-6030 and proceed with
> the changes.
>
> 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan :
>
> > Can we specify what metrics will look like? I think we should not just
> > blindly merge them.
> >
> > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Denis,
> > >
> > > Makes sense. Thanks for catching it!
> > >
> > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda 
> wrote:
> > >
> > > > If we’re taking the consolidation path for Memory and Persistence
> > > > configurations then it makes sense to merge MemoryMetrics [1] and
> > > > PersistenceMetrics [2] plus their JMX beans.
> > > >
> > > > Agree?
> > > >
> > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > apache/ignite/MemoryMetrics.html  > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > apache/ignite/
> > > > PersistenceMetrics.html
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > I like it.
> > > > >
> > > > > Alexey G, can you please chime in?
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > >> Guys,
> > > > >>
> > > > >> Here is my proposal:
> > > > >>
> > > > >> 1) MemoryPolicyConfiguration is renamed to
> > *DataRegionConfiguration*.
> > > > >> 2) PersistenceConfiguration is merged with MemoryConfiguration and
> > > > renamed
> > > > >> to ... *DataStorageConfiguration*! It has: common memory settings
> > > (e.g.
> > > > >> default data region), persistence settings (e.g. WAL) and a list
> of
> > > > >> DataRegionConfiguration beans.
> > > > >>
> > > > >> What we have in the end:
> > > > >>
> > > > >> 
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> />
> > > > >>
> > > > >>
> > > > >>
> > > > >> />
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> 
> > > > >>
> > > > >>
> > > > >> Makes sense?
> > > > >>
> > > > >> Vladimir.
> > > > >>
> > > > >>
> > > > >> On Thu, Sep 21, 2017 at 7:04 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >>> Firstly all, why not call it DataPolicy instead of MemoryPolicy?
> > > > >> Secondly,
> > > > >>> why not set data policies directly on IgniteConfiguration. And
> > > lastly,
> > > > >> how
> > > > >>> about we combine memory and disk properties in one bean with
> clear
> > > > naming
> > > > >>> convention?
> > > > >>>
> > > > >>> Here is the example. Note that all properties above must start
> with
> > > > with
> > > > >>> "Memory" or "Disk".
> > > > >>>
> > > > >>> *IgniteConfiguration cfg = new IgniteConfiguration();*
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > 
> > > > 
> > > > 
> > > > 
> > > >  *cfg.setDataPolicies(new DataPolicyConfiguration()
> > > >  .setName("bla"),.setMemoryMaxSize(1024), // must be
> > greater
> > > > >> than
> > > > >>> 0,
> > > >  since memory always needs to be enabled.
> > .setDiskMaxSize(0),
> > > //
> > > > >> if
> > > >  greater than 0, then persistence is enabled.);*
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> I think this approach is much more concise and straight forward.
> > What
> > > > do
> > > > >>> you think?
> > > > >>>
> > > > >>> D.
> > > > >>>
> > > > >>> On Wed, Sep 20, 2017 at 4:55 AM, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > >  I prefer the second. Composition over inheritance - this is how
> > all
> > > > our
> > > >  configuration is crafted.
> > > > 
> > > >  E.g. we do not have "CacheConfiguration" and "
> > > >  StoreEnabledCacheConfiguration".
> > > >  Instead, we have "CacheConfiguration.setCacheStoreFactory".
> > > > 
> > > >  On Wed, Sep 20, 2017 at 2:46 PM, Alexey Goncharuk <
> > > >  alexey.goncha...@gmail.com> wrote:
> > > > 
> > > > > Reiterating this based on some feedback from PDS users.
> > > > >
> > > > > It might be confusing to configure persistence with
> > "MemoryPolicy",
> > > > >> so
> > > > > another approach is to deprecate the old names and introduce a
> > new
> > > > >> name
> > > > > "DataRegion" because it

[GitHub] ignite pull request #2739: IGNITE-6360: NPE occurs if object with null index...

2017-09-25 Thread mcherkasov

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/2739

IGNITE-6360: NPE occurs if object with null indexed field is added



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6360

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2739.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2739


commit 905312c6e40504cba58328849cbfaa1277b8378d
Author: mcherkasov 
Date:   2017-09-25T10:58:26Z

IGNITE-6360: NPE occurs if object with null indexed field is added




---

[GitHub] ignite pull request #2727: IGNITE-6249 .NET: IgniteConfiguration.ConsistentI...

2017-09-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2727


---

[GitHub] ignite pull request #2729: IGNITE-6486 .NET: Update IsActiveOnStart document...

2017-09-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2729


---

[GitHub] ignite pull request #2738: IGNITE-6493 incorrect usage dataStreamer, must be...

2017-09-25 Thread DmitriyGovorukhin

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/2738

IGNITE-6493 incorrect usage dataStreamer, must be in try-with-resource



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6493

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2738.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2738


commit 79d036a2a0483b59d5e6ece13dd3bd8622d7df6b
Author: Dmitriy Govorukhin 
Date:   2017-09-25T09:47:31Z

IGNITE-6493 fix IgnitePdsWalTlbTest.testWalDirectOutOfMemory() incorrect 
usage dataStreamer, must be in try-with-resource




---

[jira] [Created] (IGNITE-6493) IgnitePdsWalTlbTest.testWalDirectOutOfMemory() hangs

2017-09-25 Thread Dmitriy Govorukhin (JIRA)

Dmitriy Govorukhin created IGNITE-6493:
--

 Summary: IgnitePdsWalTlbTest.testWalDirectOutOfMemory() hangs
 Key: IGNITE-6493
 URL: https://issues.apache.org/jira/browse/IGNITE-6493
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.2
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin
 Fix For: 2.3






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6492) .NET: Tests: Task.Wait should always include a timeout

2017-09-25 Thread Pavel Tupitsyn (JIRA)

Pavel Tupitsyn created IGNITE-6492:
--

 Summary: .NET: Tests: Task.Wait should always include a timeout
 Key: IGNITE-6492
 URL: https://issues.apache.org/jira/browse/IGNITE-6492
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn


Avoid hanging tests: make sure all {{Task.Wait}} calls use an overload with 
timeout.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Apache Ignite 2.3 release

2017-09-25 Thread Pavel Tupitsyn

Hi,

For Ignite.NET I would like to include at least these two:

https://issues.apache.org/jira/browse/IGNITE-5615 .NET:
IgniteConfiguration.LocalEventListeners
https://issues.apache.org/jira/browse/IGNITE-4070 .NET: Introduce
transformers to SCAN queries


Also users are asking for .NET Core and Linux support, so if we have time,
this is the main step in that direction:

https://issues.apache.org/jira/browse/IGNITE-5343 .NET: Interoperate with
JVM directly, get rid of C++ layer


Thanks,
Pavel

On Mon, Sep 25, 2017 at 11:19 AM, schernolyas 
wrote:

> Hi  Vladimir!
>
> Could you please add my PR to next relese. I mean PR
> https://github.com/apache/ignite/pull/2737 .
> The PR is very important for project Hibernate OGM.
>
> A lot of thanks!
>
> With best regards, Sergey Chernolyas
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Apache Ignite 2.3 release

2017-09-25 Thread schernolyas

Hi  Vladimir!

Could you please add my PR to next relese. I mean PR
https://github.com/apache/ignite/pull/2737 .
The PR is very important for project Hibernate OGM.

A lot of thanks!

With best regards, Sergey Chernolyas



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Warning if no table for BinaryObject

2017-09-25 Thread Pavel Tupitsyn

We could even add a configuration switch to throw an exception in such case.

I guess that in majority of use cases there is a single type per cache,
and we should have a way to enforce that.

On Fri, Sep 22, 2017 at 7:47 PM, Mikhail Cherkasov 
wrote:

> Hi all,
>
> I've seen several times that due wrong cache configuration people can't
> find
> data in cache and blame Ignite that it's buggy and doesn't work.
>
> And it's very difficult to find an error in the code, especially if you
> don't have reach experience with Ignite.
>
> One of the very common case is that user specifys a name to QueryEntity
> value type:
> queryEntity.setValueType("MyCoolName")
> and later put data to cache like:
> ignite.binary.toBinary(value)
>
> The example is simple and the error is obvious when you see this two lines
> close to each other, however, in real life, cache definition and data
> ingestion are separated by tons of code.
>
> We can save a lot of man-hours for our users if Ignite will print a warning
> If a cache has a configured QE and user puts BinaryObject with typeName
> which doesn't correspond to any QE.
>
> The warning should be printed only once, something like:
> [WARN] No table is found for %typeName% binary object.
>
> Thoughts?
>
>
> --
> Thanks,
> Mikhail.
>

Re: Apache Ignite 2.3 release

2017-09-25 Thread Vladimir Ozerov

You can find the list of 2.3 tickets which are either reported by you or
assigned to you using this report [1]. Please make sure that this list is
relevant and tickets will be completed in a matter of 2 weeks.

NB: You should be logged in to JIRA.

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20AND%20fixVersion%20%3D%202.3%20AND%20status%20not%20in%20(Resolved%2C%20Closed)%20AND%20(reporter%20%3D%20currentUser()%20or%20assignee%20%3D%20currentUser())%20ORDER%20by%20created

On Mon, Sep 25, 2017 at 10:47 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> We already fixed 300+ tickets since AI 2.1.0 release. These tickets
> include a lot of new features (namely, SQL-related) and usability
> improvements. Not to say dozens of important bug fixes. I propose to start
> planning of AI 2.3.0 release. I think we can deliver it in a month as
> follows:
>
> 25/09 - 08/10 - development and stabilization
> 09/10 - 15/10 - testing
> 15/10 - 23/10 - voting phase
>
> 23 Oct - release.
>
> What do you think? If nobody has objections on dates, let's discuss what
> else we would like to include into it.
>
> Vladimir.
>

[GitHub] ignite pull request #2737: IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

2017-09-25 Thread schernolyas

GitHub user schernolyas opened a pull request:

https://github.com/apache/ignite/pull/2737

IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

changes:
1) develop test for check all supported types for parameterized queries
2) add support BigDecimal type as type of paraameter's value

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schernolyas/ignite IGNITE-6286-DEV1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2737.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2737


commit f4e882e6dc4d829e1e143ee050108c21e55afac4
Author: Sergey Chernolyas 
Date:   2017-09-25T07:47:03Z

IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

changes:
1) develop test for check all supported types for parameterized queries
2) add support BigDecimal type as type of paraameter's value




---

Apache Ignite 2.3 release

2017-09-25 Thread Vladimir Ozerov

Igniters,

We already fixed 300+ tickets since AI 2.1.0 release. These tickets include
a lot of new features (namely, SQL-related) and usability improvements. Not
to say dozens of important bug fixes. I propose to start planning of AI
2.3.0 release. I think we can deliver it in a month as follows:

25/09 - 08/10 - development and stabilization
09/10 - 15/10 - testing
15/10 - 23/10 - voting phase

23 Oct - release.

What do you think? If nobody has objections on dates, let's discuss what
else we would like to include into it.

Vladimir.

Re: Persistence per memory policy configuration

2017-09-25 Thread Alexey Goncharuk

Guys,

I suggest we finalize the configuration changes in the original ticket
then: https://issues.apache.org/jira/browse/IGNITE-6030 and proceed with
the changes.

2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan :

> Can we specify what metrics will look like? I think we should not just
> blindly merge them.
>
> On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov 
> wrote:
>
> > Denis,
> >
> > Makes sense. Thanks for catching it!
> >
> > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda  wrote:
> >
> > > If we’re taking the consolidation path for Memory and Persistence
> > > configurations then it makes sense to merge MemoryMetrics [1] and
> > > PersistenceMetrics [2] plus their JMX beans.
> > >
> > > Agree?
> > >
> > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > apache/ignite/MemoryMetrics.html  > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> apache/ignite/
> > > PersistenceMetrics.html
> > >
> > > —
> > > Denis
> > >
> > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > > >
> > > > I like it.
> > > >
> > > > Alexey G, can you please chime in?
> > > >
> > > > D.
> > > >
> > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > wrote:
> > > >
> > > >> Guys,
> > > >>
> > > >> Here is my proposal:
> > > >>
> > > >> 1) MemoryPolicyConfiguration is renamed to
> *DataRegionConfiguration*.
> > > >> 2) PersistenceConfiguration is merged with MemoryConfiguration and
> > > renamed
> > > >> to ... *DataStorageConfiguration*! It has: common memory settings
> > (e.g.
> > > >> default data region), persistence settings (e.g. WAL) and a list of
> > > >> DataRegionConfiguration beans.
> > > >>
> > > >> What we have in the end:
> > > >>
> > > >> 
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> 
> > > >>
> > > >>
> > > >> Makes sense?
> > > >>
> > > >> Vladimir.
> > > >>
> > > >>
> > > >> On Thu, Sep 21, 2017 at 7:04 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Firstly all, why not call it DataPolicy instead of MemoryPolicy?
> > > >> Secondly,
> > > >>> why not set data policies directly on IgniteConfiguration. And
> > lastly,
> > > >> how
> > > >>> about we combine memory and disk properties in one bean with clear
> > > naming
> > > >>> convention?
> > > >>>
> > > >>> Here is the example. Note that all properties above must start with
> > > with
> > > >>> "Memory" or "Disk".
> > > >>>
> > > >>> *IgniteConfiguration cfg = new IgniteConfiguration();*
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > 
> > > 
> > > 
> > > 
> > >  *cfg.setDataPolicies(new DataPolicyConfiguration()
> > >  .setName("bla"),.setMemoryMaxSize(1024), // must be
> greater
> > > >> than
> > > >>> 0,
> > >  since memory always needs to be enabled.
> .setDiskMaxSize(0),
> > //
> > > >> if
> > >  greater than 0, then persistence is enabled.);*
> > > >>>
> > > >>>
> > > >>>
> > > >>> I think this approach is much more concise and straight forward.
> What
> > > do
> > > >>> you think?
> > > >>>
> > > >>> D.
> > > >>>
> > > >>> On Wed, Sep 20, 2017 at 4:55 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > >>> wrote:
> > > >>>
> > >  I prefer the second. Composition over inheritance - this is how
> all
> > > our
> > >  configuration is crafted.
> > > 
> > >  E.g. we do not have "CacheConfiguration" and "
> > >  StoreEnabledCacheConfiguration".
> > >  Instead, we have "CacheConfiguration.setCacheStoreFactory".
> > > 
> > >  On Wed, Sep 20, 2017 at 2:46 PM, Alexey Goncharuk <
> > >  alexey.goncha...@gmail.com> wrote:
> > > 
> > > > Reiterating this based on some feedback from PDS users.
> > > >
> > > > It might be confusing to configure persistence with
> "MemoryPolicy",
> > > >> so
> > > > another approach is to deprecate the old names and introduce a
> new
> > > >> name
> > > > "DataRegion" because it reflects the actual state when data is
> > stored
> > > >>> on
> > > > disk and partially in memory. I have two options in mind, each of
> > > >> them
> > > > looks acceptable to me, so I would like to have some feedback
> from
> > > >> the
> > > > community. Old configuration names will be deprecated (but still
> be
> > > >>> taken
> > > > if used for backward compatibility). Note, that old names
> > deprecation
> > > > handles default configuration compatibility very nicely -

[jira] [Created] (IGNITE-6491) Race in TopologyValidator.validate() and EVT_NODE_LEFT listener calls (split-brain activator)

2017-09-25 Thread Alexandr Kuramshin (JIRA)

Alexandr Kuramshin created IGNITE-6491:
--

 Summary: Race in TopologyValidator.validate() and EVT_NODE_LEFT 
listener calls (split-brain activator)
 Key: IGNITE-6491
 URL: https://issues.apache.org/jira/browse/IGNITE-6491
 Project: Ignite
  Issue Type: Bug
  Components: cache, general
Affects Versions: 2.1
Reporter: Alexandr Kuramshin
Assignee: Alexandr Kuramshin
 Fix For: 2.2


The following wrong cache {{validate}}/{{put}} sequence may occur

On node left {{GridDhtPartitionsExchangeFuture}} will be generated by the 
{{disco-event-worker}} thread.

Then the {{exchange-worker}} thread does

{noformat}
Split-brain detected [cacheName=test40, activatorTopVer=0, cacheTopVer=14]
at 
org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
at 
org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator.validate(IgniteTopologyValidatorGridSplitCacheTest.java:307)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCacheGroup(GridDhtTopologyFutureAdapter.java:64)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1456)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:115)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:668)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2278)
{noformat}

The result of validation is stored in {{grpValidRes}} with value of {{false}}.

After some delay the {{disco-event-worker}} thread will do

{noformat}
java.lang.Exception: Node is segment activator [cacheName=test40, 
activatorTopVer=14]
at 
org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
at 
org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:360)
at 
org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:349)
at 
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1463)
at 
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:859)
at 
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:844)
at 
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
at 
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2478)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2684)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2507)
{noformat}

After this invocation the result of {{SplitAwareTopologyValidator.validate}} 
should be changed to {{true}}, but it was already invoked and the result has 
been cached in {{grpValidRes}} with the value of {{false}}.

So any successive calls to {{cache.put}} causes to fail

{noformat}
Test failed.
java.lang.RuntimeException: tryPut() failed 
[gridName=cache.IgniteTopologyValidatorGridSplitCacheTest0]
at 
org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.tryPut(IgniteTopologyValidatorGridSplitCacheTest.java:262)
at 
org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.testTopologyValidator(IgniteTopologyValidatorGridSplitCacheTest.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at

40 matches

Mail list logo