date:20170926

Re: Future of Ignite transactions

2017-09-26 Thread Dmitriy Setrakyan

OK, if we must change the current behavior, let's discuss the new design.
My comments/questions are below...

On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov 
wrote:

> Folks,
>
> Sorry for late reply. I had a chat with several Ignite veterans today. We
> tried to design transactional SQL for Ignite. One of our questions was how
> to align SQL transactions with current Ignite transactions. And we failed.
> And then we came to conclusion that current transaction API is unusable. We
> have 6 pairs of modes from API standpoint and 4 real modes. This is very
> counterintuitive and cannot be mapped to any transactional framework our
> users are familiar with.
>
> So we thought how new tx API might looks like, and here is the draft.
>
> 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
> SERIALIZABLE.
>

If it is the same values as we have today, why create a new enum?


> 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
>

The word *hint* means no guarantee. When it comes to transactions, we must
have guarantees. Also, what happens to PESSIMISTIC locking?


> 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
>

But you are still proposing OPTIMISTIC_LOCKING above. So, we have
OPTIMISTIC without PESSIMISTIC?


> 4) Reads never acuire locks
> 5) Writes always acquire locks
>

Hm... what locks? I think we are getting to deep into the weeds here. Isn't
it our internal implementation detail to acquire locks or not? For example,
MVCC approach we are working on will be virtually lock-free for all
scenarios.

I would focus on transaction behavior, not locks.



> 6) *IgniteCache.withReadForUpdate()* will return special facade which will
> obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
>

This syntax is very ugly in databases and we do not need to carry this over
to Ignite. All it means is that a pessimistic lock is acquired. To be
honest, I would rather start a pessimistic transaction to get the same
behavior.


> 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> on any update
>

I really like the READ_ONLY flag. Allows us to put many optimizations in
place.


> 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> SERIALIZABLE isolation level.
>

Is there a PESSIMISTIC/SERIALIZABLE mode?


> 9) Define new API methods:
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> TransactionHint... hints)

10) Deprecate old TX start methods
>
> As a result we will have simple, clean and extensible API. Which can be
> explained to users in 5 minutes, instead of current half an hour. And which
> is perfectly aligned with upcoming transactional SQL.
>

The API does not look clean yet. Still requires lots of work.


>
> Thoughts?
>
>
> On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan 
> wrote:
>
> > Vova,
> >
> > Thanks for doing the research. The changes you are suggesting are a bit
> too
> > bold, so let's discuss them in some more detail...
> >
> > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Igniters,
> > >
> > > We are moving towards DBMS system. None of them has a notion of
> > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > UPDATE)
> > > 3) Writes do acuire explicit row locks
> > > 4) Locks are always acquired immediately once statement is executed
> > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > between transactions. Some vendors throw an exception in case of
> > conflict -
> > > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > > Others do aggressive locking - ones where two-phase locking algorithm
> is
> > > used - SQL Server, MySQL.
> > >
> > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> Instead,
> > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> (we
> > > are going in this direction) read-write conflicts are resolved in
> manner
> > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > >
> > > That said, I would propose to think on how transactions could look like
> > in
> > > future Ignite versions (say, 3.0). My rough vision:
> > >
> > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > only
> > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> have
> > > good deadlock detector and nice administrative capabilities, this would
> >

Re: Should we take care of Java 9 in Ignite 2.0 scope?

2017-09-26 Thread Nikita Ivanov

Yes, have a minimum set at Java 8 (ML contributors would greatly appreciate
that...).

--
Nikita Ivanov


On Tue, Sep 26, 2017 at 6:16 PM, Denis Magda  wrote:

> You meant Java 7 drop I guess. That's a good question. Hope this happens
> soon.
>
> Denis
>
> On Tuesday, September 26, 2017, Nikita Ivanov  wrote:
>
> > What about Java 8 to begin with?
> >
> > --
> > Nikita Ivanov
> >
> >
> > On Tue, Sep 26, 2017 at 3:16 PM, Denis Magda  > > wrote:
> >
> > > Eventually now it’s time to put all the doubts aside on when Ignite
> > should
> > > support JDK 9. The JDK is production ready and was released not long
> time
> > > ago.
> > >
> > > Is there anyone interested to enable JDK 9 for Ignite?
> > >
> > > —
> > > Denis
> > >
> > > > On Mar 24, 2017, at 10:22 AM, Denis Magda  > > wrote:
> > > >
> > > > Evgenii,
> > > >
> > > > Thanks for the investigation. Could you update the ticket with all of
> > > your findings and suggestions?
> > > >
> > > > If you feel you can’t fix it by 2.0 then let’s do it in the next
> minor
> > > release like 2.x. I don’t think that we introduce any breaking
> > > compatibilities when JDK 9 support is added later. Do you think so as
> > well?
> > > >
> > > > —
> > > > Denis
> > > >
> > > >> On Mar 24, 2017, at 9:24 AM, Evgenii Zhuravlev <
> > > e.zhuravlev...@gmail.com > wrote:
> > > >>
> > > >> Igniters,
> > > >>
> > > >> I found that maven-war-plugin, that used in ignite-websphere-test
> > > doesn't
> > > >> work with java 9. Here is issue:
> > > >> https://issues.apache.org/jira/browse/MWAR-397.They have workaround
> > > with
> > > >> setting MAVEN_OPTS, but this workaround doesn't fit us.
> > > >>
> > > >> 2017-03-24 0:51 GMT+03:00 zixu mo  > >:
> > > >>
> > > >>>
> > > >>>
> > > >>> JB IDEA2017.1 released, support jdk 9.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, Mar 24, 2017 at 12:14 AM +0800, "Anton Vinogradov" <
> > > >>> avinogra...@gridgain.com > wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> Denis,
> > > >>>
> > > >>> I'm not sure, but javac usage seems to be not our but IDE
> "problem".
> > > >>> No one call javac directly, AFAIK.
> > > >>>
> > > >>> So, any --add-exports can be added to wrapping maven, .bat/.sh or
> IDE
> > > >>> settings.
> > > >>> I'm pretty sure JB's IDEA will handle this right after Java9
> release
> > :)
> > > >>>
> > > >>> On Wed, Mar 22, 2017 at 12:19 PM, Anton Vinogradov  wrote:
> > > >>>
> > >  Denis,
> > > 
> > >  I'm not sure, but javac usage seems to be not our but IDE
> "problem".
> > >  No one call javac directly, AFAIK.
> > > 
> > >  So, any --add-exports can be added to wrapping maven, .bat/.sh or
> > IDE
> > >  settings.
> > >  I'm pretty sure JB's IDEA will handle this right after Java9
> release
> > > :)
> > > 
> > >  On Wed, Mar 22, 2017 at 1:43 AM, Denis Magda  wrote:
> > > 
> > > > For using internal classes that not exported by default, we will
> > > need to
> > > > add new args(--add-exports) when compiling (javac) *and* when
> > running
> > > > (java
> > > > ).
> > > > Is it ok?
> > > >
> > > >
> > > > Anton, are you fine with this approach?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > On Mar 21, 2017, at 11:05 AM, Evgenii Zhuravlev
> > > > wrote:
> > > >
> > > > Denis,
> > > >
> > > > I've found that some internal classes like
> sun.misc.SharedSecrets,
> > > > sun.misc.URLClassPath, sun.misc.PerfCounter, sun.misc.Cleaner
> > changed
> > > > their
> > > > packages. I can create wrapper for this classes with 2 modules,
> > that
> > > can
> > > > be
> > > > enabled by profiles for java9 and java7-8.
> > > > For using internal classes that not exported by default, we will
> > > need to
> > > > add new args(--add-exports) when compiling (javac) *and* when
> > running
> > > > (java
> > > > ).
> > > > Is it ok?
> > > >
> > > >
> > > > Also, there are a two different Unsafe classes in java 9 -
> > > > sun.misc.Unsafe
> > > > & jdk.internal.misc.Unsafe, but both of them doesn't have
> > > monitorEnter &
> > > > monitorExit methods, that we use in
> > > > org.apache.ignite.internal.util.GridUnsafe. What should I do
> with
> > > this
> > > > case?
> > > >
> > > >
> > > >
> > > > 2017-03-21 11:21 GMT+03:00 Евгений Журавлев :
> > > >
> > > > Denis,
> > > >
> > > > I found some recommendations on openjdk wiki:
> > > > https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+
> > > >>> Analysis+Tool
> > > >
> > > > 2017-03-20 20:44 GMT+03:00 Denis Magda :
> > > >
> > > > Referring to your report only the following API was removed
> > > completely.

Re: Should we take care of Java 9 in Ignite 2.0 scope?

2017-09-26 Thread Denis Magda

You meant Java 7 drop I guess. That's a good question. Hope this happens
soon.

Denis

On Tuesday, September 26, 2017, Nikita Ivanov  wrote:

> What about Java 8 to begin with?
>
> --
> Nikita Ivanov
>
>
> On Tue, Sep 26, 2017 at 3:16 PM, Denis Magda  > wrote:
>
> > Eventually now it’s time to put all the doubts aside on when Ignite
> should
> > support JDK 9. The JDK is production ready and was released not long time
> > ago.
> >
> > Is there anyone interested to enable JDK 9 for Ignite?
> >
> > —
> > Denis
> >
> > > On Mar 24, 2017, at 10:22 AM, Denis Magda  > wrote:
> > >
> > > Evgenii,
> > >
> > > Thanks for the investigation. Could you update the ticket with all of
> > your findings and suggestions?
> > >
> > > If you feel you can’t fix it by 2.0 then let’s do it in the next minor
> > release like 2.x. I don’t think that we introduce any breaking
> > compatibilities when JDK 9 support is added later. Do you think so as
> well?
> > >
> > > —
> > > Denis
> > >
> > >> On Mar 24, 2017, at 9:24 AM, Evgenii Zhuravlev <
> > e.zhuravlev...@gmail.com > wrote:
> > >>
> > >> Igniters,
> > >>
> > >> I found that maven-war-plugin, that used in ignite-websphere-test
> > doesn't
> > >> work with java 9. Here is issue:
> > >> https://issues.apache.org/jira/browse/MWAR-397.They have workaround
> > with
> > >> setting MAVEN_OPTS, but this workaround doesn't fit us.
> > >>
> > >> 2017-03-24 0:51 GMT+03:00 zixu mo  >:
> > >>
> > >>>
> > >>>
> > >>> JB IDEA2017.1 released, support jdk 9.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Mar 24, 2017 at 12:14 AM +0800, "Anton Vinogradov" <
> > >>> avinogra...@gridgain.com > wrote:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Denis,
> > >>>
> > >>> I'm not sure, but javac usage seems to be not our but IDE "problem".
> > >>> No one call javac directly, AFAIK.
> > >>>
> > >>> So, any --add-exports can be added to wrapping maven, .bat/.sh or IDE
> > >>> settings.
> > >>> I'm pretty sure JB's IDEA will handle this right after Java9 release
> :)
> > >>>
> > >>> On Wed, Mar 22, 2017 at 12:19 PM, Anton Vinogradov  wrote:
> > >>>
> >  Denis,
> > 
> >  I'm not sure, but javac usage seems to be not our but IDE "problem".
> >  No one call javac directly, AFAIK.
> > 
> >  So, any --add-exports can be added to wrapping maven, .bat/.sh or
> IDE
> >  settings.
> >  I'm pretty sure JB's IDEA will handle this right after Java9 release
> > :)
> > 
> >  On Wed, Mar 22, 2017 at 1:43 AM, Denis Magda  wrote:
> > 
> > > For using internal classes that not exported by default, we will
> > need to
> > > add new args(--add-exports) when compiling (javac) *and* when
> running
> > > (java
> > > ).
> > > Is it ok?
> > >
> > >
> > > Anton, are you fine with this approach?
> > >
> > > —
> > > Denis
> > >
> > > On Mar 21, 2017, at 11:05 AM, Evgenii Zhuravlev
> > > wrote:
> > >
> > > Denis,
> > >
> > > I've found that some internal classes like sun.misc.SharedSecrets,
> > > sun.misc.URLClassPath, sun.misc.PerfCounter, sun.misc.Cleaner
> changed
> > > their
> > > packages. I can create wrapper for this classes with 2 modules,
> that
> > can
> > > be
> > > enabled by profiles for java9 and java7-8.
> > > For using internal classes that not exported by default, we will
> > need to
> > > add new args(--add-exports) when compiling (javac) *and* when
> running
> > > (java
> > > ).
> > > Is it ok?
> > >
> > >
> > > Also, there are a two different Unsafe classes in java 9 -
> > > sun.misc.Unsafe
> > > & jdk.internal.misc.Unsafe, but both of them doesn't have
> > monitorEnter &
> > > monitorExit methods, that we use in
> > > org.apache.ignite.internal.util.GridUnsafe. What should I do with
> > this
> > > case?
> > >
> > >
> > >
> > > 2017-03-21 11:21 GMT+03:00 Евгений Журавлев :
> > >
> > > Denis,
> > >
> > > I found some recommendations on openjdk wiki:
> > > https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+
> > >>> Analysis+Tool
> > >
> > > 2017-03-20 20:44 GMT+03:00 Denis Magda :
> > >
> > > Referring to your report only the following API was removed
> > completely.
> > > The rest is still deprecated and stowed in special JDK 9 modules.
> > >
> > >
> > > "org.apache.ignite.internal.processors.hadoop.HadoopClassLoader"
> ->
> > > "sun.misc.PerfCounter (JDK internal API (JDK removed internal
> API))”;
> > >
> > >
> > > "org.apache.ignite.internal.processors.platform.memory.Pla
> > > tformMemoryPool"
> > > -> "sun.misc.Cleaner (JDK internal API (JDK removed internal
> API))”;
> > >
> > >

Re: Should we take care of Java 9 in Ignite 2.0 scope?

2017-09-26 Thread Nikita Ivanov

What about Java 8 to begin with?

--
Nikita Ivanov


On Tue, Sep 26, 2017 at 3:16 PM, Denis Magda  wrote:

> Eventually now it’s time to put all the doubts aside on when Ignite should
> support JDK 9. The JDK is production ready and was released not long time
> ago.
>
> Is there anyone interested to enable JDK 9 for Ignite?
>
> —
> Denis
>
> > On Mar 24, 2017, at 10:22 AM, Denis Magda  wrote:
> >
> > Evgenii,
> >
> > Thanks for the investigation. Could you update the ticket with all of
> your findings and suggestions?
> >
> > If you feel you can’t fix it by 2.0 then let’s do it in the next minor
> release like 2.x. I don’t think that we introduce any breaking
> compatibilities when JDK 9 support is added later. Do you think so as well?
> >
> > —
> > Denis
> >
> >> On Mar 24, 2017, at 9:24 AM, Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com> wrote:
> >>
> >> Igniters,
> >>
> >> I found that maven-war-plugin, that used in ignite-websphere-test
> doesn't
> >> work with java 9. Here is issue:
> >> https://issues.apache.org/jira/browse/MWAR-397.They have workaround
> with
> >> setting MAVEN_OPTS, but this workaround doesn't fit us.
> >>
> >> 2017-03-24 0:51 GMT+03:00 zixu mo :
> >>
> >>>
> >>>
> >>> JB IDEA2017.1 released, support jdk 9.
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Mar 24, 2017 at 12:14 AM +0800, "Anton Vinogradov" <
> >>> avinogra...@gridgain.com> wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Denis,
> >>>
> >>> I'm not sure, but javac usage seems to be not our but IDE "problem".
> >>> No one call javac directly, AFAIK.
> >>>
> >>> So, any --add-exports can be added to wrapping maven, .bat/.sh or IDE
> >>> settings.
> >>> I'm pretty sure JB's IDEA will handle this right after Java9 release :)
> >>>
> >>> On Wed, Mar 22, 2017 at 12:19 PM, Anton Vinogradov  wrote:
> >>>
>  Denis,
> 
>  I'm not sure, but javac usage seems to be not our but IDE "problem".
>  No one call javac directly, AFAIK.
> 
>  So, any --add-exports can be added to wrapping maven, .bat/.sh or IDE
>  settings.
>  I'm pretty sure JB's IDEA will handle this right after Java9 release
> :)
> 
>  On Wed, Mar 22, 2017 at 1:43 AM, Denis Magda  wrote:
> 
> > For using internal classes that not exported by default, we will
> need to
> > add new args(--add-exports) when compiling (javac) *and* when running
> > (java
> > ).
> > Is it ok?
> >
> >
> > Anton, are you fine with this approach?
> >
> > —
> > Denis
> >
> > On Mar 21, 2017, at 11:05 AM, Evgenii Zhuravlev
> > wrote:
> >
> > Denis,
> >
> > I've found that some internal classes like sun.misc.SharedSecrets,
> > sun.misc.URLClassPath, sun.misc.PerfCounter, sun.misc.Cleaner changed
> > their
> > packages. I can create wrapper for this classes with 2 modules, that
> can
> > be
> > enabled by profiles for java9 and java7-8.
> > For using internal classes that not exported by default, we will
> need to
> > add new args(--add-exports) when compiling (javac) *and* when running
> > (java
> > ).
> > Is it ok?
> >
> >
> > Also, there are a two different Unsafe classes in java 9 -
> > sun.misc.Unsafe
> > & jdk.internal.misc.Unsafe, but both of them doesn't have
> monitorEnter &
> > monitorExit methods, that we use in
> > org.apache.ignite.internal.util.GridUnsafe. What should I do with
> this
> > case?
> >
> >
> >
> > 2017-03-21 11:21 GMT+03:00 Евгений Журавлев :
> >
> > Denis,
> >
> > I found some recommendations on openjdk wiki:
> > https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+
> >>> Analysis+Tool
> >
> > 2017-03-20 20:44 GMT+03:00 Denis Magda :
> >
> > Referring to your report only the following API was removed
> completely.
> > The rest is still deprecated and stowed in special JDK 9 modules.
> >
> >
> > "org.apache.ignite.internal.processors.hadoop.HadoopClassLoader" ->
> > "sun.misc.PerfCounter (JDK internal API (JDK removed internal API))”;
> >
> >
> > "org.apache.ignite.internal.processors.platform.memory.Pla
> > tformMemoryPool"
> > -> "sun.misc.Cleaner (JDK internal API (JDK removed internal API))”;
> >
> > "org.apache.ignite.internal.util.nio.GridNioServer$Abstrac
> > tNioClientWorker"
> > -> "sun.misc.Cleaner (JDK internal API (JDK removed internal API))";
> >
> >
> > "org.apache.ignite.internal.processors.rest.protocols.GridRe
> > stProtocolAdapter"
> > -> "sun.misc.BASE64Encoder (JDK internal API (JDK removed internal
> >>> API))”;
> >
> > "org.apache.ignite.internal.util.IgniteUtils"  ->
> > "sun.misc.JavaNetAccess (JDK internal API (JDK removed internal
> API))”;
> >
> >
> > "org.apache.ignite.internal.util.IgniteUtils"  ->
> >

Re: Integration of Spark and Ignite. Prototype.

2017-09-26 Thread Valentin Kulichenko

I will review in the next few days.

-Val

On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda  wrote:

> Hello Nikolay,
>
> This is good news. Finally this capability is coming to Ignite.
>
> Val, Vladimir, could you do a preliminary review?
>
> Answering on your questions.
>
> 1. Yardstick should be enough for performance measurements. As a Spark
> user, I will be curious to know what’s the point of this integration.
> Probably we need to compare Spark + Ignite and Spark + Hive or Spark +
> RDBMS cases.
>
> 2. If Spark community is reluctant let’s include the module in
> ignite-spark integration.
>
> —
> Denis
>
> > On Sep 25, 2017, at 11:14 AM, Николай Ижиков 
> wrote:
> >
> > Hello, guys.
> >
> > Currently, I’m working on integration between Spark and Ignite [1].
> >
> > For now, I implement following:
> >* Ignite DataSource implementation(IgniteRelationProvider)
> >* DataFrame support for Ignite SQL table.
> >* IgniteCatalog implementation for a transparent resolving of ignites
> > SQL tables.
> >
> > Implementation of it can be found in PR [2]
> > It would be great if someone provides feedback for a prototype.
> >
> > I made some examples in PR so you can see how API suppose to be used [3].
> > [4].
> >
> > I need some advice. Can you help me?
> >
> > 1. How should this PR be tested?
> >
> > Of course, I need to provide some unit tests. But what about scalability
> > tests, etc.
> > Maybe we need some Yardstick benchmark or similar?
> > What are your thoughts?
> > Which scenarios should I consider in the first place?
> >
> > 2. Should we provide Spark Catalog implementation inside Ignite codebase?
> >
> > A current implementation of Spark Catalog based on *internal Spark API*.
> > Spark community seems not interested in making Catalog API public or
> > including Ignite Catalog in Spark code base [5], [6].
> >
> > *Should we include Spark internal API implementation inside Ignite code
> > base?*
> >
> > Or should we consider to include Catalog implementation in some external
> > module?
> > That will be created and released outside Ignite?(we still can support
> and
> > develop it inside Ignite community).
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-3084
> > [2] https://github.com/apache/ignite/pull/2742
> > [3] https://github.com/apache/ignite/pull/2742/files#diff-
> > f4ff509cef3018e221394474775e0905
> > [4] https://github.com/apache/ignite/pull/2742/files#diff-
> > f2b670497d81e780dfd5098c5dd8a89c
> > [5] http://apache-spark-developers-list.1001551.n3.
> > nabble.com/Spark-Core-Custom-Catalog-Integration-between-
> > Apache-Ignite-and-Apache-Spark-td22452.html
> > [6] https://issues.apache.org/jira/browse/SPARK-17767
> >
> > --
> > Nikolay Izhikov
> > nizhikov@gmail.com
>
>

Re: Should we take care of Java 9 in Ignite 2.0 scope?

2017-09-26 Thread Denis Magda

Eventually now it’s time to put all the doubts aside on when Ignite should 
support JDK 9. The JDK is production ready and was released not long time ago.

Is there anyone interested to enable JDK 9 for Ignite?

—
Denis

> On Mar 24, 2017, at 10:22 AM, Denis Magda  wrote:
> 
> Evgenii,
> 
> Thanks for the investigation. Could you update the ticket with all of your 
> findings and suggestions?
> 
> If you feel you can’t fix it by 2.0 then let’s do it in the next minor 
> release like 2.x. I don’t think that we introduce any breaking 
> compatibilities when JDK 9 support is added later. Do you think so as well?
> 
> —
> Denis
> 
>> On Mar 24, 2017, at 9:24 AM, Evgenii Zhuravlev  
>> wrote:
>> 
>> Igniters,
>> 
>> I found that maven-war-plugin, that used in ignite-websphere-test doesn't
>> work with java 9. Here is issue:
>> https://issues.apache.org/jira/browse/MWAR-397.They have workaround with
>> setting MAVEN_OPTS, but this workaround doesn't fit us.
>> 
>> 2017-03-24 0:51 GMT+03:00 zixu mo :
>> 
>>> 
>>> 
>>> JB IDEA2017.1 released, support jdk 9.
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Mar 24, 2017 at 12:14 AM +0800, "Anton Vinogradov" <
>>> avinogra...@gridgain.com> wrote:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Denis,
>>> 
>>> I'm not sure, but javac usage seems to be not our but IDE "problem".
>>> No one call javac directly, AFAIK.
>>> 
>>> So, any --add-exports can be added to wrapping maven, .bat/.sh or IDE
>>> settings.
>>> I'm pretty sure JB's IDEA will handle this right after Java9 release :)
>>> 
>>> On Wed, Mar 22, 2017 at 12:19 PM, Anton Vinogradov  wrote:
>>> 
 Denis,
 
 I'm not sure, but javac usage seems to be not our but IDE "problem".
 No one call javac directly, AFAIK.
 
 So, any --add-exports can be added to wrapping maven, .bat/.sh or IDE
 settings.
 I'm pretty sure JB's IDEA will handle this right after Java9 release :)
 
 On Wed, Mar 22, 2017 at 1:43 AM, Denis Magda  wrote:
 
> For using internal classes that not exported by default, we will need to
> add new args(--add-exports) when compiling (javac) *and* when running
> (java
> ).
> Is it ok?
> 
> 
> Anton, are you fine with this approach?
> 
> —
> Denis
> 
> On Mar 21, 2017, at 11:05 AM, Evgenii Zhuravlev
> wrote:
> 
> Denis,
> 
> I've found that some internal classes like sun.misc.SharedSecrets,
> sun.misc.URLClassPath, sun.misc.PerfCounter, sun.misc.Cleaner changed
> their
> packages. I can create wrapper for this classes with 2 modules, that can
> be
> enabled by profiles for java9 and java7-8.
> For using internal classes that not exported by default, we will need to
> add new args(--add-exports) when compiling (javac) *and* when running
> (java
> ).
> Is it ok?
> 
> 
> Also, there are a two different Unsafe classes in java 9 -
> sun.misc.Unsafe
> & jdk.internal.misc.Unsafe, but both of them doesn't have monitorEnter &
> monitorExit methods, that we use in
> org.apache.ignite.internal.util.GridUnsafe. What should I do with this
> case?
> 
> 
> 
> 2017-03-21 11:21 GMT+03:00 Евгений Журавлев :
> 
> Denis,
> 
> I found some recommendations on openjdk wiki:
> https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+
>>> Analysis+Tool
> 
> 2017-03-20 20:44 GMT+03:00 Denis Magda :
> 
> Referring to your report only the following API was removed completely.
> The rest is still deprecated and stowed in special JDK 9 modules.
> 
> 
> "org.apache.ignite.internal.processors.hadoop.HadoopClassLoader" ->
> "sun.misc.PerfCounter (JDK internal API (JDK removed internal API))”;
> 
> 
> "org.apache.ignite.internal.processors.platform.memory.Pla
> tformMemoryPool"
> -> "sun.misc.Cleaner (JDK internal API (JDK removed internal API))”;
> 
> "org.apache.ignite.internal.util.nio.GridNioServer$Abstrac
> tNioClientWorker"
> -> "sun.misc.Cleaner (JDK internal API (JDK removed internal API))";
> 
> 
> "org.apache.ignite.internal.processors.rest.protocols.GridRe
> stProtocolAdapter"
> -> "sun.misc.BASE64Encoder (JDK internal API (JDK removed internal
>>> API))”;
> 
> "org.apache.ignite.internal.util.IgniteUtils"  ->
> "sun.misc.JavaNetAccess (JDK internal API (JDK removed internal API))”;
> 
> 
> "org.apache.ignite.internal.util.IgniteUtils"  ->
> "sun.misc.SharedSecrets (JDK internal API (JDK removed internal API))”;
> 
> 
> "org.apache.ignite.internal.util.IgniteUtils"  ->
> "sun.misc.URLClassPath (JDK internal API (JDK removed internal API))”;
> 
> 
> 
> *Evgeniy*, does Oracle officially suggest replacements for deleted APIs?
> Probably it’s a matter of day to do a switch.
> 
>

Re: An issue of Ignite In-Menory Sql Grid since version 2.0.0

2017-09-26 Thread Denis Magda

Denis Mek.,

How do you suggest executing the same query without SQL? Please be specific if 
you advise discontinuing the SQL for the use case.


Alex P., Vladimir,

What’s the main reason of WITH RECURSIVE removal? Why it worked before and 
broken now?
https://issues.apache.org/jira/browse/IGNITE-5289 

  
—
Denis

> On Sep 25, 2017, at 12:31 AM, Denis Mekhanikov  wrote:
> 
> Yes, you are right about that, but Ignite has no intention to support this 
> syntax, it worked in 1.9 by chance.
> Here, I found a ticket to explicitly forbid it: 
> https://issues.apache.org/jira/browse/IGNITE-5289 
> 
> 
> So, it's better to use cache directly without SQL in your case.
> 
> Denis
> 
> пн, 25 сент. 2017 г. в 4:11, 贺波 >:
> Hi!
> I have executed the  recursive sql statement on H2 database,it executes 
> correctly.But it executes error since version 2.0.0.So  I 
> think it's a bug of Ingite,not H2.My test demo is in the attachment.
> 
> 
> 
> 
> 
> 
> At 2017-09-23 00:20:12, "Denis Mekhanikov"  > wrote:
> Hi!
> 
> Internally Ignite uses H2 to process SQL queries. Recursive queries is an 
> experimental feature of H2, so I wouldn't recommend you to use it in 
> production for now. 
> Ignite 2.0 and 2.1 don't seem to support this kind of queries, so the best 
> option for you is to modify your query if possible to avoid recursive 
> constructs, or retrieve data from cache directly, without use of SQL.
> 
> Denis
> 
> пт, 22 сент. 2017 г. в 12:37, 贺波 >:
> Hi,I used Apache Ignite in my project for more than a year,from version 1.8.0 
> to 2.2.0.I use Ignite In-Menory Sql Grid in my project.I use “with as” sql 
> function in my sql,it executes correctly in version 1.9.0,but executes error 
> since version 2.0.0.My  sql statement is:
>   with RECURSIVE children(typeId, pTypeId) AS ( 
>   SELECT typeId, pTypeId FROM ProcessDefTypePo WHERE pTypeId = '1'
>   UNION ALL 
>   SELECT ProcessDefTypePo.typeId, ProcessDefTypePo.pTypeId FROM 
> children INNER JOIN ProcessDefTypePo ON children.typeId 
> =ProcessDefTypePo.pTypeId 
>   )
>select t1.typeId,t1.pTypeId,t1.typeName,t1.description, t2.typeName as 
> pTypeName from ProcessDefTypePo t1 left join ProcessDefTypePo t2 on 
> t1.pTypeId=t2.typeId where t1.typeId not in ( select typeId from children )
> 
>The  execution error in version 2.2.0 is:
> Caused by: class org.apache.ignite.IgniteCheckedException: Unknown query 
> type: null
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2316)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:1820)
>   ... 165 more
> Caused by: java.lang.UnsupportedOperationException: Unknown query type: null
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseQuery(GridSqlQueryParser.java:1225)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseTable(GridSqlQueryParser.java:501)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseTableFilter(GridSqlQueryParser.java:465)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseSelect(GridSqlQueryParser.java:565)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseQuery(GridSqlQueryParser.java:1220)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseQueryExpression(GridSqlQueryParser.java:452)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseExpression0(GridSqlQueryParser.java:1436)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseExpression(GridSqlQueryParser.java:1267)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseExpression0(GridSqlQueryParser.java:1378)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseExpression(GridSqlQueryParser.java:1267)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseSelect(GridSqlQueryParser.java:536)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parseQuery(GridSqlQueryParser.java:1220)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parse(GridSqlQueryParser.java:1181)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.parse(GridSqlQuerySplitter.java:1604)
>   at 
>

Re: Contributors Permissions for Ignite

2017-09-26 Thread Denis Magda

Hi, done!

Welcome to the Ignite community!

Get familiar with Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

Once you got familiar and were able to run a few examples, you should pick
a Jira ticket you would like to start on. Send an email to the dev list sharing 
your JIRA id, so we can add you as a contributor in Jira.

These are the easy tickets to start with:
https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN

While those are more advanced but appealing:
https://ignite.apache.org/community/contribute.html#pick-ticket

Looking forward to your contributions!

Denis

> On Sep 26, 2017, at 2:52 AM, yashasvi kotamraju  
> wrote:
> 
> Hi
> 
> I would like to request for Contributors Permissions for Ignite Development.
> 
> JIRA Username : kotamrajuyashasvi
> 
> Presently I would like to assign myself to an Issue which I raised and fix
> it.
> http://apache-ignite-users.70518.x6.nabble.com/POJO-field-having-wrapper-type-mapped-to-cassandra-table-are-getting-initialized-to-respective-defau-td16135.html
> 
> JIRA Ticket:
> https://issues.apache.org/jira/browse/IGNITE-6500
> 
> I would like to contribute to Ignite in future as well as I would be
> working on it for considerable amount of time.
> 
> regards
> Kotamraju Yashasvi

Re: Integration of Spark and Ignite. Prototype.

2017-09-26 Thread Denis Magda

Hello Nikolay,

This is good news. Finally this capability is coming to Ignite. 

Val, Vladimir, could you do a preliminary review?

Answering on your questions.

1. Yardstick should be enough for performance measurements. As a Spark user, I 
will be curious to know what’s the point of this integration. Probably we need 
to compare Spark + Ignite and Spark + Hive or Spark + RDBMS cases.

2. If Spark community is reluctant let’s include the module in ignite-spark 
integration.

—
Denis
 
> On Sep 25, 2017, at 11:14 AM, Николай Ижиков  wrote:
> 
> Hello, guys.
> 
> Currently, I’m working on integration between Spark and Ignite [1].
> 
> For now, I implement following:
>* Ignite DataSource implementation(IgniteRelationProvider)
>* DataFrame support for Ignite SQL table.
>* IgniteCatalog implementation for a transparent resolving of ignites
> SQL tables.
> 
> Implementation of it can be found in PR [2]
> It would be great if someone provides feedback for a prototype.
> 
> I made some examples in PR so you can see how API suppose to be used [3].
> [4].
> 
> I need some advice. Can you help me?
> 
> 1. How should this PR be tested?
> 
> Of course, I need to provide some unit tests. But what about scalability
> tests, etc.
> Maybe we need some Yardstick benchmark or similar?
> What are your thoughts?
> Which scenarios should I consider in the first place?
> 
> 2. Should we provide Spark Catalog implementation inside Ignite codebase?
> 
> A current implementation of Spark Catalog based on *internal Spark API*.
> Spark community seems not interested in making Catalog API public or
> including Ignite Catalog in Spark code base [5], [6].
> 
> *Should we include Spark internal API implementation inside Ignite code
> base?*
> 
> Or should we consider to include Catalog implementation in some external
> module?
> That will be created and released outside Ignite?(we still can support and
> develop it inside Ignite community).
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://github.com/apache/ignite/pull/2742
> [3] https://github.com/apache/ignite/pull/2742/files#diff-
> f4ff509cef3018e221394474775e0905
> [4] https://github.com/apache/ignite/pull/2742/files#diff-
> f2b670497d81e780dfd5098c5dd8a89c
> [5] http://apache-spark-developers-list.1001551.n3.
> nabble.com/Spark-Core-Custom-Catalog-Integration-between-
> Apache-Ignite-and-Apache-Spark-td22452.html
> [6] https://issues.apache.org/jira/browse/SPARK-17767
> 
> --
> Nikolay Izhikov
> nizhikov@gmail.com

[GitHub] ignite pull request #2759: exception when trying to check threadId for remot...

2017-09-26 Thread voipp

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2759

exception when trying to check threadId for remote



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2759.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2759


commit 3f01e4c0155105fec2680a3fa80228a33c26712e
Author: voipp 
Date:   2017-09-26T12:08:11Z

exception when trying to check threadId for remote




---

[jira] [Created] (IGNITE-6507) Commit can be lost in network split scenario

2017-09-26 Thread Alexei Scherbakov (JIRA)

Alexei Scherbakov created IGNITE-6507:
-

 Summary: Commit can be lost in network split scenario
 Key: IGNITE-6507
 URL: https://issues.apache.org/jira/browse/IGNITE-6507
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.1
Reporter: Alexei Scherbakov
Priority: Critical
 Fix For: 2.3


Commit can be lost in network split scenario

{noformat}
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.ignite.internal.processors.cache.distributed.dht;

import org.apache.ignite.IgniteCache;
import org.apache.ignite.cache.affinity.Affinity;
import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
import org.apache.ignite.configuration.BinaryConfiguration;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.configuration.MemoryConfiguration;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.IgniteInternalFuture;
import org.apache.ignite.internal.TestRecordingCommunicationSpi;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinder;
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;

import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
import static org.apache.ignite.cache.CacheWriteSynchronizationMode.FULL_SYNC;

/**
 * Tests commit consitency in split-brain scenario.
 */
public class GridCacheGridSplitTxConsistencyTest extends GridCommonAbstractTest 
{
/** */
private static final TcpDiscoveryIpFinder IP_FINDER = new 
TcpDiscoveryVmIpFinder(true);

/**
 * {@inheritDoc}
 */
@Override protected void afterTest() throws Exception {
super.afterTest();

stopAllGrids();

GridTestUtils.deleteDbFiles();
}

/**
 * {@inheritDoc}
 */
@Override protected IgniteConfiguration getConfiguration(String gridName) 
throws Exception {
IgniteConfiguration cfg = super.getConfiguration(gridName);

cfg.setCommunicationSpi(new TestRecordingCommunicationSpi());

cfg.setConsistentId(gridName);

MemoryConfiguration memCfg = new MemoryConfiguration();
memCfg.setPageSize(1024);
memCfg.setDefaultMemoryPolicySize(100 * 1024 * 1024);

cfg.setMemoryConfiguration(memCfg);

((TcpDiscoverySpi) cfg.getDiscoverySpi()).setIpFinder(IP_FINDER);

CacheConfiguration ccfg = new CacheConfiguration();
ccfg.setName(DEFAULT_CACHE_NAME);
ccfg.setAtomicityMode(TRANSACTIONAL);
ccfg.setWriteSynchronizationMode(FULL_SYNC);
ccfg.setAffinity(new RendezvousAffinityFunction(false, 3));
ccfg.setBackups(2);

cfg.setCacheConfiguration(ccfg);

return cfg;
}

/**
 * Tests if commits are working as expected.
 * @throws Exception
 */
public void testSplitTxConsistency() throws Exception {
IgniteEx grid0 = startGrid(0);
grid0.active(true);

IgniteEx grid1 = startGrid(1);
IgniteEx grid2 = startGrid(2);

int key = 0;

Affinity aff = grid0.affinity(DEFAULT_CACHE_NAME);
assertTrue(aff.isPrimary(grid0.localNode(), key));
assertTrue(aff.isBackup(grid1.localNode(), key));
assertTrue(aff.isBackup(grid2.localNode(), key));

final TestRecordingCommunicationSpi spi0 = 
(TestRecordingCommunicationSpi) grid0.configuration().getCommunicationSpi();

spi0.blockMessages(GridDhtTxFinishRequest.class, grid1.name());
spi0.blockMessages(GridDhtTxFinishRequest.class, grid2.name());

IgniteInternalFuture fut = multithreadedAsync(new Runnable() {
@Override public void run() {
try {
spi0.waitForBlocked();

} catch (InterruptedException e) {
fail();
}

stopGrid(1);

[GitHub] ignite pull request #2758: IGNITE-6440 Test fx

2017-09-26 Thread alexpaschenko

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/2758

IGNITE-6440 Test fx



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6440

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2758


commit e56d099d4cd13bdf800f10bbe50f0d1e3c53b891
Author: Alexander Paschenko 
Date:   2017-09-26T19:27:09Z

IGNITE-6440 Test fx




---

[GitHub] ignite pull request #2757: IGNITE-6346 Fix: distributed set does not work in...

2017-09-26 Thread vk23

GitHub user vk23 opened a pull request:

https://github.com/apache/ignite/pull/2757

IGNITE-6346 Fix: distributed set does not work in REPLICATED mode

Changes:
1. Fixed bug
2. Added tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vk23/ignite ignite-6346

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2757


commit 56d173378de373fe65a26193c5b150b73f0ffeae
Author: vk 
Date:   2017-09-26T19:15:04Z

IGNITE-6346 Fix: distributed set does not work in REPLICATED mode

Changes:
1. Fixed bug
2. Added tests




---

Re: [DISCUSS] Ignite Update Checker

2017-09-26 Thread Dmitriy Setrakyan

On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov 
wrote:

> Folks,
>
> Can we add version of current node to web request? This way we will better
> understand version distribution, what might help us with certain API
> update/deprecate decisions
> E.g. http://ignite.apache.org/latest.cgi=2.2.0


Vladimir, I personally do not see a problem with that, as sending the
current version to the update checker seems very innocent to me. At the
same time, it will allow us to examine the usage of each release and make
decisions about dropping backward compatibility or spotting bugs for a
certain release.

Cos, Raul, any thoughts?


>
>
> Vladimir.
>
> On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan 
> wrote:
>
> > I think it is safe to assume at this point that everyone is in general
> > agreement, since there are no active objections.
> >
> > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > https://issues.apache.org/jira/browse/IGNITE-6305
> >
> > D.
> >
> > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > >
> > >
> > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani 
> > > wrote:
> > >
> > >> Yeah, I guess that's doable as well and requires less management
> effort
> > >> than my suggestion. We could use events [1] to store payload data
> (e.g.
> > >> IP,
> > >> version, etc.)
> > >
> > >
> > > Yes, we could use events or some other similar API provided by GA.
> > >
> > >
> > >> What the download page CGI developed in? PHP?
> > >>
> > >
> > > To be honest, no clue. I guess someone in the community can figure it
> > out:
> > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >
> > >
> > >> However, I'm not sure whether storing this data in a 3rd party
> (Google)
> > is
> > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> > >> doubt
> > >> in the PMC, it's better to ask legal@.
> > >
> > >
> > > I am not sure there is anything to ask about. The whole Ignite website
> is
> > > GA enabled, and all we are doing is accessing a standard web page from
> > the
> > > Ignite web site. The information gathered from GA is available only to
> > the
> > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > question.
> > >
> > > Even ASF website itself uses GA: https://www.apache.org/
> > > foundation/policies/privacy.html
> > >
> > >
> > >> I think Cos said it's OK; maybe Roman can pitch in.
> > >>
> > >
> > >  Sure, would be nice to hear from Roman as well.
> > >
> > >
> > >> Cheers.
> > >>
> > >> [1]
> > >> https://developers.google.com/analytics/devguides/collection
> > >> /analyticsjs/events
> > >>
> > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >> dsetrak...@apache.org>
> > >> wrote:
> > >>
> > >> > Raul,
> > >> >
> > >> > Could point about Javascript, it will not work because it executes
> in
> > >> the
> > >> > browser. This means we need a server-side script, like CGI we are
> > using
> > >> on
> > >> > our download page.
> > >> >
> > >> > How about this approach. We create something like ignite-version.cgi
> > >> script
> > >> > which will invoke a call to GA and then return the latest version.
> > >> >
> > >> > This should work, right?
> > >> >
> > >> > D.
> > >> >
> > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> raul@evosent.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Hey Dmitriy and all
> > >> > >
> > >> > > Also, since we have GA enabled for the website, we can track how
> > many
> > >> > times
> > >> > > > this page was accessed, which will be equal to the number of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by the
> > Ignite
> > >> > PMC.
> > >> > >
> > >> > >
> > >> > > Unfortunately this won't work because GA is loaded via Javascript
> on
> > >> the
> > >> > > browser, so Google will never receive the page hit.
> > >> > >
> > >> > > Given the constraints, the most viable solution is an HTTPS
> endpoint
> > >> > > running on ASF infra that Ignite invokes via a GET or POST
> request.
> > >> The
> > >> > > simplest thing is to write each request in a log file, along with
> > the
> > >> > > timestamp, the version reported by the client, maybe the IP (not
> > sure
> > >> > about
> > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> you
> > >> make
> > >> > it
> > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > creates
> > >> on
> > >> > > first startup or something.
> > >> > >
> > >> > > This endpoint would need some basic DDoS protection and
> blacklisting
> > >> to
> > >> > > prevent data spoofing.
> > >> > >
> > >> > > If we'll be implementing this endpoint anyway, then there's no
> point
> > >> > > placing another file on Git or elsewhere for reporting the latest
> > >> > versions:
> > >> > > the endpoint itself can return them.
> > >> > >
> > >> > > WDYT?
> > >> > >
> > >> >

Re: Persistence per memory policy configuration

2017-09-26 Thread Dmitriy Setrakyan

On Tue, Sep 26, 2017 at 7:33 AM, Dmitry Pavlov 
wrote:

> Hi Dmitriy, thank you for reply. Do you agree Memory Policy already became
> Ignite's term? We call this configuration now MemoryPolicy(Configuration),
> can we call new configuration elments by their existings name? We can avoid
> introduction of second Ignite's term in that case.
>

The refactoring is about merging memory and persistence configuration under
the same umbrella. The term "MemoryPolicy" does not make sense anymore,
given that it now also includes persistent configuration as well.


>
> вт, 26 сент. 2017 г. в 17:27, Dmitriy Setrakyan :
>
> > Dmitriy, we are not renaming classes, we are refactoring them. Prior to
> > this design, it was impossible to set persistence configuration on
> > per-cache basis. With this new design, users will be able to configure
> some
> > caches to be in-memory only and others to be on disk.
> >
> > Given that we are already refactoring, it only makes sense to pick
> better,
> > more appropriate names.
> >
> > D.
> >
> > On Tue, Sep 26, 2017 at 7:20 AM, Dmitry Pavlov 
> > wrote:
> >
> > > Vladimir, it is not clear for me, why we need to rename existing
> > > configuration classes. Could you explain? And if we can't get consensus
> > > now, should we pospond solution?
> > >
> > > My idea is that user needs this feature more than elegant names in
> > > configuration.
> > >
> > > Moreover once MemoryPolicyConfiguration was introduced as Ignite term
> it
> > is
> > > simpler to keep it as is, than create new terms.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 26 сент. 2017 г. в 16:59, Vladimir Ozerov :
> > >
> > > > I do not understand why we should delay with renames. Yes, it will
> > cause
> > > > questions, so we will have to put additional efforts to docs and
> > > JavaDocs.
> > > > But the earlier we do that, the better.
> > > >
> > > > On Tue, Sep 26, 2017 at 4:50 PM, Dmitry Pavlov <
> dpavlov@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Igniters, sorry for late response. I didn't catch idea of
> > renaming.
> > > > > PersistentStoreConfiguration is intuitive, and
> > > MemoryPolicyConfiguration
> > > > is
> > > > > intuitive also.
> > > > >
> > > > > If we rename these classes now, it will bring more questions to
> user
> > > > list.
> > > > > Users may be confused by old and new names and by trying to match
> it.
> > > > More
> > > > > issues can came from XML configs that users already have.
> > > > >
> > > > > Can we postpone the renaming? I suggest to finish 'persistence per
> > > memory
> > > > > policy' task without renaming, and create separate JIRA issue for
> > > > creating
> > > > > future decision?
> > > > >
> > > > > вт, 26 сент. 2017 г. в 15:25, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com
> > > > > >:
> > > > >
> > > > > > I do not like DurableMemoryConfiguration, because it's quite
> > > confusing
> > > > -
> > > > > we
> > > > > > configure in-memory caches using DurableMemory class, which
> > > immediately
> > > > > > suggests that everything will be persisted. I am not sure if this
> > is
> > > a
> > > > > > right wording choice for the documentation either. I would go
> with
> > > > > > DataStoreConfiguration and DataRegionConfiguration.
> > > > > >
> > > > > > --AG
> > > > > >
> > > > > > 2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > > > >
> > > > > > > Given that we already have a notion of CacheStore which comes
> > from
> > > > > JCache
> > > > > > > spec, I think having other stores may get confusing. I like
> > > > > > > DurableMemoryConfiguration.
> > > > > > >
> > > > > > > Other opinions?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov <
> > > > > voze...@gridgain.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dima, let's finalize the design first.
> > > > > > > >
> > > > > > > > As I understand, we are happy with idea to merge
> > > > MemoryConfiguration
> > > > > > > > and PersistentStoreConfiguration
> > > > > > > > into something what I called DataConfiguration, and to rename
> > > > > > > > MemoryPolicyConfiguration to DataRegionConfiguration.
> > > > > > > >
> > > > > > > > The only outstanding qurestion is whether DataConfiguration
> is
> > a
> > > > good
> > > > > > > name.
> > > > > > > > I am not very happy with it, so let's think of other
> > > alternatives.
> > > > > > Quick
> > > > > > > > ideas:
> > > > > > > > 1) StoreConfiguration - looks perfect to me - short and
> > > > > > self-describing,
> > > > > > > > but clashes a bit with existing CacheStore
> > > > > > > > 2) DataStoreConfiguration - same as p.1, but the word "data"
> is
> > > not
> > > > > > > > necessary IMO
> > > > > > > > 3) PageStoreConfiguration? GIves a hint to our page-based
> > > > > architecture.
> > > > > > > > 4) DurableMemoryConfiguration - aligns well with our

Re: A quick question on Ignite's B+ tree implementation

2017-09-26 Thread Denis Magda

+ dev list

Hi John,

As the one who worked on this documentation, I confirm that it’s correct.

In my opinion, the only missing thing is that an index page (node) can comprise 
key/value if the latter is of primitive type (int, float, char, String of 
specific length). *Vladimir*, *Sam*, could you confirm this point and share 
more details with John?

—
Denis

> On Sep 22, 2017, at 4:19 PM, John Wilson  wrote:
> 
> Hi,
> 
> The internal nodes of a B+ tree, by definition, store only keys while the 
> leaf nodes store (or hold pointer to) the actual data.
> 
> The documentation here, 
> https://apacheignite.readme.io/docs/memory-architecture 
> , states that each 
> index node (including internal nodes) store information to access the data 
> page and offset for the key in question (not just the leaf nodes) 
> 
> Why call it a B+ tree.
> 
> Thanks,

[jira] [Created] (IGNITE-6506) Unable to activate cluster node if it was stopped during persistent storage checkpoint

2017-09-26 Thread Joel Lang (JIRA)

Joel Lang created IGNITE-6506:
-

 Summary: Unable to activate cluster node if it was stopped during 
persistent storage checkpoint
 Key: IGNITE-6506
 URL: https://issues.apache.org/jira/browse/IGNITE-6506
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.2
Reporter: Joel Lang
Priority: Critical


I have a cluster with two nodes: A and B.

On startup, node A and B wait for each other to be connected and then node A 
will attempt to activate the cluster.

While testing high availability we find that if a node is stopped during the 
persistent store checkpoint, we cannot activate the cluster on startup without 
deleting the persistent storage directory. Specifically in the case where node 
A is stopped during checkpointing, upon the next startup it will encounter 
several exceptions during activation and then hang without completing 
activation.

Here is the log.


{noformat}
2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO  
o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.ignite.internal.exchange.time - Started exchange init 
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18, 
node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, 
addrs=[10.5.17.19, 127.0.0.1], 
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], 
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, 
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], evtNode=TcpDiscoveryNode 
[id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], 
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], 
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, 
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], 
customEvt=ChangeGlobalStateMessage 
[id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7, 
reqId=f7155dea-fede-4340-b244-7a3b65f167a8, 
initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process 
[nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, 
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory: 
/opt/mbe1/ignite/db/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work 
directory: /opt/mbe1/ignite/db/wal/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive 
directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN  
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default 
MemoryPolicy found; system default of 1GB size will be used.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory 
[memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB, 
checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=3.1 
GiB, pages=1544064, tableSize=91.0 MiB, checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start 
marker = 
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin,
 end marker = 
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state 
[lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731, 
forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, 
forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN  
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the 
middle of checkpoint. Will restore memory state and finish checkpoint on node 
start.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR 
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node 
components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, 
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
java.lang.ArrayIndexOutOfBoundsException: -1
at 
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92)
at

[jira] [Created] (IGNITE-6505) .NET: Verify output in ExamplesTest

2017-09-26 Thread Pavel Tupitsyn (JIRA)

Pavel Tupitsyn created IGNITE-6505:
--

 Summary: .NET: Verify output in ExamplesTest
 Key: IGNITE-6505
 URL: https://issues.apache.org/jira/browse/IGNITE-6505
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
Priority: Minor


{{ExamplesTest}} currently verifies that examples run without exceptions.

Add console output verification to make sure that results are adequate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Default CacheConfiguration instance can't be serialized

2017-09-26 Thread Николай Ижиков

Hello, Guys.

I found that CacheConfiguration which implements Serializable contains not
serializable fields by default.

CacheConfiguration -> affMapper -> CacheDefaultBinaryAffinityKeyMapper ->
proc -> CacheObjectBinaryProcessorImpl.

Is this a bug or I miss something?

Simple reproducer:

```
public class SimpleSerializableTet extends GridCommonAbstractTest {
public void testRemoteFilterFactoryClient() throws Exception {
Ignite g = startGrid(0);

IgniteCache cache =
g.getOrCreateCache("TEST_CACHE");
CacheConfiguration ccfg =
cache.getConfiguration(CacheConfiguration.class);

assertTrue(ccfg instanceof Serializable);
assertTrue(ccfg.getAffinityMapper() instanceof Serializable);

try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(ccfg);
out.flush();
}
}
}

```
Exception is:

```
java.io.NotSerializableException:
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
```



-- 
Nikolay Izhikov
nizhikov@gmail.com

[jira] [Created] (IGNITE-6504) Very quick checkpoint can cause AssertionError on next start from LFS

2017-09-26 Thread Ivan Rakov (JIRA)

Ivan Rakov created IGNITE-6504:
--

 Summary: Very quick checkpoint can cause AssertionError on next 
start from LFS
 Key: IGNITE-6504
 URL: https://issues.apache.org/jira/browse/IGNITE-6504
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.1
Reporter: Ivan Rakov
Assignee: Ivan Rakov
 Fix For: 2.4


Checkpoint markers are compared using their timestamps. If checkpoint took less 
than 1 millisecond, two subsequent markers will have same timestamp, which will 
lead to error:
{noformat}
java.lang.AssertionError: 
o1=/data/teamcity/tmpfs/work/db/127_0_0_1_47503/cp/1506338145591-c4f23411-e1b1-4468-856a-4419003bba93-END.bin,
 
o2=/data/teamcity/tmpfs/work/db/127_0_0_1_47503/cp/1506338145591-f76c023b-9982-40d7-a1eb-855a33b710f2-END.bin
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$4.compare(GridCacheDatabaseSharedManager.java:216)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$4.compare(GridCacheDatabaseSharedManager.java:195)
at java.util.TimSort.binarySort(TimSort.java:265)
at java.util.TimSort.sort(TimSort.java:208)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$CheckpointHistory.loadHistory(GridCacheDatabaseSharedManager.java:2704)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$CheckpointHistory.access$2600(GridCacheDatabaseSharedManager.java:2685)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1468)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:562)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:722)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:613)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2289)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Warning if no table for BinaryObject

2017-09-26 Thread Denis Magda

Mikhail,

Could you prepare a properly described ticket encompassing the suggested 
warning? Let’s do the improvement for the upcoming release.

—
Denis

> On Sep 26, 2017, at 7:23 AM, Mikhail Cherkasov  
> wrote:
> 
> Hi Dmitry,
> 
> The problem is that we don't have strong typing when defining QueryEntriy
> and a user can use an arbitrary string id to
> define a type, but he should use the same string id to obtain binary object
> builder, however, people sometimes confusing this.
> So the user can define QueryEntity with value type:
> queryEntity.setValueType("MyCoolName") and
> later put to cache the following binary object:
> ignite.binary.toBinary(value), but this object won't be indexed, because
> ignite.binary.toBinary uses class name as string id while indexing expects
> to find "MyCoolName" as id.
> 
> There's no bug in ignite about this, but I think the warning can make
> users' life easy and
> make ignite configuration more friendly to new users.
> 
> Thanks,
> Mikhail.
> 
> 
> 
> 
> On Tue, Sep 26, 2017 at 8:15 AM, Yakov Zhdanov  wrote:
> 
>> Pavel, we can optionally enforce that, but not by default.
>> 
>> Dmitry, the problem is the following. User configures Ignite cache and
>> query entity, does several puts than empty results for SQL. The reason is
>> very simple - typo in class name or simply different classes in
>> configuration and of actual cache entry. The fix is pretty simple and make
>> product more usable.
>> 
>> --Yakov
>> 
> 
> 
> 
> -- 
> Thanks,
> Mikhail.

[GitHub] ignite pull request #2756: ignite-gg-pitr-2.1.5 merge to PITR (master)

2017-09-26 Thread DmitriyGovorukhin

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/2756

ignite-gg-pitr-2.1.5 merge to PITR (master)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite PITR

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2756.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2756


commit 0b0556528098f2cb326c4d5c988f4cccd6dbe164
Author: Dmitriy Govorukhin 
Date:   2017-09-26T14:25:31Z

ignite-gg-pitr-2.1.5 merge to PITR (master)




---

Per-entry overhead - some questions

2017-09-26 Thread endianignite

Hi Igniters,

I have been looking in to the per-entry overhead for Ignite cache entries
and have a couple of questions.

1. Does each GridCacheMapEntry require a reference to GridCacheContext
or could this field be removed/optimized in some way? Is the
GridCacheContext instance the same for every entry in a particular cache?

2. Is the GridCacheMapEntry extras field (a reference to
GridCacheEntryExtras) often/usually null? If so, could we find a clever way
to remove this field and save a further 8 bytes?

The reason for asking is that I am looking at a use-case where object sizes
are very small, but the number of objects is very large, and therefore the
overhead is substantial.  In this particular case, the overhead is 70-75% of
the consumed memory.

I look forward to hearing your thoughts.

Regards
Mike



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #2755: IGNITE-5730 .NET: Fix ignite.jni.dll temp dir rac...

2017-09-26 Thread ptupitsyn

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/2755

IGNITE-5730 .NET: Fix ignite.jni.dll temp dir race



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-5730-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2755.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2755


commit f184776f8ee89fa365db08569714eb2e5b04ec63
Author: Pavel Tupitsyn 
Date:   2017-09-26T14:49:19Z

IGNITE-5730 .NET: Fix ignite.jni.dll temp dir race




---

Re: Persistence per memory policy configuration

2017-09-26 Thread Dmitry Pavlov

Hi Dmitriy, thank you for reply. Do you agree Memory Policy already became
Ignite's term? We call this configuration now MemoryPolicy(Configuration),
can we call new configuration elments by their existings name? We can avoid
introduction of second Ignite's term in that case.

вт, 26 сент. 2017 г. в 17:27, Dmitriy Setrakyan :

> Dmitriy, we are not renaming classes, we are refactoring them. Prior to
> this design, it was impossible to set persistence configuration on
> per-cache basis. With this new design, users will be able to configure some
> caches to be in-memory only and others to be on disk.
>
> Given that we are already refactoring, it only makes sense to pick better,
> more appropriate names.
>
> D.
>
> On Tue, Sep 26, 2017 at 7:20 AM, Dmitry Pavlov 
> wrote:
>
> > Vladimir, it is not clear for me, why we need to rename existing
> > configuration classes. Could you explain? And if we can't get consensus
> > now, should we pospond solution?
> >
> > My idea is that user needs this feature more than elegant names in
> > configuration.
> >
> > Moreover once MemoryPolicyConfiguration was introduced as Ignite term it
> is
> > simpler to keep it as is, than create new terms.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 26 сент. 2017 г. в 16:59, Vladimir Ozerov :
> >
> > > I do not understand why we should delay with renames. Yes, it will
> cause
> > > questions, so we will have to put additional efforts to docs and
> > JavaDocs.
> > > But the earlier we do that, the better.
> > >
> > > On Tue, Sep 26, 2017 at 4:50 PM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > Hi Igniters, sorry for late response. I didn't catch idea of
> renaming.
> > > > PersistentStoreConfiguration is intuitive, and
> > MemoryPolicyConfiguration
> > > is
> > > > intuitive also.
> > > >
> > > > If we rename these classes now, it will bring more questions to user
> > > list.
> > > > Users may be confused by old and new names and by trying to match it.
> > > More
> > > > issues can came from XML configs that users already have.
> > > >
> > > > Can we postpone the renaming? I suggest to finish 'persistence per
> > memory
> > > > policy' task without renaming, and create separate JIRA issue for
> > > creating
> > > > future decision?
> > > >
> > > > вт, 26 сент. 2017 г. в 15:25, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > >:
> > > >
> > > > > I do not like DurableMemoryConfiguration, because it's quite
> > confusing
> > > -
> > > > we
> > > > > configure in-memory caches using DurableMemory class, which
> > immediately
> > > > > suggests that everything will be persisted. I am not sure if this
> is
> > a
> > > > > right wording choice for the documentation either. I would go with
> > > > > DataStoreConfiguration and DataRegionConfiguration.
> > > > >
> > > > > --AG
> > > > >
> > > > > 2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan  >:
> > > > >
> > > > > > Given that we already have a notion of CacheStore which comes
> from
> > > > JCache
> > > > > > spec, I think having other stores may get confusing. I like
> > > > > > DurableMemoryConfiguration.
> > > > > >
> > > > > > Other opinions?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov <
> > > > voze...@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dima, let's finalize the design first.
> > > > > > >
> > > > > > > As I understand, we are happy with idea to merge
> > > MemoryConfiguration
> > > > > > > and PersistentStoreConfiguration
> > > > > > > into something what I called DataConfiguration, and to rename
> > > > > > > MemoryPolicyConfiguration to DataRegionConfiguration.
> > > > > > >
> > > > > > > The only outstanding qurestion is whether DataConfiguration is
> a
> > > good
> > > > > > name.
> > > > > > > I am not very happy with it, so let's think of other
> > alternatives.
> > > > > Quick
> > > > > > > ideas:
> > > > > > > 1) StoreConfiguration - looks perfect to me - short and
> > > > > self-describing,
> > > > > > > but clashes a bit with existing CacheStore
> > > > > > > 2) DataStoreConfiguration - same as p.1, but the word "data" is
> > not
> > > > > > > necessary IMO
> > > > > > > 3) PageStoreConfiguration? GIves a hint to our page-based
> > > > architecture.
> > > > > > > 4) DurableMemoryConfiguration - aligns well with our docs, but
> I
> > do
> > > > not
> > > > > > > like it - too long and misleading
> > > > > > >
> > > > > > > Any other ideas?
> > > > > > >
> > > > > > > I would prefer to have either [StoreConfiguration +
> > > > > > > StoreRegionConfiguration] or [PageStoreConfiguration and
> > > > > > > PageStoreRegionConfiguration]. Looks clean and simple.
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >

[GitHub] ignite pull request #2754: IGNITE-5615 .NET: IgniteConfiguration.LocalEventL...

2017-09-26 Thread ptupitsyn

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/2754

IGNITE-5615 .NET: IgniteConfiguration.LocalEventListeners



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5615

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2754


commit 30232dbdb8317122cf46a3ef7560564085efd04a
Author: Pavel Tupitsyn 
Date:   2017-09-26T14:26:01Z

IGNITE-5615 .NET: IgniteConfiguration.LocalEventListeners




---

Re: Warning if no table for BinaryObject

2017-09-26 Thread Mikhail Cherkasov

Hi Dmitry,

The problem is that we don't have strong typing when defining QueryEntriy
and a user can use an arbitrary string id to
define a type, but he should use the same string id to obtain binary object
builder, however, people sometimes confusing this.
So the user can define QueryEntity with value type:
 queryEntity.setValueType("MyCoolName") and
later put to cache the following binary object:
ignite.binary.toBinary(value), but this object won't be indexed, because
ignite.binary.toBinary uses class name as string id while indexing expects
to find "MyCoolName" as id.

There's no bug in ignite about this, but I think the warning can make
users' life easy and
make ignite configuration more friendly to new users.

Thanks,
Mikhail.

On Tue, Sep 26, 2017 at 8:15 AM, Yakov Zhdanov  wrote:

> Pavel, we can optionally enforce that, but not by default.
>
> Dmitry, the problem is the following. User configures Ignite cache and
> query entity, does several puts than empty results for SQL. The reason is
> very simple - typo in class name or simply different classes in
> configuration and of actual cache entry. The fix is pretty simple and make
> product more usable.
>
> --Yakov
>

-- 
Thanks,
Mikhail.

Re: Persistence per memory policy configuration

2017-09-26 Thread Dmitry Pavlov

Vladimir, it is not clear for me, why we need to rename existing
configuration classes. Could you explain? And if we can't get consensus
now, should we pospond solution?

My idea is that user needs this feature more than elegant names in
configuration.

Moreover once MemoryPolicyConfiguration was introduced as Ignite term it is
simpler to keep it as is, than create new terms.

Sincerely,
Dmitriy Pavlov

вт, 26 сент. 2017 г. в 16:59, Vladimir Ozerov :

> I do not understand why we should delay with renames. Yes, it will cause
> questions, so we will have to put additional efforts to docs and JavaDocs.
> But the earlier we do that, the better.
>
> On Tue, Sep 26, 2017 at 4:50 PM, Dmitry Pavlov 
> wrote:
>
> > Hi Igniters, sorry for late response. I didn't catch idea of renaming.
> > PersistentStoreConfiguration is intuitive, and MemoryPolicyConfiguration
> is
> > intuitive also.
> >
> > If we rename these classes now, it will bring more questions to user
> list.
> > Users may be confused by old and new names and by trying to match it.
> More
> > issues can came from XML configs that users already have.
> >
> > Can we postpone the renaming? I suggest to finish 'persistence per memory
> > policy' task without renaming, and create separate JIRA issue for
> creating
> > future decision?
> >
> > вт, 26 сент. 2017 г. в 15:25, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> >
> > > I do not like DurableMemoryConfiguration, because it's quite confusing
> -
> > we
> > > configure in-memory caches using DurableMemory class, which immediately
> > > suggests that everything will be persisted. I am not sure if this is a
> > > right wording choice for the documentation either. I would go with
> > > DataStoreConfiguration and DataRegionConfiguration.
> > >
> > > --AG
> > >
> > > 2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Given that we already have a notion of CacheStore which comes from
> > JCache
> > > > spec, I think having other stores may get confusing. I like
> > > > DurableMemoryConfiguration.
> > > >
> > > > Other opinions?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Dima, let's finalize the design first.
> > > > >
> > > > > As I understand, we are happy with idea to merge
> MemoryConfiguration
> > > > > and PersistentStoreConfiguration
> > > > > into something what I called DataConfiguration, and to rename
> > > > > MemoryPolicyConfiguration to DataRegionConfiguration.
> > > > >
> > > > > The only outstanding qurestion is whether DataConfiguration is a
> good
> > > > name.
> > > > > I am not very happy with it, so let's think of other alternatives.
> > > Quick
> > > > > ideas:
> > > > > 1) StoreConfiguration - looks perfect to me - short and
> > > self-describing,
> > > > > but clashes a bit with existing CacheStore
> > > > > 2) DataStoreConfiguration - same as p.1, but the word "data" is not
> > > > > necessary IMO
> > > > > 3) PageStoreConfiguration? GIves a hint to our page-based
> > architecture.
> > > > > 4) DurableMemoryConfiguration - aligns well with our docs, but I do
> > not
> > > > > like it - too long and misleading
> > > > >
> > > > > Any other ideas?
> > > > >
> > > > > I would prefer to have either [StoreConfiguration +
> > > > > StoreRegionConfiguration] or [PageStoreConfiguration and
> > > > > PageStoreRegionConfiguration]. Looks clean and simple.
> > > > >
> > > > > Vladimir.
> > > > >
> > > > >
> > > > > On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Vladimir,
> > > > > >
> > > > > > Can you please add the configuration example in the ticket?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com> wrote:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I suggest we finalize the configuration changes in the original
> > > > ticket
> > > > > > > then: https://issues.apache.org/jira/browse/IGNITE-6030 and
> > > proceed
> > > > > with
> > > > > > > the changes.
> > > > > > >
> > > > > > > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >:
> > > > > > >
> > > > > > > > Can we specify what metrics will look like? I think we should
> > not
> > > > > just
> > > > > > > > blindly merge them.
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> > > > > > voze...@gridgain.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Denis,
> > > > > > > > >
> > > > > > > > > Makes sense. Thanks for catching it!
> > > > > > > > >
> > > > > > > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda <
> > > dma...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If we’re taking the consolidation path for Memory and
> > > > Persistence
> > > > > > > > > >

[GitHub] ignite pull request #2753: ignite-5714 updateExplicitVersion for detached en...

2017-09-26 Thread voipp

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2753

ignite-5714 updateExplicitVersion for detached entry



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2753.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2753


commit d6df5781403a08171a63ccff04d5b04c6352e563
Author: voipp 
Date:   2017-09-26T14:07:45Z

ignite-5714 updateExplicitVersion for detached entry




---

Re: Persistence per memory policy configuration

2017-09-26 Thread Vladimir Ozerov

I do not understand why we should delay with renames. Yes, it will cause
questions, so we will have to put additional efforts to docs and JavaDocs.
But the earlier we do that, the better.

On Tue, Sep 26, 2017 at 4:50 PM, Dmitry Pavlov 
wrote:

> Hi Igniters, sorry for late response. I didn't catch idea of renaming.
> PersistentStoreConfiguration is intuitive, and MemoryPolicyConfiguration is
> intuitive also.
>
> If we rename these classes now, it will bring more questions to user list.
> Users may be confused by old and new names and by trying to match it. More
> issues can came from XML configs that users already have.
>
> Can we postpone the renaming? I suggest to finish 'persistence per memory
> policy' task without renaming, and create separate JIRA issue for creating
> future decision?
>
> вт, 26 сент. 2017 г. в 15:25, Alexey Goncharuk  >:
>
> > I do not like DurableMemoryConfiguration, because it's quite confusing -
> we
> > configure in-memory caches using DurableMemory class, which immediately
> > suggests that everything will be persisted. I am not sure if this is a
> > right wording choice for the documentation either. I would go with
> > DataStoreConfiguration and DataRegionConfiguration.
> >
> > --AG
> >
> > 2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Given that we already have a notion of CacheStore which comes from
> JCache
> > > spec, I think having other stores may get confusing. I like
> > > DurableMemoryConfiguration.
> > >
> > > Other opinions?
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > wrote:
> > >
> > > > Dima, let's finalize the design first.
> > > >
> > > > As I understand, we are happy with idea to merge MemoryConfiguration
> > > > and PersistentStoreConfiguration
> > > > into something what I called DataConfiguration, and to rename
> > > > MemoryPolicyConfiguration to DataRegionConfiguration.
> > > >
> > > > The only outstanding qurestion is whether DataConfiguration is a good
> > > name.
> > > > I am not very happy with it, so let's think of other alternatives.
> > Quick
> > > > ideas:
> > > > 1) StoreConfiguration - looks perfect to me - short and
> > self-describing,
> > > > but clashes a bit with existing CacheStore
> > > > 2) DataStoreConfiguration - same as p.1, but the word "data" is not
> > > > necessary IMO
> > > > 3) PageStoreConfiguration? GIves a hint to our page-based
> architecture.
> > > > 4) DurableMemoryConfiguration - aligns well with our docs, but I do
> not
> > > > like it - too long and misleading
> > > >
> > > > Any other ideas?
> > > >
> > > > I would prefer to have either [StoreConfiguration +
> > > > StoreRegionConfiguration] or [PageStoreConfiguration and
> > > > PageStoreRegionConfiguration]. Looks clean and simple.
> > > >
> > > > Vladimir.
> > > >
> > > >
> > > > On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Vladimir,
> > > > >
> > > > > Can you please add the configuration example in the ticket?
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com> wrote:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > I suggest we finalize the configuration changes in the original
> > > ticket
> > > > > > then: https://issues.apache.org/jira/browse/IGNITE-6030 and
> > proceed
> > > > with
> > > > > > the changes.
> > > > > >
> > > > > > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >:
> > > > > >
> > > > > > > Can we specify what metrics will look like? I think we should
> not
> > > > just
> > > > > > > blindly merge them.
> > > > > > >
> > > > > > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> > > > > voze...@gridgain.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Denis,
> > > > > > > >
> > > > > > > > Makes sense. Thanks for catching it!
> > > > > > > >
> > > > > > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda <
> > dma...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > If we’re taking the consolidation path for Memory and
> > > Persistence
> > > > > > > > > configurations then it makes sense to merge MemoryMetrics
> [1]
> > > and
> > > > > > > > > PersistenceMetrics [2] plus their JMX beans.
> > > > > > > > >
> > > > > > > > > Agree?
> > > > > > > > >
> > > > > > > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > > > > apache/ignite/MemoryMetrics.html <
> https://ignite.apache.org/
> > > > > > > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.
> html>
> > > > > > > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > > apache/ignite/
> > > > > > > > > PersistenceMetrics.html
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > > > > >

Re: Persistence per memory policy configuration

2017-09-26 Thread Dmitry Pavlov

Hi Igniters, sorry for late response. I didn't catch idea of renaming.
PersistentStoreConfiguration is intuitive, and MemoryPolicyConfiguration is
intuitive also.

If we rename these classes now, it will bring more questions to user list.
Users may be confused by old and new names and by trying to match it. More
issues can came from XML configs that users already have.

Can we postpone the renaming? I suggest to finish 'persistence per memory
policy' task without renaming, and create separate JIRA issue for creating
future decision?

вт, 26 сент. 2017 г. в 15:25, Alexey Goncharuk :

> I do not like DurableMemoryConfiguration, because it's quite confusing - we
> configure in-memory caches using DurableMemory class, which immediately
> suggests that everything will be persisted. I am not sure if this is a
> right wording choice for the documentation either. I would go with
> DataStoreConfiguration and DataRegionConfiguration.
>
> --AG
>
> 2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan :
>
> > Given that we already have a notion of CacheStore which comes from JCache
> > spec, I think having other stores may get confusing. I like
> > DurableMemoryConfiguration.
> >
> > Other opinions?
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Dima, let's finalize the design first.
> > >
> > > As I understand, we are happy with idea to merge MemoryConfiguration
> > > and PersistentStoreConfiguration
> > > into something what I called DataConfiguration, and to rename
> > > MemoryPolicyConfiguration to DataRegionConfiguration.
> > >
> > > The only outstanding qurestion is whether DataConfiguration is a good
> > name.
> > > I am not very happy with it, so let's think of other alternatives.
> Quick
> > > ideas:
> > > 1) StoreConfiguration - looks perfect to me - short and
> self-describing,
> > > but clashes a bit with existing CacheStore
> > > 2) DataStoreConfiguration - same as p.1, but the word "data" is not
> > > necessary IMO
> > > 3) PageStoreConfiguration? GIves a hint to our page-based architecture.
> > > 4) DurableMemoryConfiguration - aligns well with our docs, but I do not
> > > like it - too long and misleading
> > >
> > > Any other ideas?
> > >
> > > I would prefer to have either [StoreConfiguration +
> > > StoreRegionConfiguration] or [PageStoreConfiguration and
> > > PageStoreRegionConfiguration]. Looks clean and simple.
> > >
> > > Vladimir.
> > >
> > >
> > > On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > Vladimir,
> > > >
> > > > Can you please add the configuration example in the ticket?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I suggest we finalize the configuration changes in the original
> > ticket
> > > > > then: https://issues.apache.org/jira/browse/IGNITE-6030 and
> proceed
> > > with
> > > > > the changes.
> > > > >
> > > > > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > > >
> > > > > > Can we specify what metrics will look like? I think we should not
> > > just
> > > > > > blindly merge them.
> > > > > >
> > > > > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> > > > voze...@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Denis,
> > > > > > >
> > > > > > > Makes sense. Thanks for catching it!
> > > > > > >
> > > > > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda <
> dma...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > If we’re taking the consolidation path for Memory and
> > Persistence
> > > > > > > > configurations then it makes sense to merge MemoryMetrics [1]
> > and
> > > > > > > > PersistenceMetrics [2] plus their JMX beans.
> > > > > > > >
> > > > > > > > Agree?
> > > > > > > >
> > > > > > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > > > apache/ignite/MemoryMetrics.html  > > > > > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > > > > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > apache/ignite/
> > > > > > > > PersistenceMetrics.html
> > > > > > > >
> > > > > > > > —
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I like it.
> > > > > > > > >
> > > > > > > > > Alexey G, can you please chime in?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > > > > > > voze...@gridgain.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Guys,
> > > > > > > > >>
> > > > > > > > >> Here is my proposal:
> > > > > > > > >>
> > > > > > > > >> 1) MemoryPolicyConfiguration is

[GitHub] ignite pull request #2752: IGNITE-6285: Enhance persistent store paths loggi...

2017-09-26 Thread dspavlov

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/2752

IGNITE-6285: Enhance persistent store paths logging on start

Experimental commits to run TC tests, do not merge

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6285

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2752


commit b150519f05c99e71be83af9d5107ae599c90f05f
Author: dpavlov 
Date:   2017-09-26T13:31:54Z

IGNITE-6285: Enhance persistent store paths logging on start




---

Re: [DISCUSS] Ignite Update Checker

2017-09-26 Thread Vladimir Ozerov

Folks,

Can we add version of current node to web request? This way we will better
understand version distribution, what might help us with certain API
update/deprecate decisions
E.g. http://ignite.apache.org/latest.cgi=2.2.0

Vladimir.

On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan 
wrote:

> I think it is safe to assume at this point that everyone is in general
> agreement, since there are no active objections.
>
> I have filed a ticket for the 2.3 release. Let's try to make it happen:
> https://issues.apache.org/jira/browse/IGNITE-6305
>
> D.
>
> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan 
> wrote:
>
> >
> >
> > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani 
> > wrote:
> >
> >> Yeah, I guess that's doable as well and requires less management effort
> >> than my suggestion. We could use events [1] to store payload data (e.g.
> >> IP,
> >> version, etc.)
> >
> >
> > Yes, we could use events or some other similar API provided by GA.
> >
> >
> >> What the download page CGI developed in? PHP?
> >>
> >
> > To be honest, no clue. I guess someone in the community can figure it
> out:
> > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >
> >
> >> However, I'm not sure whether storing this data in a 3rd party (Google)
> is
> >> compliant with the ASF policy. I guess it's no biggie, but if there's
> >> doubt
> >> in the PMC, it's better to ask legal@.
> >
> >
> > I am not sure there is anything to ask about. The whole Ignite website is
> > GA enabled, and all we are doing is accessing a standard web page from
> the
> > Ignite web site. The information gathered from GA is available only to
> the
> > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > question.
> >
> > Even ASF website itself uses GA: https://www.apache.org/
> > foundation/policies/privacy.html
> >
> >
> >> I think Cos said it's OK; maybe Roman can pitch in.
> >>
> >
> >  Sure, would be nice to hear from Roman as well.
> >
> >
> >> Cheers.
> >>
> >> [1]
> >> https://developers.google.com/analytics/devguides/collection
> >> /analyticsjs/events
> >>
> >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >> dsetrak...@apache.org>
> >> wrote:
> >>
> >> > Raul,
> >> >
> >> > Could point about Javascript, it will not work because it executes in
> >> the
> >> > browser. This means we need a server-side script, like CGI we are
> using
> >> on
> >> > our download page.
> >> >
> >> > How about this approach. We create something like ignite-version.cgi
> >> script
> >> > which will invoke a call to GA and then return the latest version.
> >> >
> >> > This should work, right?
> >> >
> >> > D.
> >> >
> >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani  >
> >> > wrote:
> >> >
> >> > > Hey Dmitriy and all
> >> > >
> >> > > Also, since we have GA enabled for the website, we can track how
> many
> >> > times
> >> > > > this page was accessed, which will be equal to the number of
> starts.
> >> > This
> >> > > > way, the counter information is tracked and monitored by the
> Ignite
> >> > PMC.
> >> > >
> >> > >
> >> > > Unfortunately this won't work because GA is loaded via Javascript on
> >> the
> >> > > browser, so Google will never receive the page hit.
> >> > >
> >> > > Given the constraints, the most viable solution is an HTTPS endpoint
> >> > > running on ASF infra that Ignite invokes via a GET or POST request.
> >> The
> >> > > simplest thing is to write each request in a log file, along with
> the
> >> > > timestamp, the version reported by the client, maybe the IP (not
> sure
> >> > about
> >> > > the ASF rules about this concerning privacy, I guess it's OK if you
> >> make
> >> > it
> >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> creates
> >> on
> >> > > first startup or something.
> >> > >
> >> > > This endpoint would need some basic DDoS protection and blacklisting
> >> to
> >> > > prevent data spoofing.
> >> > >
> >> > > If we'll be implementing this endpoint anyway, then there's no point
> >> > > placing another file on Git or elsewhere for reporting the latest
> >> > versions:
> >> > > the endpoint itself can return them.
> >> > >
> >> > > WDYT?
> >> > >
> >> > > Cheers.
> >> > >
> >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >> > dsetrak...@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Cos, Raul,
> >> > > >
> >> > > > Thanks for the feedback. I completely agree about Maven Central
> >> being a
> >> > > 3rd
> >> > > > party repo (did not think about that initially). All your
> >> suggestions
> >> > > make
> >> > > > sense, but I would like to keep it as simple as possible, and so
> far
> >> > > > everything suggested required GIT dependencies and extra work.
> >> > > >
> >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> >> > website
> >> > > > which will have only the latest version. Every time a node starts,
> >> it
> >> > > >

[GitHub] ignite pull request #2747: Attempt to fix not-null compatibility issue.

2017-09-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2747


---

[GitHub] ignite pull request #2751: Adding lock if timeout is negative

2017-09-26 Thread voipp

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2751

Adding lock if timeout is negative



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite 
ignite-5714-local-lock-not-covered

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2751


commit c2014e7c0b447381a1ac4b0a3803ed7333f9e864
Author: voipp 
Date:   2017-09-26T13:08:05Z

Adding lock if timeout is negative




---

[GitHub] ignite pull request #2750: IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

2017-09-26 Thread schernolyas

GitHub user schernolyas opened a pull request:

https://github.com/apache/ignite/pull/2750

IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

changes:

1. develop test for check all supported types for parameterized queries
2. add support BigDecimal type as type of paraameter's value
3. refactoring (remove unused 'log.info')

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schernolyas/ignite IGNITE-6286-DEV2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2750


commit 10938b6ec77bd00037d96881c42174525fe0aa58
Author: Sergey Chernolyas 
Date:   2017-09-26T12:56:44Z

IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

changes:

1. develop test for check all supported types for parameterized queries
2. add support BigDecimal type as type of paraameter's value
3. refactoring (remove unused 'log.info')




---

[GitHub] ignite pull request #2749: IGNITE-6186: Removed redundant parameter.

2017-09-26 Thread andrey-kuznetsov

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2749

IGNITE-6186: Removed redundant parameter.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6186

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2749.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2749


commit 9646f442bb6327bb08c984c5e0caf0a9f7438640
Author: Andrey Kuznetsov 
Date:   2017-09-26T12:55:13Z

IGNITE-6186: Removed redundant parameter.




---

[GitHub] ignite pull request #2737: IGNITE-6286: fix org.h2.jdbc.JdbcSQLException

2017-09-26 Thread schernolyas

Github user schernolyas closed the pull request at:

https://github.com/apache/ignite/pull/2737


---

Re: Persistence per memory policy configuration

2017-09-26 Thread Alexey Goncharuk

I do not like DurableMemoryConfiguration, because it's quite confusing - we
configure in-memory caches using DurableMemory class, which immediately
suggests that everything will be persisted. I am not sure if this is a
right wording choice for the documentation either. I would go with
DataStoreConfiguration and DataRegionConfiguration.

--AG

2017-09-26 2:22 GMT+03:00 Dmitriy Setrakyan :

> Given that we already have a notion of CacheStore which comes from JCache
> spec, I think having other stores may get confusing. I like
> DurableMemoryConfiguration.
>
> Other opinions?
>
> D.
>
> On Mon, Sep 25, 2017 at 12:24 PM, Vladimir Ozerov 
> wrote:
>
> > Dima, let's finalize the design first.
> >
> > As I understand, we are happy with idea to merge MemoryConfiguration
> > and PersistentStoreConfiguration
> > into something what I called DataConfiguration, and to rename
> > MemoryPolicyConfiguration to DataRegionConfiguration.
> >
> > The only outstanding qurestion is whether DataConfiguration is a good
> name.
> > I am not very happy with it, so let's think of other alternatives. Quick
> > ideas:
> > 1) StoreConfiguration - looks perfect to me - short and self-describing,
> > but clashes a bit with existing CacheStore
> > 2) DataStoreConfiguration - same as p.1, but the word "data" is not
> > necessary IMO
> > 3) PageStoreConfiguration? GIves a hint to our page-based architecture.
> > 4) DurableMemoryConfiguration - aligns well with our docs, but I do not
> > like it - too long and misleading
> >
> > Any other ideas?
> >
> > I would prefer to have either [StoreConfiguration +
> > StoreRegionConfiguration] or [PageStoreConfiguration and
> > PageStoreRegionConfiguration]. Looks clean and simple.
> >
> > Vladimir.
> >
> >
> > On Mon, Sep 25, 2017 at 3:49 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Vladimir,
> > >
> > > Can you please add the configuration example in the ticket?
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 12:20 AM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > Guys,
> > > >
> > > > I suggest we finalize the configuration changes in the original
> ticket
> > > > then: https://issues.apache.org/jira/browse/IGNITE-6030 and proceed
> > with
> > > > the changes.
> > > >
> > > > 2017-09-23 17:08 GMT+03:00 Dmitriy Setrakyan  >:
> > > >
> > > > > Can we specify what metrics will look like? I think we should not
> > just
> > > > > blindly merge them.
> > > > >
> > > > > On Fri, Sep 22, 2017 at 11:01 PM, Vladimir Ozerov <
> > > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > Makes sense. Thanks for catching it!
> > > > > >
> > > > > > On Sat, Sep 23, 2017 at 8:45 AM, Denis Magda 
> > > > wrote:
> > > > > >
> > > > > > > If we’re taking the consolidation path for Memory and
> Persistence
> > > > > > > configurations then it makes sense to merge MemoryMetrics [1]
> and
> > > > > > > PersistenceMetrics [2] plus their JMX beans.
> > > > > > >
> > > > > > > Agree?
> > > > > > >
> > > > > > > [1] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > > > apache/ignite/MemoryMetrics.html  > > > > > > releases/latest/javadoc/org/apache/ignite/MemoryMetrics.html>
> > > > > > > [2] https://ignite.apache.org/releases/latest/javadoc/org/
> > > > > apache/ignite/
> > > > > > > PersistenceMetrics.html
> > > > > > >
> > > > > > > —
> > > > > > > Denis
> > > > > > >
> > > > > > > > On Sep 22, 2017, at 10:18 PM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I like it.
> > > > > > > >
> > > > > > > > Alexey G, can you please chime in?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2017 at 11:33 AM, Vladimir Ozerov <
> > > > > > voze...@gridgain.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Guys,
> > > > > > > >>
> > > > > > > >> Here is my proposal:
> > > > > > > >>
> > > > > > > >> 1) MemoryPolicyConfiguration is renamed to
> > > > > *DataRegionConfiguration*.
> > > > > > > >> 2) PersistenceConfiguration is merged with
> MemoryConfiguration
> > > and
> > > > > > > renamed
> > > > > > > >> to ... *DataStorageConfiguration*! It has: common memory
> > > settings
> > > > > > (e.g.
> > > > > > > >> default data region), persistence settings (e.g. WAL) and a
> > list
> > > > of
> > > > > > > >> DataRegionConfiguration beans.
> > > > > > > >>
> > > > > > > >> What we have in the end:
> > > > > > > >>
> > > > > > > >> 
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> > />
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> > > value="1_000_000_000"
> > > > />
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>

[GitHub] ignite pull request #2748: IGNITE-6015: Rollback on exception in commitIfLoc...

2017-09-26 Thread andrey-kuznetsov

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2748

IGNITE-6015: Rollback on exception in commitIfLocked.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6015

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2748.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2748


commit 854a0e393c171192998d77ce0238d5959f79e629
Author: Andrey Kuznetsov 
Date:   2017-09-26T11:54:08Z

IGNITE-6015: Rollback on exception in commitIfLocked.




---

[GitHub] ignite pull request #2747: Attempt to fix not-null compatibility issue.

2017-09-26 Thread devozerov

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2747

Attempt to fix not-null compatibility issue.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-12822

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2747.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2747


commit 8e91828bcce1737d2a0e222c746bfe5d1792968c
Author: devozerov 
Date:   2017-09-26T11:47:56Z

Attempt to fix not-null compatibility issue.




---

Re: Issues if -Djava.net.preferIPv4Stack=true is not set

2017-09-26 Thread Dmitriy Setrakyan

Why don't we automatically set the preferIPv4Stack to true, unless the user
explicitly set it to false?

D.

On Tue, Sep 26, 2017 at 4:35 AM, Yakov Zhdanov  wrote:

> Val, I see now.
>
> For example, this http://apache-ignite-users.70518.x6.nabble.com/How-
> to-correctly-shut-down-Ignite-Application-td12548.html could happen due to
> address passed to
> org.apache.ignite.internal.util.IgniteUtils#reachable(
> java.net.InetAddress,
> int) is NULL.
>
> I suspect there may be problems with ipv6 support as I personally cannot
> name a deployment that runs over ipv6 and I doubt if it is useful. I will
> file a ticket to output warning if prefer ipv4 property is not set. And
> also a minor ticket to test and fix ipv6 support.
>
> BTW, here are the tickets that are searchable by "ipv6" for now -
> https://issues.apache.org/jira/browse/IGNITE-2405?jql=
> text%20~%20%22ipv6%22%20and%20project%20%3D%20ignite
>
> Issue to add warning - https://issues.apache.org/jira/browse/IGNITE-6502
>
>
> --Yakov
>

Re: Issues if -Djava.net.preferIPv4Stack=true is not set

2017-09-26 Thread Yakov Zhdanov

Val, I see now.

For example, this http://apache-ignite-users.70518.x6.nabble.com/How-
to-correctly-shut-down-Ignite-Application-td12548.html could happen due to
address passed to
org.apache.ignite.internal.util.IgniteUtils#reachable(java.net.InetAddress,
int) is NULL.

I suspect there may be problems with ipv6 support as I personally cannot
name a deployment that runs over ipv6 and I doubt if it is useful. I will
file a ticket to output warning if prefer ipv4 property is not set. And
also a minor ticket to test and fix ipv6 support.

BTW, here are the tickets that are searchable by "ipv6" for now -
https://issues.apache.org/jira/browse/IGNITE-2405?jql=
text%20~%20%22ipv6%22%20and%20project%20%3D%20ignite

Issue to add warning - https://issues.apache.org/jira/browse/IGNITE-6502


--Yakov

[jira] [Created] (IGNITE-6503) Need to test Ignite in IPv6 environment

2017-09-26 Thread Yakov Zhdanov (JIRA)

Yakov Zhdanov created IGNITE-6503:
-

 Summary: Need to test Ignite in IPv6 environment
 Key: IGNITE-6503
 URL: https://issues.apache.org/jira/browse/IGNITE-6503
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov
Priority: Minor


It seems that Ignite may have problems with IPv6. Need to test it and fix if 
necessary.

Areas to check:
# build cluster on local node with different IP finders on different OSes.
# build distributed cluster with different IP finders on different OSes.
# check communication by running some yardstick benchmark (e.g. atomic put). 1 
client vs 2-3 servers will be fine.
# kill some node while benchmark is running
# add new node while benchmark is running

http://apache-ignite-developers.2346864.n4.nabble.com/Issues-if-Djava-net-preferIPv4Stack-true-is-not-set-td22372.html




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6502) Need to output warning if -Djava.net.preferIPv4Stack=true is not set

2017-09-26 Thread Yakov Zhdanov (JIRA)

Yakov Zhdanov created IGNITE-6502:
-

 Summary: Need to output warning if -Djava.net.preferIPv4Stack=true 
is not set
 Key: IGNITE-6502
 URL: https://issues.apache.org/jira/browse/IGNITE-6502
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov


Some issues were reported on dev/user list that may be caused by ipv6 usage. I 
am not sure if Ignite can properly work with ipv6 networks. This needs to be 
tested.

For now it seems to be pretty handy just to have warning on node start:

{{noformat}}
Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible 
problems in mixed environments.
{{noformat}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2746: Ignite 5714 2

2017-09-26 Thread voipp

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2746

Ignite 5714 2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2746


commit 4f93c312a05328e8513053146f6801cd8cd98ce9
Author: voipp 
Date:   2017-08-28T12:55:37Z

ignite-5714 exception when processing DhtLockRequest

commit 5012c7bb45b06f9f288e695f38a199a0474851ce
Author: voipp 
Date:   2017-09-22T14:38:52Z

Revert: ignite-5714 exception when processing DhtLockRequest

commit 7e98c8d967730bc13e2b78e1817279a4f2938f33
Author: voipp 
Date:   2017-09-26T11:18:37Z

exception when failed to unlock entry




---

Re: Future of Ignite transactions

2017-09-26 Thread Alexey Goncharuk

Dmitriy,

Agree with Vladimir here. OPTIMISTIC + (READ_COMMITTED | REPEATABLE_READ)
modes are completely unusable in real-life use-cases because they do not
allow any read-write conflict detection and thus the explicit transaction
statement can be omitted at all. The remaining combinations of tx
concurrency and isolation can be reduced to a new enum that will match SQL
isolation levels with crystal clear isolation guarantees.

I see no problem with deprecating the old API and completely removing it in
3.0.

2017-09-26 11:52 GMT+03:00 Dmitriy Setrakyan :

> On Tue, Sep 26, 2017 at 12:08 AM, Vladimir Ozerov 
> wrote:
>
> > Dima,
> >
> > My proposal would require two changes:
> > 1) Public - for sure we will continue support old API
> >
>
> Of course. But my assumption is that you would deprecate the old API,
> right? If yes, then you will never be able to remove the deprecated API.
>
> 2) Internal TX engine - these changes are inevitable for transactional SQL
> > support.
> >
>
> Internally we can do any changes we like, as long as they don't change the
> expected user behavior.
>
>
> >
> > So we have no freedom to choose whether to spend time on internals or
> not.
> > We will have to do that anyway.
> >
> > On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Vladimir,
> > >
> > > I do not think we have a luxury of changing Ignite transaction APIs. It
> > is
> > > almost the same as changing Ignite put and get APIs. Too much code has
> > > already been written to these APIs. The upgrade path will be so onerous
> > > that no one will ever take it.
> > >
> > > As far as the current transaction flags, we will have to continue to
> > > support them. Ideally we should extend them to Ignite SQL as well. If
> > there
> > > is a new transaction flag that you wish to add to Ignite SQL, please
> > > suggest it here.
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > wrote:
> > >
> > > > Dima,
> > > >
> > > > Becuase if you take any pair of our concurrency/isolation modes, you
> > will
> > > > not find anything similar in SQL. The main problem is that there is
> no
> > > > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are
> "optimistic",
> > > but
> > > > can be converted to "perssimistic" optionally, and all writes are
> > > > "pessimistic". Another problem is that some of our pairs cannot be
> used
> > > in
> > > > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> > > some
> > > > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > > > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > > > reason about. If you cannot explain API in 5 mins, it is broken :-)
> > > >
> > > > If we were many years in the past allowed to choose between Ignite
> and
> > > SQL
> > > > approaches I would prefer SQL because it is much simpler.
> > > >
> > > > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Vladimir,
> > > > >
> > > > > This looks like a complete overhaul of our transactional behavior
> > and I
> > > > do
> > > > > not think we are at a liberty to make such drastic changes. Can you
> > > > please
> > > > > explain why the current behavior would not map to the SQL
> > transactions?
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> > > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Sorry for late reply. I had a chat with several Ignite veterans
> > > today.
> > > > We
> > > > > > tried to design transactional SQL for Ignite. One of our
> questions
> > > was
> > > > > how
> > > > > > to align SQL transactions with current Ignite transactions. And
> we
> > > > > failed.
> > > > > > And then we came to conclusion that current transaction API is
> > > > unusable.
> > > > > We
> > > > > > have 6 pairs of modes from API standpoint and 4 real modes. This
> is
> > > > very
> > > > > > counterintuitive and cannot be mapped to any transactional
> > framework
> > > > our
> > > > > > users are familiar with.
> > > > > >
> > > > > > So we thought how new tx API might looks like, and here is the
> > draft.
> > > > > >
> > > > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes
> > with
> > > > > > current enum) with three standard modes - READ_COMMITTED,
> > > > > REPEATABLE_READ,
> > > > > > SERIALIZABLE.
> > > > > > 2) Define new enum *TransactionHint* - READ_ONLY,
> > OPTIMISTIC_LOCKING
> > > > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > > > 4) Reads never acuire locks
> > > > > > 5) Writes always acquire locks
> > > > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> > > which
> > > > > will
> > > > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in
> > SQL.
> > > > > > 7) *TransactionHint.READ_ONLY* -

[GitHub] ignite pull request #2745: Ignite 1.8.12

2017-09-26 Thread sk0x50

GitHub user sk0x50 opened a pull request:

https://github.com/apache/ignite/pull/2745

Ignite 1.8.12



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-1.8.12

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2745


commit 1f3b2fcd003c1f084874d5c421953da0a7cd02cb
Author: Valentin Kulichenko 
Date:   2017-03-28T01:12:17Z

IGNITE-4802 - Separate thread pool for managed services

commit 39edcc7890ce497771f532a83a57ecae318d8c88
Author: Andrey V. Mashenkov 
Date:   2017-03-28T15:56:17Z

IGNITE-4815: Fixed confusing javadoc. This closes #1613.

commit e47bf27b5582cd695a7d3de3c39d54b3df4c59cc
Author: Valentin Kulichenko 
Date:   2017-03-28T23:51:44Z

IGNITE-4762 - transactionsAllowed flag for JDBC driver

commit 336672432486830c31cb4b6f8bb1b401c276f3e5
Author: Alexey Kuznetsov 
Date:   2017-03-29T03:53:25Z

IGNITE-4659 Fixed typo.
(cherry picked from commit 6f1e970)

commit 1ddce5517815fc85136e2ead9973c8cf74054f9f
Author: Alexey Kuznetsov 
Date:   2017-03-29T04:11:33Z

Merge branch 'web-console-production' of 
https://github.com/gridgain/apache-ignite into ignite-1.8.5

commit b689624e2984ec3cf42ca5b01966937e236abcdc
Author: Alexander Paschenko 
Date:   2017-03-29T15:15:20Z

IGNITE-4868 Raw Object keys and values indexing fix

commit 357c20ab9593390fb7af25f8638188595c5f6cd4
Author: dkarachentsev 
Date:   2017-03-30T09:50:42Z

IGNITE-4284 - Fix. Failed second client node join with continuous query and 
peer class loading enabled.

commit 45a4cfe7ced27aa9d726b36326bfae8b006e84da
Author: Evgenii Zhuravlev 
Date:   2017-03-29T22:28:20Z

IGNITE-4141 - JDBC driver should always set withKeepBinary flag when 
querying cache. This fixes #1617.

(Backport from master)
(cherry picked from commit 8dd88d8)

commit d4979b87e3301f11f52531645028ecd4e947986b
Author: dkarachentsev 
Date:   2017-03-30T14:35:28Z

IGNITE-4284 - Fix. Failed second client node join with continuous query and 
peer class loading enabled.

commit 2e6bc4416a1f2fa18dad78235b3f71f6fb58a98d
Author: dkarachentsev 
Date:   2017-03-30T14:36:22Z

Cleanup unused imports.

commit e01aee0b1bbdb8ff5583728e539df165029f682d
Author: nikolay_tikhonov 
Date:   2017-03-31T17:19:52Z

Fixed SSL issue.

Signed-off-by: nikolay_tikhonov 

commit ead9f2a595b91821b1b16627dd8d752897c24c03
Author: dkarachentsev 
Date:   2017-04-03T11:46:54Z

Merge branch 'ignite-1.8.4-p1' into ignite-1.8.6

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java

commit 925ee11c2002729b1264148ee5db5700ded5a7b7
Author: Alexey Kuznetsov 
Date:   2017-04-04T09:06:01Z

Fixed typo.
(cherry picked from commit 3b84f34)

commit ce4b078c1c97cae4136c318ae38b27a50fe383cc
Author: Alexey Kuznetsov 
Date:   2017-04-04T09:14:56Z

master Updated version.
(cherry picked from commit 5469626)

commit ae435fb5474429e7ac76faadd680c19a4c657371
Author: Sergi Vladykin 
Date:   2017-04-02T12:35:04Z

ignite-2913 - SQL: EXISTS support added

commit b7ab27301b59bf93fc73b52fdf8e0bcf124fec1d
Author: Andrey V. Mashenkov 
Date:   2017-04-06T11:43:50Z

IGNITE-4832: Prevent service deployment on client by default when 
configuration is provided on startup. This closes #1748.

commit b214211eb3461746b6931c0623e086bb208e5dda
Author: Anton Vinogradov 
Date:   2017-04-06T12:00:12Z

IGNITE-4644 Value from IgniteQueue in atomic mode could be lost

commit 014161427fb603b6df7c8ecc3c0904f5df47a21d
Author: Denis Magda 
Date:   2017-02-14T01:33:32Z

IGNITE-4159: Kubernetes IP finder.
(cherry picked from commit 37c0a22)

commit 1db238402f11c67d2b28bfb7ff47955415f00c25
Author: Denis Magda 
Date:   2017-02-16T04:37:26Z

IGNITE-4159: fixing logging
(cherry picked from commit 06908d2)
(cherry picked from commit fa27ee3)

commit 5dfe16f7e91374008b9f6dfbb899364f5b2e1164
Author: Denis Magda 
Date:   2017-02-14T06:03:30Z

IGNITE-4159: using logger instead of system.out.println
(cherry picked from commit b9bf77c)

commit 6e596d1ef426b66abd866d011a8f5cf5c5c25124
Author: Andrey V. Mashenkov

Contributors Permissions for Ignite

2017-09-26 Thread yashasvi kotamraju

Hi

I would like to request for Contributors Permissions for Ignite Development.

JIRA Username : kotamrajuyashasvi

Presently I would like to assign myself to an Issue which I raised and fix
it.
http://apache-ignite-users.70518.x6.nabble.com/POJO-field-having-wrapper-type-mapped-to-cassandra-table-are-getting-initialized-to-respective-defau-td16135.html

JIRA Ticket:
https://issues.apache.org/jira/browse/IGNITE-6500

I would like to contribute to Ignite in future as well as I would be
working on it for considerable amount of time.

regards
Kotamraju Yashasvi

[GitHub] ignite pull request #2494: ignite-5714-2

2017-09-26 Thread voipp

Github user voipp closed the pull request at:

https://github.com/apache/ignite/pull/2494


---

Re: Future of Ignite transactions

2017-09-26 Thread Dmitriy Setrakyan

On Tue, Sep 26, 2017 at 12:08 AM, Vladimir Ozerov 
wrote:

> Dima,
>
> My proposal would require two changes:
> 1) Public - for sure we will continue support old API
>

Of course. But my assumption is that you would deprecate the old API,
right? If yes, then you will never be able to remove the deprecated API.

2) Internal TX engine - these changes are inevitable for transactional SQL
> support.
>

Internally we can do any changes we like, as long as they don't change the
expected user behavior.


>
> So we have no freedom to choose whether to spend time on internals or not.
> We will have to do that anyway.
>
> On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan 
> wrote:
>
> > Vladimir,
> >
> > I do not think we have a luxury of changing Ignite transaction APIs. It
> is
> > almost the same as changing Ignite put and get APIs. Too much code has
> > already been written to these APIs. The upgrade path will be so onerous
> > that no one will ever take it.
> >
> > As far as the current transaction flags, we will have to continue to
> > support them. Ideally we should extend them to Ignite SQL as well. If
> there
> > is a new transaction flag that you wish to add to Ignite SQL, please
> > suggest it here.
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Dima,
> > >
> > > Becuase if you take any pair of our concurrency/isolation modes, you
> will
> > > not find anything similar in SQL. The main problem is that there is no
> > > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic",
> > but
> > > can be converted to "perssimistic" optionally, and all writes are
> > > "pessimistic". Another problem is that some of our pairs cannot be used
> > in
> > > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> > some
> > > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > > reason about. If you cannot explain API in 5 mins, it is broken :-)
> > >
> > > If we were many years in the past allowed to choose between Ignite and
> > SQL
> > > approaches I would prefer SQL because it is much simpler.
> > >
> > > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > Vladimir,
> > > >
> > > > This looks like a complete overhaul of our transactional behavior
> and I
> > > do
> > > > not think we are at a liberty to make such drastic changes. Can you
> > > please
> > > > explain why the current behavior would not map to the SQL
> transactions?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Sorry for late reply. I had a chat with several Ignite veterans
> > today.
> > > We
> > > > > tried to design transactional SQL for Ignite. One of our questions
> > was
> > > > how
> > > > > to align SQL transactions with current Ignite transactions. And we
> > > > failed.
> > > > > And then we came to conclusion that current transaction API is
> > > unusable.
> > > > We
> > > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > > very
> > > > > counterintuitive and cannot be mapped to any transactional
> framework
> > > our
> > > > > users are familiar with.
> > > > >
> > > > > So we thought how new tx API might looks like, and here is the
> draft.
> > > > >
> > > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes
> with
> > > > > current enum) with three standard modes - READ_COMMITTED,
> > > > REPEATABLE_READ,
> > > > > SERIALIZABLE.
> > > > > 2) Define new enum *TransactionHint* - READ_ONLY,
> OPTIMISTIC_LOCKING
> > > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > > 4) Reads never acuire locks
> > > > > 5) Writes always acquire locks
> > > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> > which
> > > > will
> > > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in
> SQL.
> > > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > > exception
> > > > > on any update
> > > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into
> our
> > > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> > to
> > > > > SERIALIZABLE isolation level.
> > > > > 9) Define new API methods:
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > > TransactionHint... hints)
> > > > > 10) Deprecate old TX start methods
> > > > >
> > > > > As a result we will have simple, clean and extensible API. Which
> can
> > be
> > > > > explained to users in 5 minutes, instead of current half an hour.
> And
> > > > which
> > > > > is perfectly aligned with upcoming transactional SQL.
> > > > >
> > > > > Thoughts?
> > > > >
> > > >

[GitHub] ignite pull request #2741: IGNITE-6494 .NET: Fix CacheConfiguration.WriteSyn...

2017-09-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2741


---

IGNITE-6286

2017-09-26 Thread schernolyas

Hi!

I was prepared https://github.com/apache/ignite/pull/2737 PR. The PR  fix
https://issues.apache.org/jira/browse/IGNITE-6286 .   Please, review it

Also, the PR is very important for project Hibernate OGM for Ignite.

A lot of thank!
With best regards, Sergey Chernolyas




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Future of Ignite transactions

2017-09-26 Thread Vladimir Ozerov

Dima,

My proposal would require two changes:
1) Public - for sure we will continue support old API
2) Internal TX engine - these changes are inevitable for transactional SQL
support.

So we have no freedom to choose whether to spend time on internals or not.
We will have to do that anyway.

On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan 
wrote:

> Vladimir,
>
> I do not think we have a luxury of changing Ignite transaction APIs. It is
> almost the same as changing Ignite put and get APIs. Too much code has
> already been written to these APIs. The upgrade path will be so onerous
> that no one will ever take it.
>
> As far as the current transaction flags, we will have to continue to
> support them. Ideally we should extend them to Ignite SQL as well. If there
> is a new transaction flag that you wish to add to Ignite SQL, please
> suggest it here.
>
> D.
>
> On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov 
> wrote:
>
> > Dima,
> >
> > Becuase if you take any pair of our concurrency/isolation modes, you will
> > not find anything similar in SQL. The main problem is that there is no
> > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic",
> but
> > can be converted to "perssimistic" optionally, and all writes are
> > "pessimistic". Another problem is that some of our pairs cannot be used
> in
> > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> some
> > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > reason about. If you cannot explain API in 5 mins, it is broken :-)
> >
> > If we were many years in the past allowed to choose between Ignite and
> SQL
> > approaches I would prefer SQL because it is much simpler.
> >
> > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Vladimir,
> > >
> > > This looks like a complete overhaul of our transactional behavior and I
> > do
> > > not think we are at a liberty to make such drastic changes. Can you
> > please
> > > explain why the current behavior would not map to the SQL transactions?
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > Sorry for late reply. I had a chat with several Ignite veterans
> today.
> > We
> > > > tried to design transactional SQL for Ignite. One of our questions
> was
> > > how
> > > > to align SQL transactions with current Ignite transactions. And we
> > > failed.
> > > > And then we came to conclusion that current transaction API is
> > unusable.
> > > We
> > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > very
> > > > counterintuitive and cannot be mapped to any transactional framework
> > our
> > > > users are familiar with.
> > > >
> > > > So we thought how new tx API might looks like, and here is the draft.
> > > >
> > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > > current enum) with three standard modes - READ_COMMITTED,
> > > REPEATABLE_READ,
> > > > SERIALIZABLE.
> > > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > 4) Reads never acuire locks
> > > > 5) Writes always acquire locks
> > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> which
> > > will
> > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > exception
> > > > on any update
> > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> to
> > > > SERIALIZABLE isolation level.
> > > > 9) Define new API methods:
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > TransactionHint... hints)
> > > > 10) Deprecate old TX start methods
> > > >
> > > > As a result we will have simple, clean and extensible API. Which can
> be
> > > > explained to users in 5 minutes, instead of current half an hour. And
> > > which
> > > > is perfectly aligned with upcoming transactional SQL.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Vova,
> > > > >
> > > > > Thanks for doing the research. The changes you are suggesting are a
> > bit
> > > > too
> > > > > bold, so let's discuss them in some more detail...
> > > > >
> > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead,

[jira] [Created] (IGNITE-6501) select distinct exception in simple sqlQuery request.

2017-09-26 Thread Stanilovsky Evgeny (JIRA)

Stanilovsky Evgeny created IGNITE-6501:
--

 Summary: select distinct exception in simple sqlQuery request.
 Key: IGNITE-6501
 URL: https://issues.apache.org/jira/browse/IGNITE-6501
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1, 2.0
Reporter: Stanilovsky Evgeny
Assignee: Vladimir Ozerov


SqlQuery throws exception
{code}
Failed to parse query: SELECT distinct Value._KEY, distinct Value._VAL from 
Value
{code}
while SqlFieldsQuery works properly well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6500) While using ignite-cassandra-store, POJO field having wrapper type, mapped to Cassandra table are getting initialized to respective default value of primitive type inste

2017-09-26 Thread Yashasvi Kotamraju (JIRA)

Yashasvi Kotamraju created IGNITE-6500:
--

 Summary: While using ignite-cassandra-store, POJO field having 
wrapper type, mapped to Cassandra table are getting initialized to respective 
default value of primitive type instead of null if column value is null. 
 Key: IGNITE-6500
 URL: https://issues.apache.org/jira/browse/IGNITE-6500
 Project: Ignite
  Issue Type: Bug
  Components: cassandra
Reporter: Yashasvi Kotamraju
Priority: Minor


While using  ignite-cassandra-store, if a POJO field is of wrapper class type, 
and the column value mapped in cassandra persistent store is null then the POJO 
field is getting set to default primitive type instead of null.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6499) Compact NULL fields binary representation

2017-09-26 Thread Alexandr Kuramshin (JIRA)

Alexandr Kuramshin created IGNITE-6499:
--

 Summary: Compact NULL fields binary representation
 Key: IGNITE-6499
 URL: https://issues.apache.org/jira/browse/IGNITE-6499
 Project: Ignite
  Issue Type: Improvement
  Components: binary
Affects Versions: 2.1
Reporter: Alexandr Kuramshin
Assignee: Vladimir Ozerov


Current compact footer implementation writes offset for the every field in 
schema. Depending on serialized size of an object offset may be 1, 2 or 4 bytes.

Imagine an object with some 100 fields are null. It takes from 100 to 400 bytes 
overhead. For middle-sized objects (about 260 bytes) it doubles the memory 
usage. For a small-sized objects (about 40 bytes) the memory usage increased by 
factor 3 or 4.

Proposed two optimizations, the both should be implemented, the most optimal 
implementation should be selected dynamically upon object marshalling.

1. Write field ID and offset for the only non-null fields in footer.

2. Write footer header then field offsets for the only non-null fields as 
follows

[0] bit mask for first 8 fields, 0 - field is null, 1 - field is non-null
[1] cumulative sum of "1" bits
[2] bit mask for the next 8 fields
[3] cumulative sum of "1" bits
... and so on
[N1...N2] offset of first non-null field
[N3...N4] offset of next non-null field
... and so on

If we want to read fields from 0 to 7, then we read first footer byte, step 
through bits and find the offset index for non-null field or find that field is 
null.

If we want to read fields from 8, then we read two footer bytes, take start 
offset from the first byte, and then step through bits and find the offset 
index for non-null field or find that field is null.

This supports up to 255 non-null fields per binary object.

Overhead would be only 24 bytes per 100 null fields instead of 200 bytes for 
the middle-sized object.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Future of Ignite transactions

2017-09-26 Thread Dmitriy Setrakyan

Vladimir,

I do not think we have a luxury of changing Ignite transaction APIs. It is
almost the same as changing Ignite put and get APIs. Too much code has
already been written to these APIs. The upgrade path will be so onerous
that no one will ever take it.

As far as the current transaction flags, we will have to continue to
support them. Ideally we should extend them to Ignite SQL as well. If there
is a new transaction flag that you wish to add to Ignite SQL, please
suggest it here.

D.

On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov 
wrote:

> Dima,
>
> Becuase if you take any pair of our concurrency/isolation modes, you will
> not find anything similar in SQL. The main problem is that there is no
> "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic", but
> can be converted to "perssimistic" optionally, and all writes are
> "pessimistic". Another problem is that some of our pairs cannot be used in
> practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and some
> duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> reason about. If you cannot explain API in 5 mins, it is broken :-)
>
> If we were many years in the past allowed to choose between Ignite and SQL
> approaches I would prefer SQL because it is much simpler.
>
> On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan 
> wrote:
>
> > Vladimir,
> >
> > This looks like a complete overhaul of our transactional behavior and I
> do
> > not think we are at a liberty to make such drastic changes. Can you
> please
> > explain why the current behavior would not map to the SQL transactions?
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Folks,
> > >
> > > Sorry for late reply. I had a chat with several Ignite veterans today.
> We
> > > tried to design transactional SQL for Ignite. One of our questions was
> > how
> > > to align SQL transactions with current Ignite transactions. And we
> > failed.
> > > And then we came to conclusion that current transaction API is
> unusable.
> > We
> > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> very
> > > counterintuitive and cannot be mapped to any transactional framework
> our
> > > users are familiar with.
> > >
> > > So we thought how new tx API might looks like, and here is the draft.
> > >
> > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > current enum) with three standard modes - READ_COMMITTED,
> > REPEATABLE_READ,
> > > SERIALIZABLE.
> > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > 4) Reads never acuire locks
> > > 5) Writes always acquire locks
> > > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> > will
> > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> exception
> > > on any update
> > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > > SERIALIZABLE isolation level.
> > > 9) Define new API methods:
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > TransactionHint... hints)
> > > 10) Deprecate old TX start methods
> > >
> > > As a result we will have simple, clean and extensible API. Which can be
> > > explained to users in 5 minutes, instead of current half an hour. And
> > which
> > > is perfectly aligned with upcoming transactional SQL.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > > > Vova,
> > > >
> > > > Thanks for doing the research. The changes you are suggesting are a
> bit
> > > too
> > > > bold, so let's discuss them in some more detail...
> > > >
> > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > 2) Exclusive lock on read could be forced explicitly (SELECT ...
> FOR
> > > > > UPDATE)
> > > > > 3) Writes do acuire explicit row locks
> > > > > 4) Locks are always acquired immediately once statement is executed
> > > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely
> on
> > > > > so-called *range locks* (or *predicate locks*) to track
> dependencies
> > > > > between transactions. Some vendors throw an exception in case of
> > > > conflict -
> > > > > these are ones where snapshot-based MVCC is used -

58 matches

Mail list logo