Re: Converting TreeMap to BinaryObject causes ClassCastException (IGNITE-2852)

[Dmitriy Setrakyan]

Well, in my view we should have that discussion or close this ticket. What
are the challenges there? Is there a dev thread I can review?

On Thu, Jul 21, 2016 at 5:41 AM, Denis Magda  wrote:

> Dmitriy,
>
> Presently even the guys who spent bunch the time developing and improving
> binary marshaller don’t have a view on how to implement this feature. So
> initially we should discuss how to implement it in general and after that
> decide if the ticket can be taken over by new contributors.
>
> —
> Denis
>
> > On Jul 21, 2016, at 1:18 PM, Dmitriy Setrakyan 
> wrote:
> >
> > On Wed, Jul 20, 2016 at 2:45 PM, Denis Magda 
> wrote:
> >
> >> Hi Jens,
> >>
> >> This is not the best candidate for the first contribution and presently
> I
> >> don’t think that this feature should be supported at all.
> >>
> >
> > Denis, why not?
> >
> >
> >>
> >> I would recommend you picking up one of the tickets with “newbie”
> filter.
> >> https://issues.apache.org/jira/issues/?filter=12338037 <
> >> https://issues.apache.org/jira/issues/?filter=12338037>
> >>
> >> Regards,
> >> Denis
> >>
> >>> On Jul 19, 2016, at 9:00 PM, NoTrueScotsman <
> no.true.scots...@gmail.com>
> >> wrote:
> >>>
> >>> It happens when applying ignite.binary().toBinary(m) to a TreeMap >>> V> with a custom key K.
> >>>
> >>> I thought this would be a nice task to pick as a first (attempt of a)
> >>> contribution, however there is a ticket for this already assigned to
> >>> someone: https://issues.apache.org/jira/browse/IGNITE-2852
> >>>
> >>> I haven't seen much activity on it though. Would it be possible for me
> >>> to pick it?
> >>>
> >>> Thanks
> >>> Jens
> >>
> >>
>
>

[GitHub] ignite pull request #881: IGNITE-3390: Added system DSN support for Windows.

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/881

IGNITE-3390: Added system DSN support for Windows.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/isapego/ignite ignite-3390

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/881.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #881


commit 26708651eda06851a5f4e76161216d1493576e4f
Author: isapego 
Date:   2016-07-15T18:14:17Z

IGNITE-3390: Window stub implemented.

commit f7a0592611e3987e99eedc9516014a39a1e92db4
Author: isapego 
Date:   2016-07-19T15:24:36Z

IGNITE-3390: Implemented Window class prototype. Needs refactoring.

commit 115da0dda5903d35e0190c161fe4be3d9c691161
Author: isapego 
Date:   2016-07-19T16:37:29Z

IGNITE-3390: Splitted Window in two classes.

commit 0f193568167277817f295a3507c17292bd8a8d3a
Author: isapego 
Date:   2016-07-19T18:09:35Z

IGNITE-3390: Refactoring.

commit 62a5ca7fb736cb74e83cd7f29c16f92394912dcc
Author: isapego 
Date:   2016-07-20T10:21:15Z

IGNITE-3390: Implmented first version of the configuration window.

commit 6f504e959bdd55bd36bbaf161d4821899cff474f
Author: isapego 
Date:   2016-07-20T14:51:36Z

IGNITE-3390: Refactoring.

commit b084124b25953fc2a2516ca301a9619f59d631bf
Author: isapego 
Date:   2016-07-20T17:17:08Z

IGNITE-3390: Implemented retrieval of the configuration parameters.

commit c9b73555b55d1c7430f04730624383caf151ec16
Author: isapego 
Date:   2016-07-21T14:19:10Z

IGNITE-3390: Implemented saving and loading of the DSN.

commit 463c6099b766349c3abde80f6459eefde61e529f
Author: isapego 
Date:   2016-07-21T15:18:50Z

IGNITE-3390: Refactoring.

commit 92b8c72dd57fd91a27054c9a5eb195ddcc92ab18
Author: isapego 
Date:   2016-07-21T15:26:38Z

IGNITE-3390: Minor refactoring of error reporting.

commit 01eac0e3f0eb973996da4c1a6ec16d969750fe83
Author: isapego 
Date:   2016-07-21T15:43:05Z

IGNITE-3390: Implmented connection by the DSN.

commit dee6363c6f4d950954fa589e8dc7d291abf461d5
Author: isapego 
Date:   2016-07-21T15:46:01Z

IGNITE-3390: Improved logs.

commit e045507619b0a5675eb38cfcd8d08467e2036c42
Author: isapego 
Date:   2016-07-21T15:58:32Z

IGNITE-3390: Implemented SQLConnect.

commit 9097ad84231d1567a6ee78b97aabc7edd70f7a55
Author: Igor Sapego 
Date:   2016-07-21T16:26:29Z

IGNITE-3390: Added new files to Autotools build system.

commit 2049a986f9dcecd84d279d66b37bdfd2f5e2be4c
Author: isapego 
Date:   2016-07-21T16:34:00Z

IGNITE-3390: Fixed dsn_config for non-windows platforms.

commit 8e7f402a84415e551df90a7ac3266053a16bb084
Author: isapego 
Date:   2016-07-21T18:26:52Z

IGNITE-3390: Some fine-tuning of the configuration window.

commit 9233823e7268d777e1526052d59c71aeb96ce037
Author: isapego 
Date:   2016-07-21T18:30:40Z

Merge remote-tracking branch 'upstream/master' into ignite-3390




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #880: Ignite 3513

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/880

Ignite 3513



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-3513

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/880.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #880


commit 3e59321ef0ae1d936d94f8f804db45ceeff55844
Author: vozerov-gridgain 
Date:   2016-03-16T09:31:37Z

IGNITE-2842: IGFS: Optimized create/mkdirs operations with help of entry 
processors.

commit f57f3657b1da33abf28f885cd405780dabfd57e3
Author: vozerov-gridgain 
Date:   2016-03-16T10:22:07Z

IGNITE-2846: IGFS: Reworked IgfsMetaManager.updateInfo() operation to use 
"invoke" instead of "put".

commit 8a93c3bf1687e6f2de1a4391d95366d733a44a7d
Author: vozerov-gridgain 
Date:   2016-03-18T13:38:45Z

IGNITE-2860: IGFS: Reworked base meta operations.

commit 8e9e790e482b8911142bf8b21fa3ad7267a62db6
Author: vozerov-gridgain 
Date:   2016-03-18T14:07:58Z

IGNITE-2834: IGFS: Implemented optional metadata co-location.

commit bfa7bf6c3d693406f1dd5121488796687aebbe7d
Author: vozerov-gridgain 
Date:   2016-03-18T14:45:48Z

IGNITE-2813: IGFS: Optimized metadata values splitting file and directory 
into separate classes.

commit 8de40f2f8649c9ffecf86202f9fd4efbc3827e83
Author: thatcoach 
Date:   2016-03-18T18:15:04Z

IGNITE-2860: IGFS: Fixed minor bug in append() operation.

commit 9a4b5bd720c5ed1f96b82a457fa3eaed1bdbb132
Author: thatcoach 
Date:   2016-03-19T18:13:35Z

IGNITE-2861: IGFS: Moved metadata processors into separate top-level 
classes to simplify code. Also cleaned up IgfsMetaManager from unused code.

commit 2cd0dcb37ce43a4cb07885ddfb2e72392fc814a7
Author: vozerov-gridgain 
Date:   2016-03-21T07:29:20Z

IGNITE-2836: IGFS: Ensured that metadata can be serialized through 
BinaryMarshaller in the most efficient way.

commit 76191ff39456a30246df3aca7c026773d00a8446
Author: vozerov-gridgain 
Date:   2016-03-21T07:36:26Z

IGNITE-2861: Added missing Apache headers.

commit ee5ea53bf9c4ad897459466e0b9b5447fc93ec2a
Author: vozerov-gridgain 
Date:   2016-03-22T06:20:32Z

IGNITE-2869: IGFS: Slightly improved serialization of IgfsListingEntry.

commit 218132dc0c3764966294a5f29ad480af4af7b0ff
Author: vozerov-gridgain 
Date:   2016-03-22T06:23:29Z

IGNITE-2868: IGFS: Increased trash concurrency from 16 to 64.

commit e886ad0aa800cddb3308fa5f8400902e5879ee3c
Author: vozerov-gridgain 
Date:   2016-03-22T07:28:13Z

IGNITE-2811: IGFS: Optimized properties handling.

commit 8d95ebacaa01f3f9271a1ce0d1b991dfead1d0c1
Author: vozerov-gridgain 
Date:   2016-03-22T09:06:51Z

IGNITE-2871: IGFS: Removed "path" from IgfsEntryInfo. Purge event is never 
fired now, it will be fixed as a part of IGNITE-1679.

commit b286facc4b8c44ab1628039dded6c7527760df73
Author: vozerov-gridgain 
Date:   2016-03-22T09:34:35Z

IGNITE-2806: IGFS: Implemented relaxed consistency model.

commit 26f115734e7262d4b4b60f1c6016783f67c66986
Author: vozerov-gridgain 
Date:   2016-03-22T09:46:23Z

IGFS: Added misssing "final" modifiers to FileSystemConfiguration defaults.

commit 00a0e4b51c299871ff690bbe6d462cf80dae045e
Author: vozerov-gridgain 
Date:   2016-03-24T07:35:43Z

IGNITE-2878: IGFS: Optimzied serialization of IgfsListingEntry and 
properties map.

commit 01a6e86ec4e19d372b8efbc5c497c84f4238a46c
Author: vozerov-gridgain 
Date:   2016-03-24T10:28:30Z

IGNITE-2876: Fixed possible starvation in system pool caused by 
IgfsBlockMessage. This closes #575.

commit 59705e008267b1d5926410f95c68bb8ffb8cd93c
Author: vozerov-gridgain 
Date:   2016-03-24T11:29:35Z

IGNITE-2883: IGFS: Now IPC messages are handled in a dedicated thread pool.

commit 0412c9bfd89b8d2a377588e908f1012c845ac5bc
Author: Alexey Kuznetsov 
Date:   2016-03-30T04:43:19Z

Added detail info about keys count in partitions for offheap and swap.

commit 28d2a7bf7f35ec4b51fba872ace47cdbc255ded8
Author: Alexey Kuznetsov 
Date:   2016-03-30T07:42:12Z

Minor change to Visor classes: added setters for name and queryMetrics 
properties.

commit a70d2dc48c42f23b1ea64c2a219560efa44fb99f
Author: Alexey Kuznetsov 
Date:   2016-04-04T07:14:42Z

Minor fix for Visor query metrics.

commit ec2ec9965ae03ef3666b2035a13f444cf2765aec
Author: dkarachentsev 

Re: Ignite message ping metric

[Dmitriy Setrakyan]

To my knowledge, the GridGain Web Console [1] already does have the PING
button, which does exactly what you are asking. It’s not part of Ignite per
se, but it is still free for everyone to use.

[1] - https://console.gridgain.com/

On Thu, Jul 21, 2016 at 11:11 AM, AndreyVel  wrote:

> Hi everyone,
>
> What do you think if to add ignite message ping metric between nodes and
> display it in monitoring tools?
> This metric can be displayed as matrix [All nodes] * [All nodes]
>
> Sometime to hard find, why cluster has bad performance, one of reason can
> be bad network performance between same nodes.
> One bad network card or network configuration can kill performance whole
> cluster.
>

Re: IGNITE-2294 implementation details

[Dmitriy Setrakyan]

OK, then using your analogy, the current behavior in Ignite is MERGE for
the most part.

My preference is that Ignite SQL should work no different from traditional
databases, which means:

- INSERT is translated into *putIfAbsent()* call in Ignite
- UPDATE is translated into *replace()* call in Ignite
- MERGE is translated into *put()* call in Ignite
- For SQL BATCH calls we should delegate to Ignite batch operations, e.g.
*putAll()*

The above should hold true for atomic and transactional put/putAll calls,
as well as for the data streamer.

Does this make sense?

D.

On Thu, Jul 21, 2016 at 4:06 AM, Sergi Vladykin 
wrote:

> No, this does not make sense.
>
> There is no upsert mode in databases. There are operations: INSERT, UPDATE,
> DELETE, MERGE.
>
> I want to have clear understanding of how they have to behave in SQL
> databases and how they will actually behave in Ignite in different
> scenarios. Also I want to have clear understanding of performance
> implications of each decision here.
>
> Anything wrong with that?
>
> Sergi
>
> On Thu, Jul 21, 2016 at 1:04 PM, Dmitriy Setrakyan 
> wrote:
>
> > Serj, are you asking what will happen as of today? Then the answer to all
> > your questions is that duplicate keys are not an issue, and Ignite always
> > operates in **upsert** mode (which is essentially a *“put(…)” *method).
> >
> > However, the *“insert”* that is suggested by Alex would delegate to
> > *“putIfAbsent(…)”*, which in database world makes more sense. However, in
> > this case, the *“update”* syntax should delegate to *“replace(…)”*, as
> > update should fail in case if a key is absent.
> >
> > Considering the above, a notion of “*upsert”* or “*merge” *operation is
> > very much needed, as it will give a user an option to perform
> > “insert-or-update” in 1 call.
> >
> > Does this make sense?
> >
> > D.
> >
> > On Wed, Jul 20, 2016 at 9:39 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > I'd prefer to do MERGE operation last because in H2 it is not standard
> > ANSI
> > > SQL MERGE. Or may be not implement it at all, or may be contribute ANSI
> > > correct version to H2, then implement it on Ignite. Need to investigate
> > the
> > > semantics deeper before making any decisions here.
> > >
> > > Lets start with simple scenarios for INSERT and go through all the
> > possible
> > > cases and answer the questions:
> > > - What will happen on key conflict in TX cache?
> > > - What will happen on key conflict in Atomic cache?
> > > - What will happen with the previous two if we use DataLoader?
> > > - How to make these operations efficient (it will be simple enough to
> > > implement them with separate put/putIfAbsent operations but probably we
> > > will need some batching like putAllIfAbsent for efficiency)?
> > >
> > > As for API, we still will need to have a single entry point for all SQL
> > > queries/commands to allow any console work with it transparently. It
> > would
> > > be great if we will be able to come up with something consistent with
> > this
> > > idea on public API.
> > >
> > > Sergi
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 20, 2016 at 2:23 PM, Dmitriy Setrakyan <
> > > dsetrak...@gridgain.com>
> > > wrote:
> > >
> > > > Like the idea of merge and insert. I need more time to think about
> the
> > > API
> > > > changes.
> > > >
> > > > Sergi, what do you think?
> > > >
> > > > Dmitriy
> > > >
> > > >
> > > >
> > > > On Jul 20, 2016, at 12:36 PM, Alexander Paschenko <
> > > > alexander.a.pasche...@gmail.com> wrote:
> > > >
> > > > >> Thus, I suggest that we implement MERGE as a separate operation
> > backed
> > > > by putIfAbsent operation, while INSERT will be implemented via put.
> > > > >
> > > > > Sorry, of course I meant that MERGE has to be put-based, while
> INSERT
> > > > > has to be putIfAbsent-based.
> > > > >
> > > > > 2016-07-20 12:30 GMT+03:00 Alexander Paschenko
> > > > > :
> > > > >> Hell Igniters,
> > > > >>
> > > > >> In this thread I would like to share and discuss some thoughts on
> > DML
> > > > >> operations' implementation, so let's start and keep it here.
> > Everyone
> > > > >> is of course welcome to share their suggestions.
> > > > >>
> > > > >> For starters, I was thinking about semantics of INSERT. In
> > traditional
> > > > >> RDBMSs, INSERT works only for records whose primary keys don't
> > > > >> conflict with those of records that are already persistent - you
> > can't
> > > > >> try to insert the same key more than once because you'll get an
> > error.
> > > > >> However, semantics of cache put is obviously different - it does
> not
> > > > >> have anything about duplicate keys, it just quietly updates values
> > in
> > > > >> case of keys' duplication. Still, cache has putIfAbsent operation
> > that
> > > > >> is closer to traditional notion of INSERT, and H2's SQL dialect
> has
> > > > >> MERGE operation which 

Ignite message ping metric

[AndreyVel]

Hi everyone,

What do you think if to add ignite message ping metric between nodes and 
display it in monitoring tools?

This metric can be displayed as matrix [All nodes] * [All nodes]

Sometime to hard find, why cluster has bad performance, one of reason 
can be bad network performance between same nodes.
One bad network card or network configuration can kill performance whole 
cluster.

Re: IGNITE-2294 implementation details

[Alexander Paschenko]

I agree. It looks like we first of all need some algorithm of key
generation for new objects, and it does not necessarily have to be
involved with DDL. The first suggestion that comes to my mind on that
matter is, obviously, marking some method of the class persisted with
magical annotation - so that the value will be able to supply its own
key when needed.

Alex

2016-07-21 17:11 GMT+03:00 Sergi Vladykin :
> I think these problems we need to solve regardless of DDL.
>
> Sergi
>
> On 21 июля 2016 г., 16:40, Alexey Goncharuk 
> wrote:
>
>> Suppose I have a PersonKey {int id;} class and Person {@SqlQueryField int
>> id; @SqlQueryField String name; @SqlQueryField int age;} class.
>>
>> How would an insert statement look like?
>> INSERT INTO Person (_key, _val) values (...)
>> INSERT INTO Person (_key, _val, id, name, age) values (...) -> Obviously
>> will not be usable from any console unless we are able to parse "new
>> PersonKey(1)" statements.
>>
>> INSERT INTO Person (id, name, age) values (...) -> How do we know how to
>> construct the PersonKey? Most likely I am missing something, but this is
>> not clear for me so far.
>>

Re: Apache Ignite - New Release

[Alexandre Boudnik]

Sorry, I missed the typo:
To support the point, I would add that PosgreSQL has demonstrated the
similar behavior (inspired by unix .dot files and ls):
- they have "hidden" column: xmin and xmax in any table
- they do not appear in select * list unless they are specified explicitly
Take care,
Alexandre "Sasha" Boudnik

call me via Google Voice:
1(405) BUDNIKA
1(405) 283-6452



On Thu, Jul 21, 2016 at 12:46 PM, Alexandre Boudnik
 wrote:
> To support the point, I would add that PosgreSQL has demonstrated the
> similar behavior (inspired by unix .dot files and ls):
> - they have "hidden" column: xmin and xmax in any table
> - they appear in select * list unless they are specified explicitly
>
> I'll add some notices to the ticket
> Take care,
> Alexandre "Sasha" Boudnik
>
> call me via Google Voice:
> 1(405) BUDNIKA
> 1(405) 283-6452
>
>
>
> On Fri, Jul 15, 2016 at 6:37 PM, Valentin Kulichenko
>  wrote:
>> Agree.
>>
>> On Fri, Jul 15, 2016 at 12:59 PM, Alexey Goncharuk <
>> alexey.goncha...@gmail.com> wrote:
>>
>>> Looks like the ticket for removing _key and _value from selct * is a good
>>> candidate for 2.0.
>>>
>>> 2016-07-15 5:12 GMT-07:00 Sergi Vladykin :
>>>
>>> > We will not be able to just change this, because it will brake
>>> > compatibility. Still I believe that an option to define our SQL tables
>>> > without _key and _value fields. But this is another story you can file a
>>> > ticket for it, can we fix somehow our JDBC for now? Like returning
>>> > BinaryObject instance or something?
>>> >
>>> > Sergi
>>> >
>>> >
>>> >
>>> > On Fri, Jul 15, 2016 at 2:48 AM, Valentin Kulichenko <
>>> > valentin.kuliche...@gmail.com> wrote:
>>> >
>>> > > IGNITE-3466 is not a JDBC-only issue. This happens because 'select *'
>>> > query
>>> > > returns all the fields including _kay and _val which are created by
>>> > Ignite,
>>> > > not by user. This is actually a usability issue that pops up every now
>>> > and
>>> > > then. This is very counterintuitive that we return the fields that user
>>> > > never defined (unless he explicitly asks for them, of course) and that
>>> > > 'select *' requires class definitions on the client.
>>> > >
>>> > > Is it possible to fix this on SQL engine level instead of fixing only
>>> for
>>> > > JDBC? Sergi, what do you think?
>>> > >
>>> > > -Val
>>> > >
>>> > > On Thu, Jul 14, 2016 at 3:28 AM, Sergi Vladykin <
>>> > sergi.vlady...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > All these issues seem to be related to Jdbc driver rather than to SQL
>>> > > > engine. I think Andrey Gura was the last who worked on it. IMO they
>>> > must
>>> > > be
>>> > > > easy to fix.
>>> > > >
>>> > > > Sergi
>>> > > >
>>> > > > On Thu, Jul 14, 2016 at 9:06 AM, Denis Magda 
>>> > > wrote:
>>> > > >
>>> > > > > Yakov,
>>> > > > >
>>> > > > > I'm not the one who is eligible for review of IGNITE-3389. Assigned
>>> > it
>>> > > on
>>> > > > > Andrey Gura. Andrey please find time for review.
>>> > > > > Alexander B. when you need a review please send an email to the dev
>>> > > list
>>> > > > > and someone will assist you.
>>> > > > >
>>> > > > > As for IGNITE-3466, IGNITE-3467 and 3468 Sergi's opinion is needed.
>>> > > Sergi
>>> > > > > please have a look.
>>> > > > >
>>> > > > > --
>>> > > > > Denis
>>> > > > >
>>> > > > >
>>> > > > > On Wed, Jul 13, 2016 at 12:13 PM, Yakov Zhdanov <
>>> yzhda...@apache.org
>>> > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Sasha, ignite-3389 is in resolved state and I suppose is ready to
>>> > be
>>> > > > > > reviewed and merged. Denis, can you please do it? Make sure to
>>> > check
>>> > > TC
>>> > > > > :)
>>> > > > > >
>>> > > > > > As far as Ignite-3466..3468, Igor, can you please provide
>>> feedback
>>> > to
>>> > > > the
>>> > > > > > issues and tell us if we can fit them as well.
>>> > > > > >
>>> > > > > > --Yakov
>>> > > > > >
>>> > > > > > 2016-07-12 23:33 GMT+03:00 Alexandre Boudnik <
>>> > > > > alexander.boud...@gmail.com
>>> > > > > > >:
>>> > > > > >
>>> > > > > > > Yakov,
>>> > > > > > >
>>> > > > > > > Is it possible to include several probably easy-to-fix bugs and
>>> > > > > > > improvements; they are very annoying and they decrease the
>>> value
>>> > of
>>> > > > > > > the product? We're working on Apache Ignite based BI solution
>>> > > > > > > accelerator, and these issues impact us.
>>> > > > > > >
>>> > > > > > > I've opened them and I fix one and working on others.
>>> > > > > > >
>>> > > > > > > IGNITE-3389 - metadata result set throws NPE when closed - Pull
>>> > > > Request
>>> > > > > > > #838
>>> > > > > > > IGNITE-3466 - select * causes NoClassDefFoundError with jdbc
>>> > query
>>> > > > > tools
>>> > > > > > > IGNITE-3467 - jdbc getTables() returns catalog as null
>>> > > > > > > IGNITE-3468 - Missing Primary Key flag in getColumns()
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > 

Re: Apache Ignite - New Release

[Alexandre Boudnik]

To support the point, I would add that PosgreSQL has demonstrated the
similar behavior (inspired by unix .dot files and ls):
- they have "hidden" column: xmin and xmax in any table
- they appear in select * list unless they are specified explicitly

I'll add some notices to the ticket
Take care,
Alexandre "Sasha" Boudnik

call me via Google Voice:
1(405) BUDNIKA
1(405) 283-6452



On Fri, Jul 15, 2016 at 6:37 PM, Valentin Kulichenko
 wrote:
> Agree.
>
> On Fri, Jul 15, 2016 at 12:59 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
>> Looks like the ticket for removing _key and _value from selct * is a good
>> candidate for 2.0.
>>
>> 2016-07-15 5:12 GMT-07:00 Sergi Vladykin :
>>
>> > We will not be able to just change this, because it will brake
>> > compatibility. Still I believe that an option to define our SQL tables
>> > without _key and _value fields. But this is another story you can file a
>> > ticket for it, can we fix somehow our JDBC for now? Like returning
>> > BinaryObject instance or something?
>> >
>> > Sergi
>> >
>> >
>> >
>> > On Fri, Jul 15, 2016 at 2:48 AM, Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com> wrote:
>> >
>> > > IGNITE-3466 is not a JDBC-only issue. This happens because 'select *'
>> > query
>> > > returns all the fields including _kay and _val which are created by
>> > Ignite,
>> > > not by user. This is actually a usability issue that pops up every now
>> > and
>> > > then. This is very counterintuitive that we return the fields that user
>> > > never defined (unless he explicitly asks for them, of course) and that
>> > > 'select *' requires class definitions on the client.
>> > >
>> > > Is it possible to fix this on SQL engine level instead of fixing only
>> for
>> > > JDBC? Sergi, what do you think?
>> > >
>> > > -Val
>> > >
>> > > On Thu, Jul 14, 2016 at 3:28 AM, Sergi Vladykin <
>> > sergi.vlady...@gmail.com>
>> > > wrote:
>> > >
>> > > > All these issues seem to be related to Jdbc driver rather than to SQL
>> > > > engine. I think Andrey Gura was the last who worked on it. IMO they
>> > must
>> > > be
>> > > > easy to fix.
>> > > >
>> > > > Sergi
>> > > >
>> > > > On Thu, Jul 14, 2016 at 9:06 AM, Denis Magda 
>> > > wrote:
>> > > >
>> > > > > Yakov,
>> > > > >
>> > > > > I'm not the one who is eligible for review of IGNITE-3389. Assigned
>> > it
>> > > on
>> > > > > Andrey Gura. Andrey please find time for review.
>> > > > > Alexander B. when you need a review please send an email to the dev
>> > > list
>> > > > > and someone will assist you.
>> > > > >
>> > > > > As for IGNITE-3466, IGNITE-3467 and 3468 Sergi's opinion is needed.
>> > > Sergi
>> > > > > please have a look.
>> > > > >
>> > > > > --
>> > > > > Denis
>> > > > >
>> > > > >
>> > > > > On Wed, Jul 13, 2016 at 12:13 PM, Yakov Zhdanov <
>> yzhda...@apache.org
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Sasha, ignite-3389 is in resolved state and I suppose is ready to
>> > be
>> > > > > > reviewed and merged. Denis, can you please do it? Make sure to
>> > check
>> > > TC
>> > > > > :)
>> > > > > >
>> > > > > > As far as Ignite-3466..3468, Igor, can you please provide
>> feedback
>> > to
>> > > > the
>> > > > > > issues and tell us if we can fit them as well.
>> > > > > >
>> > > > > > --Yakov
>> > > > > >
>> > > > > > 2016-07-12 23:33 GMT+03:00 Alexandre Boudnik <
>> > > > > alexander.boud...@gmail.com
>> > > > > > >:
>> > > > > >
>> > > > > > > Yakov,
>> > > > > > >
>> > > > > > > Is it possible to include several probably easy-to-fix bugs and
>> > > > > > > improvements; they are very annoying and they decrease the
>> value
>> > of
>> > > > > > > the product? We're working on Apache Ignite based BI solution
>> > > > > > > accelerator, and these issues impact us.
>> > > > > > >
>> > > > > > > I've opened them and I fix one and working on others.
>> > > > > > >
>> > > > > > > IGNITE-3389 - metadata result set throws NPE when closed - Pull
>> > > > Request
>> > > > > > > #838
>> > > > > > > IGNITE-3466 - select * causes NoClassDefFoundError with jdbc
>> > query
>> > > > > tools
>> > > > > > > IGNITE-3467 - jdbc getTables() returns catalog as null
>> > > > > > > IGNITE-3468 - Missing Primary Key flag in getColumns()
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Take care,
>> > > > > > > Alexandre "Sasha" Boudnik
>> > > > > > >
>> > > > > > > call me via Google Voice:
>> > > > > > > 1(405) BUDNIKA
>> > > > > > > 1(405) 283-6452
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jul 11, 2016 at 11:46 AM, Yakov Zhdanov <
>> > > yzhda...@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > Guys,
>> > > > > > > >
>> > > > > > > > We have recently found and fixed several issues in the
>> product
>> > > > along
>> > > > > > with
>> > > > > > > > number of smaller fixes and optimizations and I would like to
>> > > > release
>> > > > > > > these
>> > 

[GitHub] ignite pull request #879: IGNITE-3512 .NET: IBinaryObject.ToBuilder loses ty...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/879

IGNITE-3512 .NET: IBinaryObject.ToBuilder loses type name



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-3512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/879.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #879


commit 11c7e8fe53461f84924e5c29893865191f236c8c
Author: Anton Vinogradov 
Date:   2016-07-20T08:31:14Z

Documentation fix

commit 9b55658749d0e2a869bbb3614034d8aa1f0e95c1
Author: vozerov-gridgain 
Date:   2016-07-20T11:14:50Z

IGNITE-3405: IGFS: Restricted path modes interleaving, so that now only 
DUAL -> PRIMARY and DUAL -> PROXY paths are possible.

commit 6c5218f4d67c8e247f59dbe8deb58b51db2954a2
Author: vozerov-gridgain 
Date:   2016-07-20T11:15:11Z

Merge remote-tracking branch 'upstream/gridgain-7.6.2' into gridgain-7.6.2

commit c25cd4600bd7254d051048034ad4781deb833aae
Author: Pavel Tupitsyn 
Date:   2016-07-21T14:11:12Z

IGNITE-3512 .NET: IBinaryObject.ToBuilder loses type name

commit e4a95a3927d8cac0dd2839d4f483f50b691015cc
Author: Pavel Tupitsyn 
Date:   2016-07-21T14:15:01Z

wip

commit 0e8547b81033fc3fe053121460830fde22b88529
Author: Pavel Tupitsyn 
Date:   2016-07-21T14:35:48Z

Fix metadata propagation

commit 3a7f35ea56a320cccfd275cafdef557764c59d14
Author: Pavel Tupitsyn 
Date:   2016-07-21T14:36:25Z

wip

commit 77e706ce5e4541d1c4c555caca647574e650e607
Author: Pavel Tupitsyn 
Date:   2016-07-21T14:39:54Z

wip

commit 14d99f50e906490bc5342dd673c520fc9cb5b033
Author: Pavel Tupitsyn 
Date:   2016-07-21T15:23:47Z

synopsis

commit 8af51e7013123388b122c5618c71254ce63d740c
Author: Pavel Tupitsyn 
Date:   2016-07-21T15:32:23Z

Fix meta update

commit b3152299deb10d3a4d89f8e288bafd115904aae5
Author: Pavel Tupitsyn 
Date:   2016-07-21T15:43:53Z

wip




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IGNITE-3399 ready for review

[NoTrueScotsman]

Thanks
Jens

Re: IGNITE-2294 implementation details

[Alexey Goncharuk]

Suppose I have a PersonKey {int id;} class and Person {@SqlQueryField int
id; @SqlQueryField String name; @SqlQueryField int age;} class.

How would an insert statement look like?
INSERT INTO Person (_key, _val) values (...)
INSERT INTO Person (_key, _val, id, name, age) values (...) -> Obviously
will not be usable from any console unless we are able to parse "new
PersonKey(1)" statements.

INSERT INTO Person (id, name, age) values (...) -> How do we know how to
construct the PersonKey? Most likely I am missing something, but this is
not clear for me so far.

[jira] [Created] (IGNITE-3535) Hadoop: make IgniteHadoopWeightedMapReducePlanner default one.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3535:
---

 Summary: Hadoop: make IgniteHadoopWeightedMapReducePlanner default 
one.
 Key: IGNITE-3535
 URL: https://issues.apache.org/jira/browse/IGNITE-3535
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Most probably we should even remove {{IgniteHadoopMapReducePlanner}} 
implementation as it is too inefficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3534) Hadoop: create factory for UserNameMapper.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3534:
---

 Summary: Hadoop: create factory for UserNameMapper.
 Key: IGNITE-3534
 URL: https://issues.apache.org/jira/browse/IGNITE-3534
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


We need to be compliant for other parts of Ignite API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3533) Hadoop: create factory for map-reduce planner.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3533:
---

 Summary: Hadoop: create factory for map-reduce planner.
 Key: IGNITE-3533
 URL: https://issues.apache.org/jira/browse/IGNITE-3533
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


We need it to be compliant with other part of Ignite API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3532) Hadoop: Move HadoopMapReducePlanner interface to public space.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3532:
---

 Summary: Hadoop: Move HadoopMapReducePlanner interface to public 
space.
 Key: IGNITE-3532
 URL: https://issues.apache.org/jira/browse/IGNITE-3532
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently {{HadoopMapReducePlanner}} is located inside private package, but is 
exposed to {{HadoopConfiguration}}. 

We must move this interface to public package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

I don't see why we need to start with DDL. We have table definitions with
key and value definitions in them, so what is the problem?

Sergi

On Thu, Jul 21, 2016 at 2:14 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> For me the main question here is how we define Key and Value for a cache
> when using SQL interface. Unless we are going to re-architect the whole
> Ignite, it will still be a key-value storage in the first place, so
> whenever we do an INSERT, we must generate Key and Value. Given that values
> in INSERT are supposed to be fields of either Key or Value, this confuses
> me a lot.
>
> Maybe we should come up with DDL first?
>
> 2016-07-21 14:06 GMT+03:00 Sergi Vladykin :
>
> > No, this does not make sense.
> >
> > There is no upsert mode in databases. There are operations: INSERT,
> UPDATE,
> > DELETE, MERGE.
> >
> > I want to have clear understanding of how they have to behave in SQL
> > databases and how they will actually behave in Ignite in different
> > scenarios. Also I want to have clear understanding of performance
> > implications of each decision here.
> >
> > Anything wrong with that?
> >
> > Sergi
> >
> > On Thu, Jul 21, 2016 at 1:04 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Serj, are you asking what will happen as of today? Then the answer to
> all
> > > your questions is that duplicate keys are not an issue, and Ignite
> always
> > > operates in **upsert** mode (which is essentially a *“put(…)” *method).
> > >
> > > However, the *“insert”* that is suggested by Alex would delegate to
> > > *“putIfAbsent(…)”*, which in database world makes more sense. However,
> in
> > > this case, the *“update”* syntax should delegate to *“replace(…)”*, as
> > > update should fail in case if a key is absent.
> > >
> > > Considering the above, a notion of “*upsert”* or “*merge” *operation is
> > > very much needed, as it will give a user an option to perform
> > > “insert-or-update” in 1 call.
> > >
> > > Does this make sense?
> > >
> > > D.
> > >
> > > On Wed, Jul 20, 2016 at 9:39 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > I'd prefer to do MERGE operation last because in H2 it is not
> standard
> > > ANSI
> > > > SQL MERGE. Or may be not implement it at all, or may be contribute
> ANSI
> > > > correct version to H2, then implement it on Ignite. Need to
> investigate
> > > the
> > > > semantics deeper before making any decisions here.
> > > >
> > > > Lets start with simple scenarios for INSERT and go through all the
> > > possible
> > > > cases and answer the questions:
> > > > - What will happen on key conflict in TX cache?
> > > > - What will happen on key conflict in Atomic cache?
> > > > - What will happen with the previous two if we use DataLoader?
> > > > - How to make these operations efficient (it will be simple enough to
> > > > implement them with separate put/putIfAbsent operations but probably
> we
> > > > will need some batching like putAllIfAbsent for efficiency)?
> > > >
> > > > As for API, we still will need to have a single entry point for all
> SQL
> > > > queries/commands to allow any console work with it transparently. It
> > > would
> > > > be great if we will be able to come up with something consistent with
> > > this
> > > > idea on public API.
> > > >
> > > > Sergi
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jul 20, 2016 at 2:23 PM, Dmitriy Setrakyan <
> > > > dsetrak...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Like the idea of merge and insert. I need more time to think about
> > the
> > > > API
> > > > > changes.
> > > > >
> > > > > Sergi, what do you think?
> > > > >
> > > > > Dmitriy
> > > > >
> > > > >
> > > > >
> > > > > On Jul 20, 2016, at 12:36 PM, Alexander Paschenko <
> > > > > alexander.a.pasche...@gmail.com> wrote:
> > > > >
> > > > > >> Thus, I suggest that we implement MERGE as a separate operation
> > > backed
> > > > > by putIfAbsent operation, while INSERT will be implemented via put.
> > > > > >
> > > > > > Sorry, of course I meant that MERGE has to be put-based, while
> > INSERT
> > > > > > has to be putIfAbsent-based.
> > > > > >
> > > > > > 2016-07-20 12:30 GMT+03:00 Alexander Paschenko
> > > > > > :
> > > > > >> Hell Igniters,
> > > > > >>
> > > > > >> In this thread I would like to share and discuss some thoughts
> on
> > > DML
> > > > > >> operations' implementation, so let's start and keep it here.
> > > Everyone
> > > > > >> is of course welcome to share their suggestions.
> > > > > >>
> > > > > >> For starters, I was thinking about semantics of INSERT. In
> > > traditional
> > > > > >> RDBMSs, INSERT works only for records whose primary keys don't
> > > > > >> conflict with those of records that are already persistent - you
> > > can't
> > > > > >> try to insert the same key more than once because you'll get an
> > > error.
> > > > > >> However, semantics of cache put is 

[jira] [Created] (IGNITE-3531) IGFS: Rename format() to clear().

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3531:
---

 Summary: IGFS: Rename format() to clear().
 Key: IGNITE-3531
 URL: https://issues.apache.org/jira/browse/IGNITE-3531
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently format() method is used to quickly clear all IGFS data without 
touching secondary file system. This way it is equal to {{IgniteCache.clear()}} 
semantics. 

Let's rename format -> clear for consistency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3530) IGFS: "setTimes" method is missing from IgfsSecondaryFileSystem interface.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3530:
---

 Summary: IGFS: "setTimes" method is missing from 
IgfsSecondaryFileSystem interface.
 Key: IGNITE-3530
 URL: https://issues.apache.org/jira/browse/IGNITE-3530
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


It is simply not implemented! Let's add it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3529) IGFS: Simplify meta and data cache configuration.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3529:
---

 Summary: IGFS: Simplify meta and data cache configuration.
 Key: IGNITE-3529
 URL: https://issues.apache.org/jira/browse/IGNITE-3529
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently IGFS configuration is rather complex because user have to manually do 
the following:
1) Configure meta cache
2) Configure data cache 
3) Wire them up with IGFS through {{FileSystemConfiguration.*CacheName}} 
properties. 

Instead, I propose to do the following:
1) Add two properties directly to {{FileSystemConfiguration}}:
- dataCacheConfiguration
- metaCacheConfiguration

2) Names of these cache will be ignored and overwritten with cache name unique 
to concrete IGFS . E.g. *_data and *_meta, where * is IGFS name.

3) *THE MOST IMPORTANT THING* - provide sensible defaults. 

This way normally user will not bother about cache config at all. He will only 
need to add IGFS config bean and set it's name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3528) IGFS: Review IgfsSecondaryFileSystem.open() method return type.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3528:
---

 Summary: IGFS: Review IgfsSecondaryFileSystem.open() method return 
type.
 Key: IGNITE-3528
 URL: https://issues.apache.org/jira/browse/IGNITE-3528
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently we return weird {{IgfsSecondaryFileSystemPositionedReadable}} 
interface. It's goal is clear - we need to have a seekable stream. Need to 
review it accurately and decide whether any changes or renamings are required 
here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3527) IGFS: Review "rename", "delete" and "mkdirs" return types in IgniteFileSystem and IgfsSecondaryFileSystem:

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3527:
---

 Summary: IGFS: Review "rename", "delete" and "mkdirs" return types 
in IgniteFileSystem and IgfsSecondaryFileSystem:
 Key: IGNITE-3527
 URL: https://issues.apache.org/jira/browse/IGNITE-3527
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently their semantics are not clear:
- boolean delete()
- void mkdirs()
- void rename();

They all must have the same return type and semantics. The question is which 
semantics to choose:
1) Return boolean and (almost) never throw exceptions. This is "JDK way". I 
personally do not like it because user will have to both check for true/false 
and use try/catch.
2) Return void and throw exception if something went wrong. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3526) IGFS: Review "perNodeBatchSize" and "perNodeParallelBatchCount" properties.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3526:
---

 Summary: IGFS: Review "perNodeBatchSize" and 
"perNodeParallelBatchCount" properties.
 Key: IGNITE-3526
 URL: https://issues.apache.org/jira/browse/IGNITE-3526
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


These properties are extremely weird form user perspective. We need to review 
how they are actually used inside IGFS code and either remove them or refactor 
to something more useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3525) IGFS: Move fragmentizer-related properties into a separate class.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3525:
---

 Summary: IGFS: Move fragmentizer-related properties into a 
separate class.
 Key: IGNITE-3525
 URL: https://issues.apache.org/jira/browse/IGNITE-3525
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Let's move the following properties there:
- fragmentizerConccurrentFiles;
- fragmentizerThrottlingDelay;
- fragmentizerThrottlingBlockLength.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3524) IGFS: Do not allow nulls for FileSystemConfiguration.name.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3524:
---

 Summary: IGFS: Do not allow nulls for FileSystemConfiguration.name.
 Key: IGNITE-3524
 URL: https://issues.apache.org/jira/browse/IGNITE-3524
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


As a part of our general approach, let's do not allow nulls as IGFS names.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3523) Remove "initialize default path modes" feature.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3523:
---

 Summary: Remove "initialize default path modes" feature.
 Key: IGNITE-3523
 URL: https://issues.apache.org/jira/browse/IGNITE-3523
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently IGFS can create several paths by default, which will forcefully work 
in different modes. This will never require in practice, but caused some 
problems, e.g. performance degradation in our Hadoop FileSystem implementaions

Let's just remove that feature along with relevant property in 
{{FileSystemConfiguration}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3522) IGFS: Rename "streamBufferSize" to "bufferSize" in FileSystemConfiguration.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3522:
---

 Summary: IGFS: Rename "streamBufferSize" to "bufferSize" in 
FileSystemConfiguration.
 Key: IGNITE-3522
 URL: https://issues.apache.org/jira/browse/IGNITE-3522
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


It will look more consistent with "blockSize" property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3521) IGFS: Remove "max space" notion.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3521:
---

 Summary: IGFS: Remove "max space" notion.
 Key: IGNITE-3521
 URL: https://issues.apache.org/jira/browse/IGNITE-3521
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


We have "max space" concept in IGFS which governs maximum amount of local data 
available for IGFS. This concept looks a bit weird because we do not have the 
same thing in caches. 
Moreover, we have several conflicting configuration parameters:
1) {{IgfsPerBlockLruEvictionPolicy}} where we also can specify maximum size.
2) {{CacheConfiguration.offheapMaxMemory}} which also governs evictions.

It looks like we should simply remove "max space" property from IGFS 
configuration and do not control it anyhow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3520) IGFS: Remove task execution methods.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3520:
---

 Summary: IGFS: Remove task execution methods.
 Key: IGNITE-3520
 URL: https://issues.apache.org/jira/browse/IGNITE-3520
 Project: Ignite
  Issue Type: Task
  Components: IGFS
Affects Versions: 1.6
Reporter: Vladimir Ozerov
 Fix For: 2.0


We have several task execution methods on IGFS API. They were never used in 
practice because normally users will achieve the same things using Hadoop or 
Spark frameworks. 

I propose to remove them altogether. This way we will be able to focus solely 
on file system semantics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Converting TreeMap to BinaryObject causes ClassCastException (IGNITE-2852)

[Denis Magda]

Dmitriy,

Presently even the guys who spent bunch the time developing and improving 
binary marshaller don’t have a view on how to implement this feature. So 
initially we should discuss how to implement it in general and after that 
decide if the ticket can be taken over by new contributors.

—
Denis

> On Jul 21, 2016, at 1:18 PM, Dmitriy Setrakyan  wrote:
> 
> On Wed, Jul 20, 2016 at 2:45 PM, Denis Magda  wrote:
> 
>> Hi Jens,
>> 
>> This is not the best candidate for the first contribution and presently I
>> don’t think that this feature should be supported at all.
>> 
> 
> Denis, why not?
> 
> 
>> 
>> I would recommend you picking up one of the tickets with “newbie” filter.
>> https://issues.apache.org/jira/issues/?filter=12338037 <
>> https://issues.apache.org/jira/issues/?filter=12338037>
>> 
>> Regards,
>> Denis
>> 
>>> On Jul 19, 2016, at 9:00 PM, NoTrueScotsman 
>> wrote:
>>> 
>>> It happens when applying ignite.binary().toBinary(m) to a TreeMap>> V> with a custom key K.
>>> 
>>> I thought this would be a nice task to pick as a first (attempt of a)
>>> contribution, however there is a ticket for this already assigned to
>>> someone: https://issues.apache.org/jira/browse/IGNITE-2852
>>> 
>>> I haven't seen much activity on it though. Would it be possible for me
>>> to pick it?
>>> 
>>> Thanks
>>> Jens
>> 
>> 

Re: New contributor: Support primitive type names in QueryEntity (IGNITE-3399)

[NoTrueScotsman]

Thanks Semyon.

I created a pull request a short while ago, however I am not quite
sure what I need to do in TeamCity. Do I need to register and run the
test suite that I changed, or can I move the ticket to ready for
review?

Thanks
Jens

On Thu, Jul 21, 2016 at 1:07 PM, Semyon Boikov  wrote:
> Hi,
>
> I added you in the contributors list.
>
> On Thu, Jul 21, 2016 at 1:54 PM, NoTrueScotsman 
> wrote:
>
>> Hi all,
>>
>> I'd like to attempt a fix for IGNITE-3399 (currently unclaimed in
>> Jira) as my newbie contribution.
>>
>> Could you add me to the list of Ignite contributors?
>> Jira username: thehoff
>>
>> Cheers
>> Jens
>>

[GitHub] ignite pull request #878: IGNITE-3399 Add support for primitive type names i...

[jayho]

GitHub user jayho opened a pull request:

https://github.com/apache/ignite/pull/878

IGNITE-3399 Add support for primitive type names in QueryEntity

- add optional primitive type mapping to class resolution
- add test to IgniteCacheQuerySelfTestSuite

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jayho/ignite ignite-3399

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/878.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #878


commit de76dcd9156a7d0bd0ff4f9a57df0a527a908abd
Author: Jens Hoffmann 
Date:   2016-07-21T12:15:23Z

IGNITE-3399 Add support for primitive type names in QueryEntity

- add optional primitive type mapping to class resolution
- add test to IgniteCacheQuerySelfTestSuite




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: NullPointerException when stopping IgniteSemaphore

[Vladisav Jelisavcic]

Sure, I'll do the review.

It is lazy initialized, same as latch and other datastructures.

Regards,
Vladisav

On Thu, Jul 21, 2016 at 10:24 AM, Yakov Zhdanov  wrote:

> Vladislav, can you please review and merge to master the fix and test case
> suggested by Krome?
>
> Vlad, btw, why don't we init semaphore during construction time?
>
> https://issues.apache.org/jira/browse/IGNITE-3515
>
> Thanks!
>
> --Yakov
>

Re: IGNITE-2294 implementation details

[Alexey Goncharuk]

For me the main question here is how we define Key and Value for a cache
when using SQL interface. Unless we are going to re-architect the whole
Ignite, it will still be a key-value storage in the first place, so
whenever we do an INSERT, we must generate Key and Value. Given that values
in INSERT are supposed to be fields of either Key or Value, this confuses
me a lot.

Maybe we should come up with DDL first?

2016-07-21 14:06 GMT+03:00 Sergi Vladykin :

> No, this does not make sense.
>
> There is no upsert mode in databases. There are operations: INSERT, UPDATE,
> DELETE, MERGE.
>
> I want to have clear understanding of how they have to behave in SQL
> databases and how they will actually behave in Ignite in different
> scenarios. Also I want to have clear understanding of performance
> implications of each decision here.
>
> Anything wrong with that?
>
> Sergi
>
> On Thu, Jul 21, 2016 at 1:04 PM, Dmitriy Setrakyan 
> wrote:
>
> > Serj, are you asking what will happen as of today? Then the answer to all
> > your questions is that duplicate keys are not an issue, and Ignite always
> > operates in **upsert** mode (which is essentially a *“put(…)” *method).
> >
> > However, the *“insert”* that is suggested by Alex would delegate to
> > *“putIfAbsent(…)”*, which in database world makes more sense. However, in
> > this case, the *“update”* syntax should delegate to *“replace(…)”*, as
> > update should fail in case if a key is absent.
> >
> > Considering the above, a notion of “*upsert”* or “*merge” *operation is
> > very much needed, as it will give a user an option to perform
> > “insert-or-update” in 1 call.
> >
> > Does this make sense?
> >
> > D.
> >
> > On Wed, Jul 20, 2016 at 9:39 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > I'd prefer to do MERGE operation last because in H2 it is not standard
> > ANSI
> > > SQL MERGE. Or may be not implement it at all, or may be contribute ANSI
> > > correct version to H2, then implement it on Ignite. Need to investigate
> > the
> > > semantics deeper before making any decisions here.
> > >
> > > Lets start with simple scenarios for INSERT and go through all the
> > possible
> > > cases and answer the questions:
> > > - What will happen on key conflict in TX cache?
> > > - What will happen on key conflict in Atomic cache?
> > > - What will happen with the previous two if we use DataLoader?
> > > - How to make these operations efficient (it will be simple enough to
> > > implement them with separate put/putIfAbsent operations but probably we
> > > will need some batching like putAllIfAbsent for efficiency)?
> > >
> > > As for API, we still will need to have a single entry point for all SQL
> > > queries/commands to allow any console work with it transparently. It
> > would
> > > be great if we will be able to come up with something consistent with
> > this
> > > idea on public API.
> > >
> > > Sergi
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 20, 2016 at 2:23 PM, Dmitriy Setrakyan <
> > > dsetrak...@gridgain.com>
> > > wrote:
> > >
> > > > Like the idea of merge and insert. I need more time to think about
> the
> > > API
> > > > changes.
> > > >
> > > > Sergi, what do you think?
> > > >
> > > > Dmitriy
> > > >
> > > >
> > > >
> > > > On Jul 20, 2016, at 12:36 PM, Alexander Paschenko <
> > > > alexander.a.pasche...@gmail.com> wrote:
> > > >
> > > > >> Thus, I suggest that we implement MERGE as a separate operation
> > backed
> > > > by putIfAbsent operation, while INSERT will be implemented via put.
> > > > >
> > > > > Sorry, of course I meant that MERGE has to be put-based, while
> INSERT
> > > > > has to be putIfAbsent-based.
> > > > >
> > > > > 2016-07-20 12:30 GMT+03:00 Alexander Paschenko
> > > > > :
> > > > >> Hell Igniters,
> > > > >>
> > > > >> In this thread I would like to share and discuss some thoughts on
> > DML
> > > > >> operations' implementation, so let's start and keep it here.
> > Everyone
> > > > >> is of course welcome to share their suggestions.
> > > > >>
> > > > >> For starters, I was thinking about semantics of INSERT. In
> > traditional
> > > > >> RDBMSs, INSERT works only for records whose primary keys don't
> > > > >> conflict with those of records that are already persistent - you
> > can't
> > > > >> try to insert the same key more than once because you'll get an
> > error.
> > > > >> However, semantics of cache put is obviously different - it does
> not
> > > > >> have anything about duplicate keys, it just quietly updates values
> > in
> > > > >> case of keys' duplication. Still, cache has putIfAbsent operation
> > that
> > > > >> is closer to traditional notion of INSERT, and H2's SQL dialect
> has
> > > > >> MERGE operation which corresponds to semantics of cache put.
> Thus, I
> > > > >> suggest that we implement MERGE as a separate operation backed by
> > > > >> putIfAbsent operation, while INSERT will be 

Re: New contributor: Support primitive type names in QueryEntity (IGNITE-3399)

[Semyon Boikov]

Hi,

I added you in the contributors list.

On Thu, Jul 21, 2016 at 1:54 PM, NoTrueScotsman 
wrote:

> Hi all,
>
> I'd like to attempt a fix for IGNITE-3399 (currently unclaimed in
> Jira) as my newbie contribution.
>
> Could you add me to the list of Ignite contributors?
> Jira username: thehoff
>
> Cheers
> Jens
>

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

No, this does not make sense.

There is no upsert mode in databases. There are operations: INSERT, UPDATE,
DELETE, MERGE.

I want to have clear understanding of how they have to behave in SQL
databases and how they will actually behave in Ignite in different
scenarios. Also I want to have clear understanding of performance
implications of each decision here.

Anything wrong with that?

Sergi

On Thu, Jul 21, 2016 at 1:04 PM, Dmitriy Setrakyan 
wrote:

> Serj, are you asking what will happen as of today? Then the answer to all
> your questions is that duplicate keys are not an issue, and Ignite always
> operates in **upsert** mode (which is essentially a *“put(…)” *method).
>
> However, the *“insert”* that is suggested by Alex would delegate to
> *“putIfAbsent(…)”*, which in database world makes more sense. However, in
> this case, the *“update”* syntax should delegate to *“replace(…)”*, as
> update should fail in case if a key is absent.
>
> Considering the above, a notion of “*upsert”* or “*merge” *operation is
> very much needed, as it will give a user an option to perform
> “insert-or-update” in 1 call.
>
> Does this make sense?
>
> D.
>
> On Wed, Jul 20, 2016 at 9:39 PM, Sergi Vladykin 
> wrote:
>
> > I'd prefer to do MERGE operation last because in H2 it is not standard
> ANSI
> > SQL MERGE. Or may be not implement it at all, or may be contribute ANSI
> > correct version to H2, then implement it on Ignite. Need to investigate
> the
> > semantics deeper before making any decisions here.
> >
> > Lets start with simple scenarios for INSERT and go through all the
> possible
> > cases and answer the questions:
> > - What will happen on key conflict in TX cache?
> > - What will happen on key conflict in Atomic cache?
> > - What will happen with the previous two if we use DataLoader?
> > - How to make these operations efficient (it will be simple enough to
> > implement them with separate put/putIfAbsent operations but probably we
> > will need some batching like putAllIfAbsent for efficiency)?
> >
> > As for API, we still will need to have a single entry point for all SQL
> > queries/commands to allow any console work with it transparently. It
> would
> > be great if we will be able to come up with something consistent with
> this
> > idea on public API.
> >
> > Sergi
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Jul 20, 2016 at 2:23 PM, Dmitriy Setrakyan <
> > dsetrak...@gridgain.com>
> > wrote:
> >
> > > Like the idea of merge and insert. I need more time to think about the
> > API
> > > changes.
> > >
> > > Sergi, what do you think?
> > >
> > > Dmitriy
> > >
> > >
> > >
> > > On Jul 20, 2016, at 12:36 PM, Alexander Paschenko <
> > > alexander.a.pasche...@gmail.com> wrote:
> > >
> > > >> Thus, I suggest that we implement MERGE as a separate operation
> backed
> > > by putIfAbsent operation, while INSERT will be implemented via put.
> > > >
> > > > Sorry, of course I meant that MERGE has to be put-based, while INSERT
> > > > has to be putIfAbsent-based.
> > > >
> > > > 2016-07-20 12:30 GMT+03:00 Alexander Paschenko
> > > > :
> > > >> Hell Igniters,
> > > >>
> > > >> In this thread I would like to share and discuss some thoughts on
> DML
> > > >> operations' implementation, so let's start and keep it here.
> Everyone
> > > >> is of course welcome to share their suggestions.
> > > >>
> > > >> For starters, I was thinking about semantics of INSERT. In
> traditional
> > > >> RDBMSs, INSERT works only for records whose primary keys don't
> > > >> conflict with those of records that are already persistent - you
> can't
> > > >> try to insert the same key more than once because you'll get an
> error.
> > > >> However, semantics of cache put is obviously different - it does not
> > > >> have anything about duplicate keys, it just quietly updates values
> in
> > > >> case of keys' duplication. Still, cache has putIfAbsent operation
> that
> > > >> is closer to traditional notion of INSERT, and H2's SQL dialect has
> > > >> MERGE operation which corresponds to semantics of cache put. Thus, I
> > > >> suggest that we implement MERGE as a separate operation backed by
> > > >> putIfAbsent operation, while INSERT will be implemented via put.
> > > >>
> > > >> And one more, probably more important thing: I suggest that we
> create
> > > >> separate class Update and corresponding operation update() in
> > > >> IgniteCache. The reasons are as follows:
> > > >>
> > > >> - Query bears some flags that are clearly redundant for Update (page
> > > >> size, locality)
> > > >> - query() method in IgniteCache (one that accepts Query) and query()
> > > >> methods in GridQueryIndexing return iterators. So, if we strive to
> > > >> leave interfaces unchanged, we still will introduce some design
> > > >> ugliness like query methods returning empty iterators for certain
> > > >> queries, and/or query flags that indicate whether it's an update
> query
> > > 

New contributor: Support primitive type names in QueryEntity (IGNITE-3399)

[NoTrueScotsman]

Hi all,

I'd like to attempt a fix for IGNITE-3399 (currently unclaimed in
Jira) as my newbie contribution.

Could you add me to the list of Ignite contributors?
Jira username: thehoff

Cheers
Jens

Re: Converting TreeMap to BinaryObject causes ClassCastException (IGNITE-2852)

[Dmitriy Setrakyan]

On Wed, Jul 20, 2016 at 2:45 PM, Denis Magda  wrote:

> Hi Jens,
>
> This is not the best candidate for the first contribution and presently I
> don’t think that this feature should be supported at all.
>

Denis, why not?


>
> I would recommend you picking up one of the tickets with “newbie” filter.
> https://issues.apache.org/jira/issues/?filter=12338037 <
> https://issues.apache.org/jira/issues/?filter=12338037>
>
> Regards,
> Denis
>
> > On Jul 19, 2016, at 9:00 PM, NoTrueScotsman 
> wrote:
> >
> > It happens when applying ignite.binary().toBinary(m) to a TreeMap > V> with a custom key K.
> >
> > I thought this would be a nice task to pick as a first (attempt of a)
> > contribution, however there is a ticket for this already assigned to
> > someone: https://issues.apache.org/jira/browse/IGNITE-2852
> >
> > I haven't seen much activity on it though. Would it be possible for me
> > to pick it?
> >
> > Thanks
> > Jens
>
>

Re: Rework "withAsync" in Apache 2.0

[Andrey Kornev]

+1 "Async" suffix.



From: Vladimir Ozerov 
Sent: Thursday, July 21, 2016 2:41 AM
To: dev@ignite.apache.org
Subject: Re: Rework "withAsync" in Apache 2.0

Moreover, have a look at *CompletableFuture *interface:

handle
handle*Async*

thenApply
thenApply*Async*

And so on. One more argument to return methods with "Async" suffix from
good old GridGain days.

On Thu, Jul 21, 2016 at 12:38 PM, Vladimir Ozerov 
wrote:

> It is a matter of taste. In .NET we have "Async" methods near synchronous
> methods because it is common practice in .NET world:
> V Get(K key);
> Task GetAsync(K key);
>
> For Java we can go the same way, or introduce separate interface. It will
> not be identical to synchronous, because it will have different return
> types and probably will contain only those methods which support async
> execution.
>
> Personally I like .NET approach. It is simple and straightforward. User do
> not have to bother about whether current instance is sync or async (like
> now), and will not have to perform additional steps like
> *IgniteCache.withAsync().get()*. Do you want to call GET asynchronously?
> No problem, *IgniteCache.getAsync()*. Simple, expressive, straightforward.
>
> The drawback is that our interfaces will become "heavier". But it is 2016
> year, we all have IDEs. I do not see any problems if we will have 62 + 36 =
> 98 methods instead of current 62 on cache API.
>
> Vladimir.
>
>
> On Thu, Jul 21, 2016 at 12:21 PM, Dmitriy Setrakyan  > wrote:
>
>> Do I understand correctly that the community is proposing to have 2
>> identical interfaces, one for sync operations and another for async
>> operations?
>>
>> On Thu, Jul 21, 2016 at 12:15 PM, Sergi Vladykin <
>> sergi.vlady...@gmail.com>
>> wrote:
>>
>> > +1
>> >
>> > Finally it is time to drop this "cool feature" from Ignite!
>> >
>> > Sergi
>> >
>> > On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov > >
>> > wrote:
>> >
>> > > Alex.
>> > >
>> > > Of course, some distributed operations will involve some kind of
>> > asynchrony
>> > > even in synchronous mode. My point is that we should not blindly do
>> > things
>> > > like that:
>> > >
>> > > V get(K key) {
>> > > return getAsync(key),get();
>> > > }
>> > >
>> > > Instead, get() has it's own path, getAsync() another path. But of
>> course
>> > > they could share some common places. E.g. I remember we already fixed
>> > some
>> > > cache operations in this regard when users hit async semaphore limit
>> when
>> > > calling synchronous gets.
>> > >
>> > > Another point is that async instances may possibly accept
>> user-provided
>> > > Executor.
>> > >
>> > > Vladimir,
>> > >
>> > > On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov > >
>> > > wrote:
>> > >
>> > > > Another issue which usually confuses users is Ignite 'implementation
>> > > > details' of asynchronous execution: it operation is local it can be
>> > > > executed from calling thread (for example, if 'async put' is
>> executed
>> > in
>> > > > atomic cache from primary node then cache store will be updated from
>> > > > calling thread). Does it make sense to fix this as well?
>> > > >
>> > > >
>> > > > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov <
>> yzhda...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Agree with Alex. Vova, please go on with issues taking Alex's
>> > comments
>> > > > into
>> > > > > consideration.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > --Yakov
>> > > > >
>> > > > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
>> > > alexey.goncha...@gmail.com
>> > > > >:
>> > > > >
>> > > > > > Big +1 on this in general.
>> > > > > >
>> > > > > > I would also relax our guarantees on operations submitted from
>> the
>> > > same
>> > > > > > thread. Currently we guarantee that sequential invocations of
>> async
>> > > > > > operations happen in the same order. I think that if a user
>> wants
>> > > such
>> > > > > > guarantees, he must define these dependencies explicitly by
>> calling
>> > > > > chain()
>> > > > > > on returning futures.
>> > > > > >
>> > > > > > This change will significantly improve cache operations
>> performance
>> > > in
>> > > > > > async mode.
>> > > > > >
>> > > > > > 3) Sync operations normally* should not* be implemented through
>> > > async.
>> > > > > This
>> > > > > > > is a long story - if we delegate to async, then we have to
>> bother
>> > > > with
>> > > > > > > additional threads, associated back-pressure control and all
>> that
>> > > > crap.
>> > > > > > > Sync call must be sync unless there is a very strong reason
>> to go
>> > > > > through
>> > > > > > > async path.
>> > > > > > >
>> > > > > > Not sure about this, though. In most cases a cache operation
>> > implies
>> > > > > > request/response over the network, so I think we should have
>> > explicit
>> > > > > > synchronous counterparts only for methods that are guaranteed
>> 

Re: Rework "withAsync" in Apache 2.0

[Dmitriy Setrakyan]

My opinion is that if we do away with current “withAsync()” paradigm, then
we should simply add the *async* methods directly to the original
interface. At least this way it will be easier to tell which methods
support async execution, and which don’t.

I don’t think it makes sense to have 2 separate interfaces, one for sync
and another for async.

D.

On Thu, Jul 21, 2016 at 1:01 PM, Sergi Vladykin 
wrote:

> I'm ok with any of these ways. Probably having methods with "Async" suffix
> is simpler, having a separate interface for all the async methods is a bit
> cleaner. Current IgniteAsyncSupport definitely was a big mistake.
>
> Sergi
>
> On Thu, Jul 21, 2016 at 12:41 PM, Vladimir Ozerov 
> wrote:
>
> > Moreover, have a look at *CompletableFuture *interface:
> >
> > handle
> > handle*Async*
> >
> > thenApply
> > thenApply*Async*
> >
> > And so on. One more argument to return methods with "Async" suffix from
> > good old GridGain days.
> >
> > On Thu, Jul 21, 2016 at 12:38 PM, Vladimir Ozerov 
> > wrote:
> >
> > > It is a matter of taste. In .NET we have "Async" methods near
> synchronous
> > > methods because it is common practice in .NET world:
> > > V Get(K key);
> > > Task GetAsync(K key);
> > >
> > > For Java we can go the same way, or introduce separate interface. It
> will
> > > not be identical to synchronous, because it will have different return
> > > types and probably will contain only those methods which support async
> > > execution.
> > >
> > > Personally I like .NET approach. It is simple and straightforward. User
> > do
> > > not have to bother about whether current instance is sync or async
> (like
> > > now), and will not have to perform additional steps like
> > > *IgniteCache.withAsync().get()*. Do you want to call GET
> asynchronously?
> > > No problem, *IgniteCache.getAsync()*. Simple, expressive,
> > straightforward.
> > >
> > > The drawback is that our interfaces will become "heavier". But it is
> 2016
> > > year, we all have IDEs. I do not see any problems if we will have 62 +
> > 36 =
> > > 98 methods instead of current 62 on cache API.
> > >
> > > Vladimir.
> > >
> > >
> > > On Thu, Jul 21, 2016 at 12:21 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > > wrote:
> > >
> > >> Do I understand correctly that the community is proposing to have 2
> > >> identical interfaces, one for sync operations and another for async
> > >> operations?
> > >>
> > >> On Thu, Jul 21, 2016 at 12:15 PM, Sergi Vladykin <
> > >> sergi.vlady...@gmail.com>
> > >> wrote:
> > >>
> > >> > +1
> > >> >
> > >> > Finally it is time to drop this "cool feature" from Ignite!
> > >> >
> > >> > Sergi
> > >> >
> > >> > On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Alex.
> > >> > >
> > >> > > Of course, some distributed operations will involve some kind of
> > >> > asynchrony
> > >> > > even in synchronous mode. My point is that we should not blindly
> do
> > >> > things
> > >> > > like that:
> > >> > >
> > >> > > V get(K key) {
> > >> > > return getAsync(key),get();
> > >> > > }
> > >> > >
> > >> > > Instead, get() has it's own path, getAsync() another path. But of
> > >> course
> > >> > > they could share some common places. E.g. I remember we already
> > fixed
> > >> > some
> > >> > > cache operations in this regard when users hit async semaphore
> limit
> > >> when
> > >> > > calling synchronous gets.
> > >> > >
> > >> > > Another point is that async instances may possibly accept
> > >> user-provided
> > >> > > Executor.
> > >> > >
> > >> > > Vladimir,
> > >> > >
> > >> > > On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov <
> > sboi...@gridgain.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Another issue which usually confuses users is Ignite
> > 'implementation
> > >> > > > details' of asynchronous execution: it operation is local it can
> > be
> > >> > > > executed from calling thread (for example, if 'async put' is
> > >> executed
> > >> > in
> > >> > > > atomic cache from primary node then cache store will be updated
> > from
> > >> > > > calling thread). Does it make sense to fix this as well?
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov <
> > >> yzhda...@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Agree with Alex. Vova, please go on with issues taking Alex's
> > >> > comments
> > >> > > > into
> > >> > > > > consideration.
> > >> > > > >
> > >> > > > > Thanks!
> > >> > > > >
> > >> > > > > --Yakov
> > >> > > > >
> > >> > > > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
> > >> > > alexey.goncha...@gmail.com
> > >> > > > >:
> > >> > > > >
> > >> > > > > > Big +1 on this in general.
> > >> > > > > >
> > >> > > > > > I would also relax our guarantees on operations submitted
> from
> > >> the
> > >> > > same
> > >> > > > > > thread. Currently we guarantee that sequential invocations
> of
> > >> 

Re: Rework "withAsync" in Apache 2.0

[Andrey Kornev]

Totally agree with this particular suggestion. On more than just a few 
occasions I've been greatly confused by  Ignite's "asynchronous" operations 
that in reality are blocking under some circumstances and non-blocking under 
others... Go figure! The reason I was given for such design was performance, if 
I remember correctly. I hope the Ignite experts on this mailing list will be 
able to provide more specifics if the community is interested.


In general an asynchronous operation is the one that returns a future 
immediately without any blocking of the calling thread. The outcome of the 
invocation is then communicated back exclusively via the future.


With the growing popularity of reactive programming style (a la Rx Java, etc.), 
I think it is crucial to get this fixed.


Regards

Andrey


From: Semyon Boikov 
Sent: Thursday, July 21, 2016 1:04 AM
To: dev@ignite.apache.org
Subject: Re: Rework "withAsync" in Apache 2.0

Another issue which usually confuses users is Ignite 'implementation
details' of asynchronous execution: it operation is local it can be
executed from calling thread (for example, if 'async put' is executed in
atomic cache from primary node then cache store will be updated from
calling thread). Does it make sense to fix this as well?


On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov  wrote:

> Agree with Alex. Vova, please go on with issues taking Alex's comments into
> consideration.
>
> Thanks!
>
> --Yakov
>
> 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk :
>
> > Big +1 on this in general.
> >
> > I would also relax our guarantees on operations submitted from the same
> > thread. Currently we guarantee that sequential invocations of async
> > operations happen in the same order. I think that if a user wants such
> > guarantees, he must define these dependencies explicitly by calling
> chain()
> > on returning futures.
> >
> > This change will significantly improve cache operations performance in
> > async mode.
> >
> > 3) Sync operations normally* should not* be implemented through async.
> This
> > > is a long story - if we delegate to async, then we have to bother with
> > > additional threads, associated back-pressure control and all that crap.
> > > Sync call must be sync unless there is a very strong reason to go
> through
> > > async path.
> > >
> > Not sure about this, though. In most cases a cache operation implies
> > request/response over the network, so I think we should have explicit
> > synchronous counterparts only for methods that are guaranteed to be
> local.
> >
>

Re: ReadWriteUpdateLock

[Vladimir Ozerov]

I hardly can image where we can get concrete numbers of method usages. RWU
lock is just a base .NET primitive. There are no plain RW lock there, only
RWU. Users are just used to it.

But again, my point is - let's implement standard RW lock first and then
think about such extensions.

On Wed, Jul 20, 2016 at 4:29 PM, Yakov Zhdanov  wrote:

> Vova, we discuss this without having any numbers. Any real-life use case?
>
> --Yakov
>
> 2016-07-20 9:48 GMT+03:00 Vladimir Ozerov :
>
> > I believe this could be a useful addition to Ignite. RWU-locks are not
> > widely-spread in Java. But for example in .NET this is a base concurrency
> > primitive. Of course users can handle "update" situations differently,
> e.g.
> > "release read lock, then acquire write lock, then re-check condition,
> ...".
> > But this is exactly where "update" semantics could free user from this
> > burden. I think we will see some demand for this feature at least from
> .NET
> > community.
> >
> > On the other hand, I fully agree with Yakov that "update" is a kind of
> > corner case. Standard "read lock" and "write lock" are much more common.
> We
> > should implement distributed RWlock first.
> >
> > For RWULock we can at least create a JIRA ticket for now. If someone from
> > community would like to implement it - then why not?
> >
> > Vladimir.
> >
> > On Tue, Jul 19, 2016 at 1:56 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Hi Vlad!
> > >
> > > Thanks for bringing this up.
> > >
> > > I looked through concurrency-interest discussion, and I don't think we
> > > should do this in Ignite. At least now. I am not sure if this will give
> > any
> > > advantage since only one thread can acquire UPDATE lock at the same
> time.
> > > Btw, was there any benchmark published comparing UpdateLock  vs RWLock
> > > implementations?
> > >
> > > I think that in many cases read then update scenarios can be handled
> with
> > > some kind of volatile or atomic read and then acquiring the ordinary
> lock
> > > or by CAS operation. For the rest of cases we already have RWLock.
> > >
> > > And one more point - nobody asked for it. So, I ask - Does anyone need
> it
> > > in Ignite?
> > >
> > > Thanks!
> > >
> > > --Yakov
> > >
> > > 2016-07-18 22:55 GMT+03:00 Vladisav Jelisavcic :
> > >
> > > > Hi everyone,
> > > >
> > > > cross-posting from JIRA:
> > > > I recently came across this post:
> > > > http://codereview.stackexchange.com/a/31231
> > > >
> > > > Do you think ReadWriteUpdateLock is something we can put to good use
> > here
> > > > in Ignite?
> > > >
> > > > This kind of lock should be more efficient for read-before-write
> > > patterns.
> > > >
> > > > Best regards,
> > > > Vladisav
> > > >
> > >
> >
>

Re: Rework "withAsync" in Apache 2.0

[Vladimir Ozerov]

Moreover, have a look at *CompletableFuture *interface:

handle
handle*Async*

thenApply
thenApply*Async*

And so on. One more argument to return methods with "Async" suffix from
good old GridGain days.

On Thu, Jul 21, 2016 at 12:38 PM, Vladimir Ozerov 
wrote:

> It is a matter of taste. In .NET we have "Async" methods near synchronous
> methods because it is common practice in .NET world:
> V Get(K key);
> Task GetAsync(K key);
>
> For Java we can go the same way, or introduce separate interface. It will
> not be identical to synchronous, because it will have different return
> types and probably will contain only those methods which support async
> execution.
>
> Personally I like .NET approach. It is simple and straightforward. User do
> not have to bother about whether current instance is sync or async (like
> now), and will not have to perform additional steps like
> *IgniteCache.withAsync().get()*. Do you want to call GET asynchronously?
> No problem, *IgniteCache.getAsync()*. Simple, expressive, straightforward.
>
> The drawback is that our interfaces will become "heavier". But it is 2016
> year, we all have IDEs. I do not see any problems if we will have 62 + 36 =
> 98 methods instead of current 62 on cache API.
>
> Vladimir.
>
>
> On Thu, Jul 21, 2016 at 12:21 PM, Dmitriy Setrakyan  > wrote:
>
>> Do I understand correctly that the community is proposing to have 2
>> identical interfaces, one for sync operations and another for async
>> operations?
>>
>> On Thu, Jul 21, 2016 at 12:15 PM, Sergi Vladykin <
>> sergi.vlady...@gmail.com>
>> wrote:
>>
>> > +1
>> >
>> > Finally it is time to drop this "cool feature" from Ignite!
>> >
>> > Sergi
>> >
>> > On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov > >
>> > wrote:
>> >
>> > > Alex.
>> > >
>> > > Of course, some distributed operations will involve some kind of
>> > asynchrony
>> > > even in synchronous mode. My point is that we should not blindly do
>> > things
>> > > like that:
>> > >
>> > > V get(K key) {
>> > > return getAsync(key),get();
>> > > }
>> > >
>> > > Instead, get() has it's own path, getAsync() another path. But of
>> course
>> > > they could share some common places. E.g. I remember we already fixed
>> > some
>> > > cache operations in this regard when users hit async semaphore limit
>> when
>> > > calling synchronous gets.
>> > >
>> > > Another point is that async instances may possibly accept
>> user-provided
>> > > Executor.
>> > >
>> > > Vladimir,
>> > >
>> > > On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov > >
>> > > wrote:
>> > >
>> > > > Another issue which usually confuses users is Ignite 'implementation
>> > > > details' of asynchronous execution: it operation is local it can be
>> > > > executed from calling thread (for example, if 'async put' is
>> executed
>> > in
>> > > > atomic cache from primary node then cache store will be updated from
>> > > > calling thread). Does it make sense to fix this as well?
>> > > >
>> > > >
>> > > > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov <
>> yzhda...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Agree with Alex. Vova, please go on with issues taking Alex's
>> > comments
>> > > > into
>> > > > > consideration.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > --Yakov
>> > > > >
>> > > > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
>> > > alexey.goncha...@gmail.com
>> > > > >:
>> > > > >
>> > > > > > Big +1 on this in general.
>> > > > > >
>> > > > > > I would also relax our guarantees on operations submitted from
>> the
>> > > same
>> > > > > > thread. Currently we guarantee that sequential invocations of
>> async
>> > > > > > operations happen in the same order. I think that if a user
>> wants
>> > > such
>> > > > > > guarantees, he must define these dependencies explicitly by
>> calling
>> > > > > chain()
>> > > > > > on returning futures.
>> > > > > >
>> > > > > > This change will significantly improve cache operations
>> performance
>> > > in
>> > > > > > async mode.
>> > > > > >
>> > > > > > 3) Sync operations normally* should not* be implemented through
>> > > async.
>> > > > > This
>> > > > > > > is a long story - if we delegate to async, then we have to
>> bother
>> > > > with
>> > > > > > > additional threads, associated back-pressure control and all
>> that
>> > > > crap.
>> > > > > > > Sync call must be sync unless there is a very strong reason
>> to go
>> > > > > through
>> > > > > > > async path.
>> > > > > > >
>> > > > > > Not sure about this, though. In most cases a cache operation
>> > implies
>> > > > > > request/response over the network, so I think we should have
>> > explicit
>> > > > > > synchronous counterparts only for methods that are guaranteed
>> to be
>> > > > > local.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Rework "withAsync" in Apache 2.0

[Vladimir Ozerov]

It is a matter of taste. In .NET we have "Async" methods near synchronous
methods because it is common practice in .NET world:
V Get(K key);
Task GetAsync(K key);

For Java we can go the same way, or introduce separate interface. It will
not be identical to synchronous, because it will have different return
types and probably will contain only those methods which support async
execution.

Personally I like .NET approach. It is simple and straightforward. User do
not have to bother about whether current instance is sync or async (like
now), and will not have to perform additional steps like
*IgniteCache.withAsync().get()*. Do you want to call GET asynchronously? No
problem, *IgniteCache.getAsync()*. Simple, expressive, straightforward.

The drawback is that our interfaces will become "heavier". But it is 2016
year, we all have IDEs. I do not see any problems if we will have 62 + 36 =
98 methods instead of current 62 on cache API.

Vladimir.


On Thu, Jul 21, 2016 at 12:21 PM, Dmitriy Setrakyan 
wrote:

> Do I understand correctly that the community is proposing to have 2
> identical interfaces, one for sync operations and another for async
> operations?
>
> On Thu, Jul 21, 2016 at 12:15 PM, Sergi Vladykin  >
> wrote:
>
> > +1
> >
> > Finally it is time to drop this "cool feature" from Ignite!
> >
> > Sergi
> >
> > On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Alex.
> > >
> > > Of course, some distributed operations will involve some kind of
> > asynchrony
> > > even in synchronous mode. My point is that we should not blindly do
> > things
> > > like that:
> > >
> > > V get(K key) {
> > > return getAsync(key),get();
> > > }
> > >
> > > Instead, get() has it's own path, getAsync() another path. But of
> course
> > > they could share some common places. E.g. I remember we already fixed
> > some
> > > cache operations in this regard when users hit async semaphore limit
> when
> > > calling synchronous gets.
> > >
> > > Another point is that async instances may possibly accept user-provided
> > > Executor.
> > >
> > > Vladimir,
> > >
> > > On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov 
> > > wrote:
> > >
> > > > Another issue which usually confuses users is Ignite 'implementation
> > > > details' of asynchronous execution: it operation is local it can be
> > > > executed from calling thread (for example, if 'async put' is executed
> > in
> > > > atomic cache from primary node then cache store will be updated from
> > > > calling thread). Does it make sense to fix this as well?
> > > >
> > > >
> > > > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov  >
> > > > wrote:
> > > >
> > > > > Agree with Alex. Vova, please go on with issues taking Alex's
> > comments
> > > > into
> > > > > consideration.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --Yakov
> > > > >
> > > > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > >:
> > > > >
> > > > > > Big +1 on this in general.
> > > > > >
> > > > > > I would also relax our guarantees on operations submitted from
> the
> > > same
> > > > > > thread. Currently we guarantee that sequential invocations of
> async
> > > > > > operations happen in the same order. I think that if a user wants
> > > such
> > > > > > guarantees, he must define these dependencies explicitly by
> calling
> > > > > chain()
> > > > > > on returning futures.
> > > > > >
> > > > > > This change will significantly improve cache operations
> performance
> > > in
> > > > > > async mode.
> > > > > >
> > > > > > 3) Sync operations normally* should not* be implemented through
> > > async.
> > > > > This
> > > > > > > is a long story - if we delegate to async, then we have to
> bother
> > > > with
> > > > > > > additional threads, associated back-pressure control and all
> that
> > > > crap.
> > > > > > > Sync call must be sync unless there is a very strong reason to
> go
> > > > > through
> > > > > > > async path.
> > > > > > >
> > > > > > Not sure about this, though. In most cases a cache operation
> > implies
> > > > > > request/response over the network, so I think we should have
> > explicit
> > > > > > synchronous counterparts only for methods that are guaranteed to
> be
> > > > > local.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Rework "withAsync" in Apache 2.0

[Dmitriy Setrakyan]

Do I understand correctly that the community is proposing to have 2
identical interfaces, one for sync operations and another for async
operations?

On Thu, Jul 21, 2016 at 12:15 PM, Sergi Vladykin 
wrote:

> +1
>
> Finally it is time to drop this "cool feature" from Ignite!
>
> Sergi
>
> On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov 
> wrote:
>
> > Alex.
> >
> > Of course, some distributed operations will involve some kind of
> asynchrony
> > even in synchronous mode. My point is that we should not blindly do
> things
> > like that:
> >
> > V get(K key) {
> > return getAsync(key),get();
> > }
> >
> > Instead, get() has it's own path, getAsync() another path. But of course
> > they could share some common places. E.g. I remember we already fixed
> some
> > cache operations in this regard when users hit async semaphore limit when
> > calling synchronous gets.
> >
> > Another point is that async instances may possibly accept user-provided
> > Executor.
> >
> > Vladimir,
> >
> > On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov 
> > wrote:
> >
> > > Another issue which usually confuses users is Ignite 'implementation
> > > details' of asynchronous execution: it operation is local it can be
> > > executed from calling thread (for example, if 'async put' is executed
> in
> > > atomic cache from primary node then cache store will be updated from
> > > calling thread). Does it make sense to fix this as well?
> > >
> > >
> > > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov 
> > > wrote:
> > >
> > > > Agree with Alex. Vova, please go on with issues taking Alex's
> comments
> > > into
> > > > consideration.
> > > >
> > > > Thanks!
> > > >
> > > > --Yakov
> > > >
> > > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > >:
> > > >
> > > > > Big +1 on this in general.
> > > > >
> > > > > I would also relax our guarantees on operations submitted from the
> > same
> > > > > thread. Currently we guarantee that sequential invocations of async
> > > > > operations happen in the same order. I think that if a user wants
> > such
> > > > > guarantees, he must define these dependencies explicitly by calling
> > > > chain()
> > > > > on returning futures.
> > > > >
> > > > > This change will significantly improve cache operations performance
> > in
> > > > > async mode.
> > > > >
> > > > > 3) Sync operations normally* should not* be implemented through
> > async.
> > > > This
> > > > > > is a long story - if we delegate to async, then we have to bother
> > > with
> > > > > > additional threads, associated back-pressure control and all that
> > > crap.
> > > > > > Sync call must be sync unless there is a very strong reason to go
> > > > through
> > > > > > async path.
> > > > > >
> > > > > Not sure about this, though. In most cases a cache operation
> implies
> > > > > request/response over the network, so I think we should have
> explicit
> > > > > synchronous counterparts only for methods that are guaranteed to be
> > > > local.
> > > > >
> > > >
> > >
> >
>

Re: Ignite monitoring

[Dmitriy Setrakyan]

On Wed, Jul 20, 2016 at 7:35 PM, Igor Rudyak  wrote:

> Andrey,
>
> Is Web Console you are talking about the same thing as GridGain Web Console
> (http://ignite.apache.org/addons.html#web-console)? If yes it has
> monitoring tab which allows to monitor some JVM and cache metrics (
> https://console.gridgain.com/monitoring).
>

Not exactly. GridGain web console is based on Ignite web console, but
GridGain added more features to it, management tab is one of those features.

You should feel free to add a monitoring tab as part of Ignite web console
or work with GridGain directly to add more features to the management tab.


> Igor Rudyak
>
>
> On Wed, Jul 20, 2016 at 2:46 AM, Andrey Novikov 
> wrote:
>
> > Igor,
> >
> > Ignite does not have monitoring in Web Console, only Configuration and
> SQL.
> > But command-line Visor in Ignite have some monitoring features and can be
> > used  to add new.
> >
> > On Wed, Jul 20, 2016 at 1:19 AM, Igor Rudyak  wrote:
> >
> > > Are there any documentation regarding how to use Ignite web console?
> How
> > to
> > > add new metrics to monitor?
> > >
> > > On Mon, Jul 18, 2016 at 10:47 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > wrote:
> > >
> > > > On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <
> > > akuznet...@gridgain.com
> > > > >
> > > > wrote:
> > > >
> > > > > I think we should have some general API and we could call it from
> web
> > > > > console and/or from other places.
> > > > >
> > > >
> > > > Agree. The server side support should be sufficient to enable
> different
> > > > monitoring connections, including command-line visor, web console, or
> > JMX
> > > > beans.
> > > >
> > > >
> > > > > 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> > > > > dsetrak...@apache.org>
> > > > > написал:
> > > > >
> > > > > > I think we can add this functionality to Ignite web console, no?
> > > > > >
> > > > > > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <
> > > > voze...@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Igor,
> > > > > > >
> > > > > > > I think that built-in monitoring facility will add great value
> to
> > > the
> > > > > > > product. We have to deal with user performance issues pretty
> > often,
> > > > and
> > > > > > it
> > > > > > > is always a kind of pain to get to the bottom of the problem.
> We
> > > have
> > > > > to
> > > > > > > ask users for configuration, logs, system config, etc, etc..
> > > Instead,
> > > > > it
> > > > > > > would be great if we had a single big "switch". If user has
> > > > performance
> > > > > > > issue, he turns it on, then perform problematic operations, and
> > > then
> > > > > > dumps
> > > > > > > all collected data. We can collect dozens of things:
> > > > > > > 1) OS/JVM information
> > > > > > > 2) Ignite configs, logs, etc..
> > > > > > > 3) Performance data (CPU, RAM, IO)
> > > > > > > 4) Metrics
> > > > > > > 5) JMX data (both Ignite and JVM)
> > > > > > > 6) Some internal tracing (SQL query plans, how long it takes
> > > messages
> > > > > to
> > > > > > > pass between nodes, etc.)
> > > > > > >
> > > > > > > I think the most important part here is good infrastructure
> > > > > (interfaces)
> > > > > > > and API. So that we can start with something very simple, like
> > > > > collecting
> > > > > > > configs from all nodes, or starting/stopping shell commands,
> and
> > > then
> > > > > > > gradually add more and more tracing facilities.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <
> irud...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Yakov, as for now I just have well structured scripts to
> setup
> > > > > Ganglia
> > > > > > > > agent on Ignite hosts to monitor system metrics like CPU,
> RAM,
> > IO
> > > > and
> > > > > > etc
> > > > > > > > (this scripts already included in Ignite 1.6).
> > > > > > > >
> > > > > > > > Also experimented with displaying JVM metrics by providing
> java
> > > > agent
> > > > > > and
> > > > > > > > specifying MBeans to collect metrics from. But it's rather
> > draft
> > > > > > version.
> > > > > > > > The second problem is, there are plenty of MBeans in Ignite
> - I
> > > > just
> > > > > > > don't
> > > > > > > > know which to select from.
> > > > > > > >
> > > > > > > > Anyway, the original idea was to check with the community if
> it
> > > > makes
> > > > > > > sense
> > > > > > > > to have such monitoring functionality out of the box.
> > > > > > > >
> > > > > > > > Igor Rudyak
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <
> > > > yzhda...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Igor, can you please share the changes to scripts you did
> to
> > > > > support
> > > > > > > > > monitoring? Can it be done by defining and exporting
> > 

Re: Rework "withAsync" in Apache 2.0

[Sergi Vladykin]

+1

Finally it is time to drop this "cool feature" from Ignite!

Sergi

On Thu, Jul 21, 2016 at 11:13 AM, Vladimir Ozerov 
wrote:

> Alex.
>
> Of course, some distributed operations will involve some kind of asynchrony
> even in synchronous mode. My point is that we should not blindly do things
> like that:
>
> V get(K key) {
> return getAsync(key),get();
> }
>
> Instead, get() has it's own path, getAsync() another path. But of course
> they could share some common places. E.g. I remember we already fixed some
> cache operations in this regard when users hit async semaphore limit when
> calling synchronous gets.
>
> Another point is that async instances may possibly accept user-provided
> Executor.
>
> Vladimir,
>
> On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov 
> wrote:
>
> > Another issue which usually confuses users is Ignite 'implementation
> > details' of asynchronous execution: it operation is local it can be
> > executed from calling thread (for example, if 'async put' is executed in
> > atomic cache from primary node then cache store will be updated from
> > calling thread). Does it make sense to fix this as well?
> >
> >
> > On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov 
> > wrote:
> >
> > > Agree with Alex. Vova, please go on with issues taking Alex's comments
> > into
> > > consideration.
> > >
> > > Thanks!
> > >
> > > --Yakov
> > >
> > > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> > >
> > > > Big +1 on this in general.
> > > >
> > > > I would also relax our guarantees on operations submitted from the
> same
> > > > thread. Currently we guarantee that sequential invocations of async
> > > > operations happen in the same order. I think that if a user wants
> such
> > > > guarantees, he must define these dependencies explicitly by calling
> > > chain()
> > > > on returning futures.
> > > >
> > > > This change will significantly improve cache operations performance
> in
> > > > async mode.
> > > >
> > > > 3) Sync operations normally* should not* be implemented through
> async.
> > > This
> > > > > is a long story - if we delegate to async, then we have to bother
> > with
> > > > > additional threads, associated back-pressure control and all that
> > crap.
> > > > > Sync call must be sync unless there is a very strong reason to go
> > > through
> > > > > async path.
> > > > >
> > > > Not sure about this, though. In most cases a cache operation implies
> > > > request/response over the network, so I think we should have explicit
> > > > synchronous counterparts only for methods that are guaranteed to be
> > > local.
> > > >
> > >
> >
>

NullPointerException when stopping IgniteSemaphore

[Yakov Zhdanov]

Vladislav, can you please review and merge to master the fix and test case
suggested by Krome?

Vlad, btw, why don't we init semaphore during construction time?

https://issues.apache.org/jira/browse/IGNITE-3515

Thanks!

--Yakov

[GitHub] ignite pull request #877: Ignite 1525

[avinogradovgg]

GitHub user avinogradovgg opened a pull request:

https://github.com/apache/ignite/pull/877

Ignite 1525



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-1525

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #877


commit 01a6e86ec4e19d372b8efbc5c497c84f4238a46c
Author: vozerov-gridgain 
Date:   2016-03-24T10:28:30Z

IGNITE-2876: Fixed possible starvation in system pool caused by 
IgfsBlockMessage. This closes #575.

commit 59705e008267b1d5926410f95c68bb8ffb8cd93c
Author: vozerov-gridgain 
Date:   2016-03-24T11:29:35Z

IGNITE-2883: IGFS: Now IPC messages are handled in a dedicated thread pool.

commit 0412c9bfd89b8d2a377588e908f1012c845ac5bc
Author: Alexey Kuznetsov 
Date:   2016-03-30T04:43:19Z

Added detail info about keys count in partitions for offheap and swap.

commit 28d2a7bf7f35ec4b51fba872ace47cdbc255ded8
Author: Alexey Kuznetsov 
Date:   2016-03-30T07:42:12Z

Minor change to Visor classes: added setters for name and queryMetrics 
properties.

commit a70d2dc48c42f23b1ea64c2a219560efa44fb99f
Author: Alexey Kuznetsov 
Date:   2016-04-04T07:14:42Z

Minor fix for Visor query metrics.

commit ec2ec9965ae03ef3666b2035a13f444cf2765aec
Author: dkarachentsev 
Date:   2016-03-30T10:29:36Z

Now cache plugins are able to unwrap entries passed to EntryProcessor.

commit e85a7170534cb66f40386cba689cfe632f4e66db
Author: ashutak 
Date:   2016-04-05T11:56:01Z

ignite-2835: Fixed BinaryObjectOffHeapImpl leakage to public code

commit d33b4340a68553e59e4adecf78fea79af55bf2ae
Author: sboikov 
Date:   2016-04-05T13:42:50Z

ignite-2835 Minor test changes.

commit 2397552daf6d8cff9b59515c1c8983abdc60f5f4
Author: vdpyatkov 
Date:   2016-04-05T15:03:55Z

IGNITE-2822
Continuous query local listener can be notified with empty list of events

commit 87f7522c56b5735442f8d52dbc63756de53f2464
Author: sboikov 
Date:   2016-04-06T06:46:47Z

gridgain-7.5.11 Increased default affinity history size.

commit 86347272e712a0e61a5a763dda3458fde79f29c4
Author: Anton Vinogradov 
Date:   2016-04-06T08:35:36Z

IGNITE-2947
BinaryContext doesn't honor custom loader set through 
IgniteConfiguration.classLoader

commit b2c934d7abe6dfe86e1d09ec592875df987f2120
Author: Alexey Kuznetsov 
Date:   2016-04-06T10:31:21Z

 IGNITE-2963 Fixed special case with Date for Oracle JDBC driver.

commit e7ab8eef504cdcf8987941e8b7a552ada451aec8
Author: sboikov 
Date:   2016-04-06T14:38:04Z

gridgain-7.5.11 Affinity history cleanup ignoring client discovery event.

commit d92e617a71c31ccaa3fd6e4954aa4ccaf494733c
Author: Valentin Kulichenko 
Date:   2016-04-07T01:10:45Z

IGNITE-2951 - Stability fixes for cluster with many clients

commit bbe1a7e52f177283969e81127216838f37b4205f
Author: Anton Vinogradov 
Date:   2016-04-07T14:42:33Z

IGNITE-2947
BinaryContext doesn't honor custom loader set through 
IgniteConfiguration.classLoader
Rollback

commit d6a2aae09dc84f3368e850d4eec30065e0ec3b9d
Author: Anton Vinogradov 
Date:   2016-04-08T07:32:13Z

 IGNITE-2947 BinaryContext doesn't honor custom loader set through 
IgniteConfiguration.classLoader

commit 7b8bd7c866985e5d497158b4e82cc239702b3953
Author: vsisko 
Date:   2016-04-08T10:25:47Z

Compatibility. Fixed version.

commit c28faddb51c02be70d7692141f215972fa3e976e
Author: Anton Vinogradov 
Date:   2016-04-11T14:41:32Z

IGNITE-2965 Failed to read class name from file on deserialization

commit 18e355bb1e466cc5ded836b228a721adceb09df5
Author: ashutak 
Date:   2016-04-11T16:48:00Z

ignite-2645: Fixed assertion error in ATOMIC cache for invokeAll and cache 
store

commit 4e34f58e3affb6fd73807333808e1b8ede8e8b64
Author: Pavel Tupitsyn 
Date:   2016-04-13T14:54:46Z

IGNITE-2979: .NET: Implemented ability to use Java filter in .NET-based 
continuous queries.

commit e050ac0e8ce421e8039a22d4848c3165e7a62842
Author: vozerov-gridgain 
Date:   2016-04-13T15:11:00Z

IGNITE-2979: .NET: Implemented ability to use Java filter in .NET-based 
continuous queries.

commit 73152db78bb439968d0dc909cbfcd9c534dadc01
Author: vozerov-gridgain 
Date:   2016-04-14T06:47:01Z

IGNITE-2977:  Fixed .NET compilation and tests.

commit 

Re: Rework "withAsync" in Apache 2.0

[Vladimir Ozerov]

Alex.

Of course, some distributed operations will involve some kind of asynchrony
even in synchronous mode. My point is that we should not blindly do things
like that:

V get(K key) {
return getAsync(key),get();
}

Instead, get() has it's own path, getAsync() another path. But of course
they could share some common places. E.g. I remember we already fixed some
cache operations in this regard when users hit async semaphore limit when
calling synchronous gets.

Another point is that async instances may possibly accept user-provided
Executor.

Vladimir,

On Thu, Jul 21, 2016 at 11:04 AM, Semyon Boikov 
wrote:

> Another issue which usually confuses users is Ignite 'implementation
> details' of asynchronous execution: it operation is local it can be
> executed from calling thread (for example, if 'async put' is executed in
> atomic cache from primary node then cache store will be updated from
> calling thread). Does it make sense to fix this as well?
>
>
> On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov 
> wrote:
>
> > Agree with Alex. Vova, please go on with issues taking Alex's comments
> into
> > consideration.
> >
> > Thanks!
> >
> > --Yakov
> >
> > 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk  >:
> >
> > > Big +1 on this in general.
> > >
> > > I would also relax our guarantees on operations submitted from the same
> > > thread. Currently we guarantee that sequential invocations of async
> > > operations happen in the same order. I think that if a user wants such
> > > guarantees, he must define these dependencies explicitly by calling
> > chain()
> > > on returning futures.
> > >
> > > This change will significantly improve cache operations performance in
> > > async mode.
> > >
> > > 3) Sync operations normally* should not* be implemented through async.
> > This
> > > > is a long story - if we delegate to async, then we have to bother
> with
> > > > additional threads, associated back-pressure control and all that
> crap.
> > > > Sync call must be sync unless there is a very strong reason to go
> > through
> > > > async path.
> > > >
> > > Not sure about this, though. In most cases a cache operation implies
> > > request/response over the network, so I think we should have explicit
> > > synchronous counterparts only for methods that are guaranteed to be
> > local.
> > >
> >
>

Re: Rework "withAsync" in Apache 2.0

[Semyon Boikov]

Another issue which usually confuses users is Ignite 'implementation
details' of asynchronous execution: it operation is local it can be
executed from calling thread (for example, if 'async put' is executed in
atomic cache from primary node then cache store will be updated from
calling thread). Does it make sense to fix this as well?


On Thu, Jul 21, 2016 at 10:55 AM, Yakov Zhdanov  wrote:

> Agree with Alex. Vova, please go on with issues taking Alex's comments into
> consideration.
>
> Thanks!
>
> --Yakov
>
> 2016-07-21 10:43 GMT+03:00 Alexey Goncharuk :
>
> > Big +1 on this in general.
> >
> > I would also relax our guarantees on operations submitted from the same
> > thread. Currently we guarantee that sequential invocations of async
> > operations happen in the same order. I think that if a user wants such
> > guarantees, he must define these dependencies explicitly by calling
> chain()
> > on returning futures.
> >
> > This change will significantly improve cache operations performance in
> > async mode.
> >
> > 3) Sync operations normally* should not* be implemented through async.
> This
> > > is a long story - if we delegate to async, then we have to bother with
> > > additional threads, associated back-pressure control and all that crap.
> > > Sync call must be sync unless there is a very strong reason to go
> through
> > > async path.
> > >
> > Not sure about this, though. In most cases a cache operation implies
> > request/response over the network, so I think we should have explicit
> > synchronous counterparts only for methods that are guaranteed to be
> local.
> >
>

Re: Rework "withAsync" in Apache 2.0

[Yakov Zhdanov]

Agree with Alex. Vova, please go on with issues taking Alex's comments into
consideration.

Thanks!

--Yakov

2016-07-21 10:43 GMT+03:00 Alexey Goncharuk :

> Big +1 on this in general.
>
> I would also relax our guarantees on operations submitted from the same
> thread. Currently we guarantee that sequential invocations of async
> operations happen in the same order. I think that if a user wants such
> guarantees, he must define these dependencies explicitly by calling chain()
> on returning futures.
>
> This change will significantly improve cache operations performance in
> async mode.
>
> 3) Sync operations normally* should not* be implemented through async. This
> > is a long story - if we delegate to async, then we have to bother with
> > additional threads, associated back-pressure control and all that crap.
> > Sync call must be sync unless there is a very strong reason to go through
> > async path.
> >
> Not sure about this, though. In most cases a cache operation implies
> request/response over the network, so I think we should have explicit
> synchronous counterparts only for methods that are guaranteed to be local.
>

Re: Rework "withAsync" in Apache 2.0

[Alexey Goncharuk]

Big +1 on this in general.

I would also relax our guarantees on operations submitted from the same
thread. Currently we guarantee that sequential invocations of async
operations happen in the same order. I think that if a user wants such
guarantees, he must define these dependencies explicitly by calling chain()
on returning futures.

This change will significantly improve cache operations performance in
async mode.

3) Sync operations normally* should not* be implemented through async. This
> is a long story - if we delegate to async, then we have to bother with
> additional threads, associated back-pressure control and all that crap.
> Sync call must be sync unless there is a very strong reason to go through
> async path.
>
Not sure about this, though. In most cases a cache operation implies
request/response over the network, so I think we should have explicit
synchronous counterparts only for methods that are guaranteed to be local.

[jira] [Created] (IGNITE-3519) Web console: add ability to specify a node filter in start caches configuration

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-3519:
--

 Summary: Web console: add ability to specify a node filter in 
start caches configuration
 Key: IGNITE-3519
 URL: https://issues.apache.org/jira/browse/IGNITE-3519
 Project: Ignite
  Issue Type: Sub-task
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Rework "withAsync" in Apache 2.0

[Vladimir Ozerov]

Hi folks,

There was a discussion about not very good "withAsync()" API design [1]. As
we are approaching Ignite 2.0 it is a good time to rework it. My
considerations on how good API should look like and how should it be
implemented:

1) Sync and async must be *different interfaces*.
2) All async methods must return *futures*. No thread-locals.
3) Sync operations normally* should not* be implemented through async. This
is a long story - if we delegate to async, then we have to bother with
additional threads, associated back-pressure control and all that crap.
Sync call must be sync unless there is a very strong reason to go through
async path.
4) Last, but not least - we should re-design our futures to be more like
Java 8 *CompletableFuture*. It should be able to accept Executors to
continue computation, it should have lots of methods to deal with
continuations and joins, etc.. Ideally, it would be cool if could remove it
and use CompletableFuture, but it is clearly too early to drop Java 7
support. So we should at least try making our futures similar to
CompletableFuture.

I will create relevant tickets soon.

Thoughts?

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Net-separate-methods-for-async-operations-td3901.html