Re: binary output from cache

[ALEKSEY KUZNETSOV]

thanks for explanation

пт, 10 Фев 2017 г., 20:12 Alexey Goncharuk :

> Aleksey,
>
> It can give a performance benefit for example if you access only a subset
> of the fields of a large object. In this case, deserializing only required
> fields is much faster than deserializing the whole object.
>
> --AG
>
> 2017-02-10 10:26 GMT+03:00 ALEKSEY KUZNETSOV :
>
> > How could it give perfomance improvement ?
> >
> > чт, 9 февр. 2017 г. в 23:21, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Alexey,
> > >
> > > Generally users want to work with their model objects, so we
> deserialize
> > > them on read by default. BinaryObject API is usually used in more
> > > complicated use cases, like when values are read on servers side, or
> when
> > > it gives performance improvement.
> > >
> > > -Val
> > >
> > > On Thu, Feb 9, 2017 at 10:36 AM, ALEKSEY KUZNETSOV <
> > > alkuznetsov...@gmail.com
> > > > wrote:
> > >
> > > > Guys, i have a dumb question. Why do we need to set withKeepBinary()?
> > > What
> > > > i mean is why dont we just have it hardcoded ?
> > > >
> > > > --
> > > >
> > > > *Best Regards,*
> > > >
> > > > *Kuznetsov Aleksey*
> > > >
> > >
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

Re: IGNITE-13

[Dmitriy Setrakyan]

On Thu, Feb 9, 2017 at 11:26 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Vadim,
>
> I don't think it makes much sense to invest into OptimizedMarshaller.
> However, I would check if this optimization is applicable to
> BinaryMarshaller, and if yes, implement it.
>

Val, in this case can you please update the ticket?


>
> -Val
>
> On Thu, Feb 9, 2017 at 11:05 PM, Вадим Опольский 
> wrote:
>
> > Dear sirs!
> >
> > I want to resolve issue IGNITE-13 -
> > https://issues.apache.org/jira/browse/IGNITE-13
> >
> > Is it actual?
> >
> > Vadim Opolski
> >
>

Re: Rethink native SQL API in Apache Ignite 2.0

[Dmitriy Setrakyan]

I think the proper process here is to create a JIRA ticket and propose the
design there.

D.

On Fri, Feb 10, 2017 at 1:37 AM, Sergi Vladykin 
wrote:

> I agree with Vladimir that we have to improve our API with respect to the
> latest changes in our SQL capabilities.
>
> As for the given API proposal, it is a bit harsh right now, but it is ok
> for the first draft, overall direction looks good.
>
> Also IMO we should not drop the existing API, thus adding a new one is not
> urgent for 2.0, because adding new API is not a breaking change.
>
> Sergi
>
> 2017-02-10 4:55 GMT+03:00 Dmitriy Setrakyan :
>
> > Vladimir,
> >
> > Will these changes be backward compatible?
> >
> > Sergi, given your SQL expertise, can you please comment here as well?
> >
> > D.
> >
> > On Tue, Feb 7, 2017 at 3:34 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Our SQL engine becomes more and more sophisticated. Initially we had
> only
> > > SELECTs, now we have DML, in AI 2.x we will have DDL.
> > >
> > > This is very cool, but it seems that we completely forgot about
> extending
> > > our native SQL API (IgniteCache.query, SqlQuery, SqlFieldsQuery) in
> > > response to these changes. For example:
> > >
> > > 1) How one should execute DML/DDL command and validate the result?
> > > int updateCnt = IgniteCache.query(SqlQuery).getAll().get(0);
> > >
> > > Counter-intuitive and too verbose.
> > >
> > > 2) How one should perform batched operations? One by one?
> > > IgniteCache.query(SqlQuery.setArgs(1));
> > > IgniteCache.query(SqlQuery.setArgs(2));
> > > ...
> > > IgniteCache.query(SqlQuery.setArgs(N));
> > >
> > > I think it is time to rework our API so that it supports all recent
> > > features in consistent way and is extensible enough for future
> > improvements
> > > (e.g. transactional SQL).
> > >
> > > Probably we can take ideas behind JDBC standard as starting point and
> > move
> > > SQL to separate API. Several very rough examples:
> > >
> > > 1) Getting facade:
> > > IgniteSql sql = ignite.sql("MY_SCHEMA");
> > >
> > > 2) Running SELECT:
> > > QueryCursor cursor = sql.select(SqlCommand);
> > > QueryCursor cursor = sql.selectFields(SqlCommand); // No more
> > need
> > > for separate SqlQuery and SqlQueryFields classes.
> > >
> > > 3) Running DML/DDL:
> > > SqlCommandResult res = sql.execute(SqlCommand);
> > > or
> > > int updatedCnt = sql.execute(SqlCommand);
> > >
> > > 4) Running batch commands:
> > > SqlCommand cmd = new SqlCommand(...).addBatch(arg1).addBatch(arg2);
> > > SqlCommandResult res = sql.execute(cmd);
> > >
> > > 5) Re-use query parsing results (PreparedStatement):
> > > SqlCommand qry = sql.prepare("SELECT ...");
> > >
> > > Our JDBC driver is not applicable here because it is either not
> peformant
> > > enough (V1), or starts unnecessary client inside (V2).
> > >
> > > Thoughts? Does anyone else think it is time to re-approach SQL API?
> > >
> > > Vladimir.
> > >
> >
>

Re: jdbc vs jdbc2 packages

[Dmitriy Setrakyan]

I generally like the idea of supporting thin JDBC clients. Often it is not
feasible to start a client node on JDBC side.

On Fri, Feb 10, 2017 at 1:04 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Yakov,
>
> There are two separate aspects that we can improve:
>
> 1. Message routing as you described. I agree it can be useful in some
> scenarios and I definitely not against this feature, but honestly I haven't
> seen a lot of requests for this so far.
> 2. Server can initiate connection with client. This really blows users's
> minds :)
>
> I only meant that the latter is much more critical in my view. Configuring
> port forwarding on the cluster can be complicated, but it absolutely makes
> sense, while doing the same on client just sounds crazy. Client should be
> able to just connect, without having public IP and without additional
> configuration on the router.
>
> -Val
>
> On Fri, Feb 10, 2017 at 1:58 AM, Yakov Zhdanov 
> wrote:
>
> > Val, can you please explain your statement. You suggest users have port
> > forwarding for dozens of nodes in their topologies so client can initiate
> > connect to any of them? I don't think this is possible and routing seems
> to
> > be the only possible option.
> >
> > --Yakov
> >
> > 2017-02-10 0:04 GMT+03:00 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Yakov,
> > >
> > > 1. Yes, I will file a ticket.
> > >
> > > 4. I meant that server can currently initiate connection with client,
> and
> > > that's the main problem here. Is there a way to avoid this? Message
> > routing
> > > you're referring to can also be useful in some cases, but much less
> > > critical.
> > >
> > > -Val
> > >
> > > On Wed, Feb 8, 2017 at 9:20 PM, Yakov Zhdanov 
> > > wrote:
> > >
> > > > Val,
> > > >
> > > > 1. Our clients should stop require persistent store implementation if
> > > they
> > > > do not need it. Can you please file a ticket? I know you fixed some
> > > places
> > > > already. As an idea I would keep everything in binary format until we
> > > > really need it. Will that work?
> > > > 2. We can try adding the very first step to fetch the configuration
> and
> > > > then proceed with normal start.
> > > > https://issues.apache.org/jira/browse/IGNITE-4675
> > > > 3. Agree, but user needs to define the closures then. I would think
> on
> > > how
> > > > to put this to a product.
> > > > 4. This needs to be implemented :) I mean we can communicate to a
> > client
> > > > through server it is connected to.
> > > >
> > > > Thanks!
> > > > --
> > > > Yakov Zhdanov, Director R
> > > > *GridGain Systems*
> > > > www.gridgain.com
> > > >
> > > > 2017-02-09 5:19 GMT+07:00 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com
> > > > >:
> > > >
> > > > > Yakov,
> > > > >
> > > > > I agree that investing in the legacy client doesn't make sense -
> it's
> > > > slow
> > > > > and outdated. Regarding your points:
> > > > >
> > > > > 1. This is just another build step, but the JAR is going to pretty
> > fat
> > > I
> > > > > think (it will have to include Spring). Not ideal, but definitely
> > > better
> > > > > than what we have now. However, our clients also often require some
> > > user
> > > > > classes, like CacheStore implementations. This is also a problem.
> > > > >
> > > > > 2. That's a great idea! Actually, I'm not sure why we require to
> have
> > > > full
> > > > > verbose config on client that is consistent with server. Why not
> > fetch
> > > > the
> > > > > configuration from cluster during join? Not sure how hard this
> change
> > > is,
> > > > > but it can be a very big usability improvement. And surely, JDBC
> > driver
> > > > > should be able to config with host:port without config file.
> > > > >
> > > > > 3. This can be already achieved with Compute Grid, no? I don't
> think
> > we
> > > > > need to add anything here.
> > > > >
> > > > > Another issue with clients is that they currently can't work behind
> > NAT
> > > > > without additional config which is not very trivial
> > (AddressResolver).
> > > Is
> > > > > it possible to avoid server->client connections in communication
> SPI?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Feb 7, 2017 at 9:09 PM, Yakov Zhdanov  >
> > > > wrote:
> > > > >
> > > > > > "undeprecating" - lol :D
> > > > > > Consider introducing @Un annotation which negates all annotations
> > on
> > > > the
> > > > > > same level and below.
> > > > > >
> > > > > > I would probably agree with Val and Vova, but adding features to
> > > > > > thin-client seems questionable to me.
> > > > > >
> > > > > > Is these possible:
> > > > > > 1. avoid dependencies on client machine and require
> ignite-jdbc.jar
> > > > only
> > > > > > (e.g. gathering dependencies inside the jar).
> > > > > > 2. make it possible to provide just address and port to send join
> > > > request
> > > > > > to without providing the entire IgniteConfiguration. Client 

Re: DML data streaming

[Dmitriy Setrakyan]

On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov 
wrote:

> I propose to ship streaming with INSERT support only for now. This is
> enough for multitude cases and will add value to Ignite 1.9 immediately. We
> can think about correct streaming UPDATE/DELETE architecture separately .It
> is much more difficult thing, we cannot support it in a clean way right now
> due to multiple "_key" and "_val" usages over the code base.
>

Vova, I disagree. If all parts of the key are present, then we can always
construct a key in all cases. For these operations we can always support
streaming. For all other operations, we can delegate to standard MR, but
still perform most operations on the same node, as I suggested in another
email.

Re: DML data streaming

[Dmitriy Setrakyan]

On Fri, Feb 10, 2017 at 12:55 AM, Alexander Paschenko <
alexander.a.pasche...@gmail.com> wrote:

> And to avoid further confusion: UPDATE and DELETE are simply
> impossible in streaming mode when the key is not completely defined as
> long as data streamer operates with key-value pairs and not just
> tuples of named values. That's why we can't do DELETE from Person
> WHERE id1 = 5 from prev example with streamer - the Key { id1 = 5, id2
> = 0 } that would be constructed from such query is just one key and is
> handled by streamer as such while semantically that query is not about
> ONE key but about ALL keys where id1 = 5.
>

I completely agree. However, we should still optimize the MR here, since
the keys selected from one table (or cache) will probably end up on the
same node as the same keys inserted, updated, or deleted in another cache,
so these operations will likely still be local to the node.

Re: DML data streaming

[Dmitriy Setrakyan]

On Fri, Feb 10, 2017 at 12:49 AM, Alexander Paschenko <
alexander.a.pasche...@gmail.com> wrote:

> Dima,
> >
> > There are several ways to handle it. I would check how other databases
> > handle it, maybe we can borrow something. To the least, we should log
> such
> > errors in the log for now.
> >
>
> Logging errors would mean introducing some kind of stream receiver to
> do that and thus that would be really the same performance penalty for
> the successful operations. I think we should go with that optional
> flag for semantics after all.
>

I am OK  with introducing some error trap and plug it into configuration
(maybe some interface with onError(...) callback). However, we should never
swallow error, we should always print all errors to the log.  Let's not
worry about the performance in case of errors.

>
> > You don't have to use _key. Primary key is usually a field in the class,
> so
> > you can use a normal column name. In any case, we should remove any usage
> > of _key before 2.0 is released.
> >
> > Again, if user does not have to specify _key on INSERT, then it is very
> > unclear to me, why user would need to specify _key for UPDATE or DELETE.
> > Something smells here. Can you please provide an example?
> >
>
> UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
> optimized cases - i.e. those where _key (and possibly _val) are
> explicitly specified by the user thus allowing us to map UPDATE and
> DELETE directly to cache's replace and remove operations without
> messing with entry processors and doing map-reduce SELECT by given
> criteria.
>
> Say, we have Person { firstName, secondName } with key class Key { id1,
> id2 }
>
> If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
> there's no need to do any SELECT - we can just call IgniteCache.remove
> on that key.
>
> But if I say DELETE from Person WHERE id1 = 5 then there's no way to
> avoid MR - we have to find all keys that interest us first by doing
> SELECT as long as we know only partly about what keys the user wants
> to be affected.
>
> It works in the same way for UPDATE. And I hope that it's clear how
> it's different from INSERT - there's no MR by definition (we don't
> allow INSERT FROM SELECT in streaming mode).
>

Do we allow INSERT from SELECT in non-streaming mode?


>
> AGAIN: this all is said only about streaming mode; non streaming mode
> does those optimizations too, but it also allows complex conditions,
> while streaming mode does not allow them to keep things fast and avoid
> MR.
>
> That's the reason why I suggest that we drop UPDATE and DELETE from
> DML streaming as they mean messing with those soon-hidden columns.
>
> Still we could optimize stuff like DELETE from Person WHERE id1 = 5
> AND id2 = 6 - query involves ALL fields of key AND compares only for
> equality AND has no complex expressions - we can construct key
> unambiguously and still call remove directly.
>

Exactly my point. If all key fields are present, we can construct the key
ourselves and still delegate to cache.put(..) or cache.remove(..). For all
cases where all the key fields are not present we should do regular MR. I
am assuming that this applies to UPDATE and DELETE operation. My vote is to
implement this functionality.


>
> But to me it does not sound like a really great reason to leave UPDATE
> and DELETE in DML - the users will have to write some specific queries
> to use that while all other stuff will just be declined in that mode.
> And, as I said before, UPDATE and DELETE don't probably perfectly fit
> with primary data streamer use cases - after all, modifying existing
> stuff is not what data streamer is about.
>

I am not sure what this means. We have to work in the same way as regular
RDBMS systems. I would not try to reinvent the bicycle here. All UPDATE,
DELETE, and INSERT operations should be part of DML.


>
> And regarding hiding columns: it's unclear how things will look like
> for caches like  when we remove _key and _val as long as
> tables for such cases currently have nothing but those two columns.
>

Again, think about standard RDBMS systems. None of them have _key or _val,
and therefore neither should we.

Re: IGNITE-817: [Test] Disabled tests of GridCacheOffHeapTest

[Denis Magda]

Yakov, Sam?

Please join and share your thoughts.

—
Denis

> On Feb 10, 2017, at 7:08 AM, Александр Меньшиков  wrote:
> 
> Denis, thank you for information. I see JMH in pom file in benchmarks
> module. So this must be removed or it's okay for micro-benchmarks?
> 
> 2017-02-10 0:04 GMT+03:00 Denis Magda :
> 
>> Alexander,
>> 
>> We use Yardstick benchmarks for performance measurements of Ignite:
>> https://github.com/apacheignite/yardstick-ignite > apacheignite/yardstick-ignite>
>> 
>> In 1.9 it will be much easier to execute the benchmarks directly from
>> Apache Ignite bundle:
>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4212-Ignite-
>> Benchmarking-Simplification-and-Automation-td13079.html <
>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4212-Ignite-
>> Benchmarking-Simplification-and-Automation-td13079.html>
>> 
>> As for GridCacheOffHeapTest it looks like a candidate for removal. *Yakov,
>> Sam*, do we still need this test?
>> 
>>> On Feb 9, 2017, at 6:25 AM, Александр Меньшиков 
>> wrote:
>>> 
>>> Hello, everyone.
>>> 
>>> GridCacheOffHeapTest  is about the Off-heap benchmarks.
>>> 
>>> As I understand it right now we haven't clear vision how better make
>>> benchmarks in Ignite.
>>> 
>>> If it's not true, please make me know.
>>> 
>>> I think it's a good idea to make benchmarks with JMH framework. It can be
>>> load with maven, right now it's part of OpenJDK project. But it will be
>>> included in Java 9 like standart framework for benchmarks.
>>> 
>>> So what do you think, Igniters?
>> 
>> 

Re: DML data streaming

[Denis Magda]

In general, the data streamer approach should be mostly used for data loading 
scenarios. The data is usually loaded with INSERTS which means that the 
scenario is already supported and we’re free to merge the changes to 1.9.

If you UPDATE or DELETE data in the streaming mode then you are required to set 
dataStreamer.allowOverwrite = true, making sure that the updates coming from 
the streamer side are consistent with transactions that might be executed in 
parallel. In this mode the streamer switches to a slower mode pushing the data 
with cache.writeAll() and cache.removeAll() methods. 

At all, considering real-life use cases it’s more than enough to support the 
streaming mode for INSERTS only and describe it properly in the documentation.

—
Denis

> On Feb 10, 2017, at 3:36 AM, Vladimir Ozerov  wrote:
> 
> I propose to ship streaming with INSERT support only for now. This is
> enough for multitude cases and will add value to Ignite 1.9 immediately. We
> can think about correct streaming UPDATE/DELETE architecture separately .It
> is much more difficult thing, we cannot support it in a clean way right now
> due to multiple "_key" and "_val" usages over the code base.
> 
> On Fri, Feb 10, 2017 at 11:55 AM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
> 
>> And to avoid further confusion: UPDATE and DELETE are simply
>> impossible in streaming mode when the key is not completely defined as
>> long as data streamer operates with key-value pairs and not just
>> tuples of named values. That's why we can't do DELETE from Person
>> WHERE id1 = 5 from prev example with streamer - the Key { id1 = 5, id2
>> = 0 } that would be constructed from such query is just one key and is
>> handled by streamer as such while semantically that query is not about
>> ONE key but about ALL keys where id1 = 5.
>> 
>> - Alex
>> 
>> 2017-02-10 11:49 GMT+03:00 Alexander Paschenko
>> :
>>> Dima,
 
 There are several ways to handle it. I would check how other databases
 handle it, maybe we can borrow something. To the least, we should log
>> such
 errors in the log for now.
 
>>> 
>>> Logging errors would mean introducing some kind of stream receiver to
>>> do that and thus that would be really the same performance penalty for
>>> the successful operations. I think we should go with that optional
>>> flag for semantics after all.
>>> 
 You don't have to use _key. Primary key is usually a field in the
>> class, so
 you can use a normal column name. In any case, we should remove any
>> usage
 of _key before 2.0 is released.
 
 Again, if user does not have to specify _key on INSERT, then it is very
 unclear to me, why user would need to specify _key for UPDATE or DELETE.
 Something smells here. Can you please provide an example?
 
>>> 
>>> UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
>>> optimized cases - i.e. those where _key (and possibly _val) are
>>> explicitly specified by the user thus allowing us to map UPDATE and
>>> DELETE directly to cache's replace and remove operations without
>>> messing with entry processors and doing map-reduce SELECT by given
>>> criteria.
>>> 
>>> Say, we have Person { firstName, secondName } with key class Key { id1,
>> id2 }
>>> 
>>> If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
>>> there's no need to do any SELECT - we can just call IgniteCache.remove
>>> on that key.
>>> 
>>> But if I say DELETE from Person WHERE id1 = 5 then there's no way to
>>> avoid MR - we have to find all keys that interest us first by doing
>>> SELECT as long as we know only partly about what keys the user wants
>>> to be affected.
>>> 
>>> It works in the same way for UPDATE. And I hope that it's clear how
>>> it's different from INSERT - there's no MR by definition (we don't
>>> allow INSERT FROM SELECT in streaming mode).
>>> 
>>> AGAIN: this all is said only about streaming mode; non streaming mode
>>> does those optimizations too, but it also allows complex conditions,
>>> while streaming mode does not allow them to keep things fast and avoid
>>> MR.
>>> 
>>> That's the reason why I suggest that we drop UPDATE and DELETE from
>>> DML streaming as they mean messing with those soon-hidden columns.
>>> 
>>> Still we could optimize stuff like DELETE from Person WHERE id1 = 5
>>> AND id2 = 6 - query involves ALL fields of key AND compares only for
>>> equality AND has no complex expressions - we can construct key
>>> unambiguously and still call remove directly.
>>> 
>>> But to me it does not sound like a really great reason to leave UPDATE
>>> and DELETE in DML - the users will have to write some specific queries
>>> to use that while all other stuff will just be declined in that mode.
>>> And, as I said before, UPDATE and DELETE don't probably perfectly fit
>>> with primary data streamer use cases - after all, modifying existing
>>> stuff is not what 

Re: DML & Transactions

[Denis Magda]

Hello,

Yes, MVCC has already been being designed and developed by multiple community 
members including Sergi.

If you prefer to take a big feature for dev then I would suggest considering 
these tasks:
https://ignite.apache.org/community/contribute.html#pick-ticket 


—
Denis

> On Feb 9, 2017, at 2:22 AM, Vyacheslav Daradur  wrote:
> 
> Hello everyone.
> 
> Sergi, you wrote about the MVCC.
> 
> It's very interesting and perspective.
> 
> Someone already does it?
> 
> 2017-02-09 9:25 GMT+03:00 Dmitriy Setrakyan :
> 
>> On Wed, Feb 8, 2017 at 10:20 PM, Yakov Zhdanov 
>> wrote:
>> 
>>> Guys, I think we should explicitly state this on readme.io - Do not use
>>> DML
>>> within transactions.
>>> 
>>> And probably throw an exception on such attempts.
>>> 
>> 
>> Absolutely agree. I would say that DML within transactions is not supported
>> yet. We expert to have support for it in version 2.x.
>> 
>> 
>>> --Yakov
>>> 
>>> 2017-02-09 12:08 GMT+07:00 Sergi Vladykin :
>>> 
 Not yet, we need MVCC over PageMemory for this. Probably it will arrive
>>> in
 some 2.x version.
 
 Sergi
 
 2017-02-09 7:42 GMT+03:00 Yakov Zhdanov :
 
> Guys (esp Alex P and Sergi),
> 
> Do we preserve transactional semantics of updates?
> 
> In other words, does the following make sense? and when it does not?
> 
> startTx()
> insert into blabla...
> update bla bla...
> commit()
> 
> --Yakov
> 
 
>>> 
>> 

[GitHub] ignite pull request #1519: Ignite 4633

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/1519

Ignite 4633

DDL base stuff

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4633

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1519


commit 69b0500dc4b10abab4d37283d24b11ad5d99e898
Author: Alexander Paschenko 
Date:   2017-02-02T14:40:22Z

IGNITE-4633 Introduced component type for DDL statements processor

commit fd10c59a4599d5c759ac7ca3acdd8b20c2428266
Author: Alexander Paschenko 
Date:   2017-02-02T16:18:40Z

IGNITE-4633 first steps on distributed DDL API

commit 65d1667633e0c4aa100259fcb9b1d7f569930018
Author: Alexander Paschenko 
Date:   2017-02-03T17:34:50Z

IGNITE-4633 Distributed DDL API - continuing with INIT/ACK stage

commit 7888a1ff8b8263b7b0e5074612d1dbf76e2493a3
Author: Alexander Paschenko 
Date:   2017-02-06T14:52:21Z

IGNITE-4633 Distributed DDL API - added error/ack/res messages, msg 
factory, topic for DDL messages

commit d1727540c5b0d7c359e323de5a20e6fd7adb0ec4
Author: devozerov 
Date:   2017-02-06T15:44:00Z

Review comments.

commit 5567b7c637b48fc334d572493aa50c2812d0a886
Author: Alexander Paschenko 
Date:   2017-02-06T16:01:21Z

IGNITE-4633 More placeholders

commit e52759ae82df15f23e43b9e60c69dbdfd76a5662
Author: Alexander Paschenko 
Date:   2017-02-07T19:23:47Z

IGNITE-4633 Few more steps on DDL API messaging proto impl

commit 7e0d0872e9952e84fcb371642c253c83a11d
Author: Alexander Paschenko 
Date:   2017-02-08T16:05:04Z

IGNITE-4633 Logic and messaging fixes

commit 1af6649a71c8eec12af09bbc9438f7ffe5c8a062
Author: Alexander Paschenko 
Date:   2017-02-09T20:58:05Z

IGNITE-4633 Removed redundant message and clarified cancellation stuff

commit 6e4dc348463ed6233878b4b040dd0e3d4d0ea65f
Author: Alexander Paschenko 
Date:   2017-02-10T15:58:16Z

IGNITE-4633 DDL INIT/ACK finished.

commit f1d16180bbc7b131791396d6c6fbdbdd40a85516
Author: Alexander Paschenko 
Date:   2017-02-10T17:38:45Z

Test name extraction in GridTestUtils

commit 73d35c2ba214a37fa5422aa1a57d9c5df3e24abb
Author: Alexander Paschenko 
Date:   2017-02-10T22:37:32Z

IGNITE-4633 DDL future cancellation/processor stop




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: jdbc vs jdbc2 packages

[Valentin Kulichenko]

Yakov,

There are two separate aspects that we can improve:

1. Message routing as you described. I agree it can be useful in some
scenarios and I definitely not against this feature, but honestly I haven't
seen a lot of requests for this so far.
2. Server can initiate connection with client. This really blows users's
minds :)

I only meant that the latter is much more critical in my view. Configuring
port forwarding on the cluster can be complicated, but it absolutely makes
sense, while doing the same on client just sounds crazy. Client should be
able to just connect, without having public IP and without additional
configuration on the router.

-Val

On Fri, Feb 10, 2017 at 1:58 AM, Yakov Zhdanov  wrote:

> Val, can you please explain your statement. You suggest users have port
> forwarding for dozens of nodes in their topologies so client can initiate
> connect to any of them? I don't think this is possible and routing seems to
> be the only possible option.
>
> --Yakov
>
> 2017-02-10 0:04 GMT+03:00 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Yakov,
> >
> > 1. Yes, I will file a ticket.
> >
> > 4. I meant that server can currently initiate connection with client, and
> > that's the main problem here. Is there a way to avoid this? Message
> routing
> > you're referring to can also be useful in some cases, but much less
> > critical.
> >
> > -Val
> >
> > On Wed, Feb 8, 2017 at 9:20 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Val,
> > >
> > > 1. Our clients should stop require persistent store implementation if
> > they
> > > do not need it. Can you please file a ticket? I know you fixed some
> > places
> > > already. As an idea I would keep everything in binary format until we
> > > really need it. Will that work?
> > > 2. We can try adding the very first step to fetch the configuration and
> > > then proceed with normal start.
> > > https://issues.apache.org/jira/browse/IGNITE-4675
> > > 3. Agree, but user needs to define the closures then. I would think on
> > how
> > > to put this to a product.
> > > 4. This needs to be implemented :) I mean we can communicate to a
> client
> > > through server it is connected to.
> > >
> > > Thanks!
> > > --
> > > Yakov Zhdanov, Director R
> > > *GridGain Systems*
> > > www.gridgain.com
> > >
> > > 2017-02-09 5:19 GMT+07:00 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Yakov,
> > > >
> > > > I agree that investing in the legacy client doesn't make sense - it's
> > > slow
> > > > and outdated. Regarding your points:
> > > >
> > > > 1. This is just another build step, but the JAR is going to pretty
> fat
> > I
> > > > think (it will have to include Spring). Not ideal, but definitely
> > better
> > > > than what we have now. However, our clients also often require some
> > user
> > > > classes, like CacheStore implementations. This is also a problem.
> > > >
> > > > 2. That's a great idea! Actually, I'm not sure why we require to have
> > > full
> > > > verbose config on client that is consistent with server. Why not
> fetch
> > > the
> > > > configuration from cluster during join? Not sure how hard this change
> > is,
> > > > but it can be a very big usability improvement. And surely, JDBC
> driver
> > > > should be able to config with host:port without config file.
> > > >
> > > > 3. This can be already achieved with Compute Grid, no? I don't think
> we
> > > > need to add anything here.
> > > >
> > > > Another issue with clients is that they currently can't work behind
> NAT
> > > > without additional config which is not very trivial
> (AddressResolver).
> > Is
> > > > it possible to avoid server->client connections in communication SPI?
> > > >
> > > > -Val
> > > >
> > > > On Tue, Feb 7, 2017 at 9:09 PM, Yakov Zhdanov 
> > > wrote:
> > > >
> > > > > "undeprecating" - lol :D
> > > > > Consider introducing @Un annotation which negates all annotations
> on
> > > the
> > > > > same level and below.
> > > > >
> > > > > I would probably agree with Val and Vova, but adding features to
> > > > > thin-client seems questionable to me.
> > > > >
> > > > > Is these possible:
> > > > > 1. avoid dependencies on client machine and require ignite-jdbc.jar
> > > only
> > > > > (e.g. gathering dependencies inside the jar).
> > > > > 2. make it possible to provide just address and port to send join
> > > request
> > > > > to without providing the entire IgniteConfiguration. Client node
> > sends
> > > > join
> > > > > request to the cluster with flag that this is jdbc-driver
> connection
> > > and
> > > > > server-side topology omits configuration validation and forces
> client
> > > to
> > > > > set some properties if this is necessary (e.g. CommunicationSpi
> > > > > implementation class and settings)
> > > > > 3. add possibility to offload complex reduce processing to server.
> > > Which
> > > > > may be very helpful for main client-server use cases when clients
> > 

Re: IGNITE-4212 (Ignite Benchmarking Simplification and Automation)

[Denis Magda]

Oleg,

Thanks a lot, we’re almost done. Please address the latest comments from my 
side.
https://issues.apache.org/jira/browse/IGNITE-4212 


Sergey K., please take over from this point trying to use the feature created 
by Oleg for Ignite 1.9 benchmarking purposes.
If you’re happy with it then Oleg is free to merge the changes into 1.9 and 
release them there.

—
Denis

> On Feb 10, 2017, at 2:50 AM, Oleg Ostanin  wrote:
> 
> I'm sorry, there wasn't any problem with the build, it was local problem on
> my machine.
> 
> On Fri, Feb 10, 2017 at 11:54 AM, Oleg Ostanin 
> wrote:
> 
>> Hi.
>> 
>> I just discovered that some functionality does not work correctly. I will
>> present fixed version later.
>> 
>> On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
>> wrote:
>> 
>>> https://drive.google.com/open?id=0B9teTzJ9iIvHckl6eVQ3N0R4a3c
>>> 
>>> On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
>>> wrote:
>>> 
 I've fixed README.txt and created new DEVNOTES.txt. Also renamed sample
 benchmark configuration file to benchmark-sample.properties.
 Please look for the results:
 
 
 On Tue, Jan 31, 2017 at 9:18 PM, Denis Magda  wrote:
 
> Oleg,
> 
> Thanks for the clarification.
> 
> My opinion is that we should leave ‘modules/yardstick/README.txt’ and
> ‘modules/yardstick/DEVNOTES.txt’ either unchanged or have only those
> instructions there that explain how to build and run benchmarks from
> ‘modules/yardstick’. This existing files can refer to the sources and
> compiled benchmarks that are in Ignite binary releases but this should be 
> a
> couple of statements, no more.
> 
> As for the instructions related to the sources and binaries added to
> Ignite binaries, preferably they need to be added to auto-generated
> README.txt. *Anton*, is it feasible to do?
> 
> Finally, when apply the reviews notes please build and share Ignite
> binaries with your patch. Want to see the new project structure and final
> version of the instructions. Presently I can’t merge your changes due to
> some conflicts.
> 
> *Anton*, please review modifications in the build procedures.
> 
> —
> Denis
> 
>> On Jan 31, 2017, at 3:12 AM, Oleg Ostanin 
> wrote:
>> 
>> Hi Denis,
>> 
>> Yes, we have included Ignite Yardstick source files with its pom.xml
> in
>> Ignite binary release. "Building from standalone sources" is the
> building
>> instruction for these source files. "Building from Ignite Sources" is
> the
>> instruction for building Ignite Yardstick from `modules/yardstick` in
>> Ignite source files.
>> 
>> On Tue, Jan 31, 2017 at 3:31 AM, Denis Magda 
> wrote:
>> 
>>> Hi Oleg,
>>> 
>>> Great progress, thanks for keep driving this!
>>> 
>>> I’ve left some minor notes in GitHub’s pull-request. I have the
> following
>>> questions aside:
>>> 
>>> - What is the difference between "Building from standalone sources"
> and
>>> "Building from Ignite Sources"? In my understanding, a user downloads
>>> Apache Ignite release that has all the sources locally.
>>> 
>>> - I do remember we planned to add the benchmarks sources in a form
> of a
>>> ready to be used project with its own pom.xml (similar to examples).
> Did
>>> you put this task off?
>>> 
>>> —
>>> Denis
>>> 
 On Jan 27, 2017, at 2:13 AM, Oleg Ostanin 
> wrote:
 
 Hi!
 
 I've changed the README.txt and DEVNOTES.txt files. Also added a
> simple
 config file for quick and easy start. Please take a look at them
> and tell
 me what you think.
 
 https://github.com/apache/ignite/pull/1471
 
 On Wed, Dec 28, 2016 at 8:59 AM, Ilya Suntsov <
> isunt...@gridgain.com>
>>> wrote:
 
> Denis,
> 
> I think we can remove all configs except:
> 
> benchmark-multicast.properties
> 
> benchmark.properties
> 
> ignite-base-config.xml
> 
> ignite-localhost-config.xml
> 
> ignite-multicast-config.xml
> 
> 2016-12-28 2:49 GMT+03:00 Denis Magda :
> 
>> I would have only those configs that are useful. Ilya Suntsov,
> basing
>>> on
>> your experience, please suggest which configs makes sense to
> include
>>> into
>> every Ignite release.
>> 
>> Oleg, also please note that community decided to include not only
> the
>> benchmarking binaries but the sources as well into every Apache
> 

Re: binary output from cache

[Alexey Goncharuk]

Aleksey,

It can give a performance benefit for example if you access only a subset
of the fields of a large object. In this case, deserializing only required
fields is much faster than deserializing the whole object.

--AG

2017-02-10 10:26 GMT+03:00 ALEKSEY KUZNETSOV :

> How could it give perfomance improvement ?
>
> чт, 9 февр. 2017 г. в 23:21, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Alexey,
> >
> > Generally users want to work with their model objects, so we deserialize
> > them on read by default. BinaryObject API is usually used in more
> > complicated use cases, like when values are read on servers side, or when
> > it gives performance improvement.
> >
> > -Val
> >
> > On Thu, Feb 9, 2017 at 10:36 AM, ALEKSEY KUZNETSOV <
> > alkuznetsov...@gmail.com
> > > wrote:
> >
> > > Guys, i have a dumb question. Why do we need to set withKeepBinary()?
> > What
> > > i mean is why dont we just have it hardcoded ?
> > >
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

general question

[ALEKSEY KUZNETSOV]

Guys, i have a question.Why do we store fields position information in
FieldAccessor before hashcode evaluation? Test example is in
: 
org.apache.ignite.internal.binary.BinaryFieldIdentityResolverSelfTest#testHashCode
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[GitHub] ignite pull request #1473: IGNITE-3196

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4685) Propagate platform plugin configuration to Java

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-4685:
--

 Summary: Propagate platform plugin configuration to Java
 Key: IGNITE-4685
 URL: https://issues.apache.org/jira/browse/IGNITE-4685
 Project: Ignite
  Issue Type: Sub-task
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.0


{{IPluginConfiguration}} implementations should be able to propagate data to 
Java side and update corresponding {{PluginConfiguration}} there.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: IGNITE-3196 - ready for review

[Vyacheslav Daradur]

Thanks for your explanation.

2017-02-10 18:18 GMT+03:00 Pavel Tupitsyn :

> You can close it.
>
> On Fri, Feb 10, 2017 at 6:16 PM, Vyacheslav Daradur 
> wrote:
>
> > I meant what I need to do with opened PR?
> > I need to close it or to leave it open for future merge?
> >
> > 2017-02-10 17:42 GMT+03:00 Pavel Tupitsyn :
> >
> > > The ticket is targeted for 2.0 because this change may affect existing
> > > code.
> > > 1.9 is planned in the near future, and minor versions should not break
> > > existing code.
> > >
> > > Pavel
> > >
> > > On Fri, Feb 10, 2017 at 5:24 PM, Vyacheslav Daradur <
> daradu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Pavel, thanks.
> > > >
> > > > What about PR to master-branch?
> > > >
> > > > 2017-02-10 16:55 GMT+03:00 Pavel Tupitsyn :
> > > >
> > > > > Merged to ignite-2.0.
> > > > >
> > > > > Thank you for the contribution, Vyacheslav!
> > > > >
> > > > > On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda 
> > > wrote:
> > > > >
> > > > > > + Vladimir Ozerov
> > > > > >
> > > > > > It would be better if Vladimir Ozerov does the final review
> > > considering
> > > > > > all the changes in .NET, C++ and Java.
> > > > > >
> > > > > > Vladimir, could you do that?
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur <
> > > daradu...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > +Denis
> > > > > > >
> > > > > > > >>Ok, so we agree on .NET and C++ parts, only Java part is to
> be
> > > > > > reviewed.
> > > > > > > >>Denis, can you have a look?
> > > > > > >
> > > > > > > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn <
> ptupit...@apache.org
> > > > >  > > > > > ptupit...@apache.org>>:
> > > > > > > Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > > > reviewed.
> > > > > > >
> > > > > > > Denis, can you have a look?
> > > > > > >
> > > > > > > Pavel
> > > > > > >
> > > > > > > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego <
> > isap...@gridgain.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks good to me.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Igor
> > > > > > > >
> > > > > > > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> > > > > > daradu...@gmail.com >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok, thanks for explanations.
> > > > > > > > >
> > > > > > > > > What about this task?
> > > > > > > > >
> > > > > > > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego <
> isap...@gridgain.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > >> But that's Ok. Since we use int8_t for bytes in C++ as
> well
> > I
> > > > > guess
> > > > > > > > >> your -0x80 may have more sense than 0x80.
> > > > > > > > >>
> > > > > > > > >> Best Regards,
> > > > > > > > >> Igor
> > > > > > > > >>
> > > > > > > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego <
> > > > isap...@gridgain.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> I was just curious.
> > > > > > > > >>>
> > > > > > > > >>> In C++ both constants 0x80 and -0x80 are of type 'int'
> and
> > > have
> > > > > the
> > > > > > > > same
> > > > > > > > >>> lower byte, so they give the same result. Though first
> > number
> > > > is
> > > > > > > > actually
> > > > > > > > >>> 0x0080 when the second one is 0xFF80.
> > > > > > > > >>>
> > > > > > > > >>> So it's just made a minus sign look a little redundant
> and
> > > > > > pointless to
> > > > > > > > >>> me
> > > > > > > > >>> in C++ code.
> > > > > > > > >>>
> > > > > > > > >>> Best Regards,
> > > > > > > > >>> Igor
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > > > > > > daradu...@gmail.com 
> > > > > > > > >>> > wrote:
> > > > > > > > >>>
> > > > > > > >  Byte.MIN_VALUE = -128 = -0x80
> > > > > > > >  Byte.MAX_VALUE = 127 = 0x7F
> > > > > > > > 
> > > > > > > >  It is just more evident for me.
> > > > > > > > 
> > > > > > > >  Maybe, I just have the Java programming style.
> > > > > > > > 
> > > > > > > >  In Java:
> > > > > > > >  byte a = 100 | -0x80;  // compiled
> > > > > > > >  byte b = 100 | 0x80;  // doesn't compile, explicit type
> > > > casting
> > > > > is
> > > > > > > >  neccessary (byte)(100 | 0x80)
> > > > > > > >  System.out.println(a | -0x80); // -28
> > > > > > > >  System.out.println(a | 0x80); // 228 - cast to int
> > > > > > > > 
> > > > > > > >  Is it bad style?
> > > > > > > > 
> > > > > > > >  2017-02-06 20:04 GMT+03:00 Igor Sapego <
> > > isap...@gridgain.com
> > > > > > >:
> > > > > > > > 
> > > > > > > > > Vyacheslav,
> > > > > > > > >
> > > > > > > > > Overall looks good. But why do 

Re: IGNITE-3196 - ready for review

[Pavel Tupitsyn]

You can close it.

On Fri, Feb 10, 2017 at 6:16 PM, Vyacheslav Daradur 
wrote:

> I meant what I need to do with opened PR?
> I need to close it or to leave it open for future merge?
>
> 2017-02-10 17:42 GMT+03:00 Pavel Tupitsyn :
>
> > The ticket is targeted for 2.0 because this change may affect existing
> > code.
> > 1.9 is planned in the near future, and minor versions should not break
> > existing code.
> >
> > Pavel
> >
> > On Fri, Feb 10, 2017 at 5:24 PM, Vyacheslav Daradur  >
> > wrote:
> >
> > > Pavel, thanks.
> > >
> > > What about PR to master-branch?
> > >
> > > 2017-02-10 16:55 GMT+03:00 Pavel Tupitsyn :
> > >
> > > > Merged to ignite-2.0.
> > > >
> > > > Thank you for the contribution, Vyacheslav!
> > > >
> > > > On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda 
> > wrote:
> > > >
> > > > > + Vladimir Ozerov
> > > > >
> > > > > It would be better if Vladimir Ozerov does the final review
> > considering
> > > > > all the changes in .NET, C++ and Java.
> > > > >
> > > > > Vladimir, could you do that?
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur <
> > daradu...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > +Denis
> > > > > >
> > > > > > >>Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > > > reviewed.
> > > > > > >>Denis, can you have a look?
> > > > > >
> > > > > > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn  > > >  > > > > ptupit...@apache.org>>:
> > > > > > Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > > reviewed.
> > > > > >
> > > > > > Denis, can you have a look?
> > > > > >
> > > > > > Pavel
> > > > > >
> > > > > > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego <
> isap...@gridgain.com
> > > > > > wrote:
> > > > > >
> > > > > > > Looks good to me.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Igor
> > > > > > >
> > > > > > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> > > > > daradu...@gmail.com >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Ok, thanks for explanations.
> > > > > > > >
> > > > > > > > What about this task?
> > > > > > > >
> > > > > > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego  > > > > >:
> > > > > > > >
> > > > > > > >> But that's Ok. Since we use int8_t for bytes in C++ as well
> I
> > > > guess
> > > > > > > >> your -0x80 may have more sense than 0x80.
> > > > > > > >>
> > > > > > > >> Best Regards,
> > > > > > > >> Igor
> > > > > > > >>
> > > > > > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego <
> > > isap...@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> I was just curious.
> > > > > > > >>>
> > > > > > > >>> In C++ both constants 0x80 and -0x80 are of type 'int' and
> > have
> > > > the
> > > > > > > same
> > > > > > > >>> lower byte, so they give the same result. Though first
> number
> > > is
> > > > > > > actually
> > > > > > > >>> 0x0080 when the second one is 0xFF80.
> > > > > > > >>>
> > > > > > > >>> So it's just made a minus sign look a little redundant and
> > > > > pointless to
> > > > > > > >>> me
> > > > > > > >>> in C++ code.
> > > > > > > >>>
> > > > > > > >>> Best Regards,
> > > > > > > >>> Igor
> > > > > > > >>>
> > > > > > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > > > > > daradu...@gmail.com 
> > > > > > > >>> > wrote:
> > > > > > > >>>
> > > > > > >  Byte.MIN_VALUE = -128 = -0x80
> > > > > > >  Byte.MAX_VALUE = 127 = 0x7F
> > > > > > > 
> > > > > > >  It is just more evident for me.
> > > > > > > 
> > > > > > >  Maybe, I just have the Java programming style.
> > > > > > > 
> > > > > > >  In Java:
> > > > > > >  byte a = 100 | -0x80;  // compiled
> > > > > > >  byte b = 100 | 0x80;  // doesn't compile, explicit type
> > > casting
> > > > is
> > > > > > >  neccessary (byte)(100 | 0x80)
> > > > > > >  System.out.println(a | -0x80); // -28
> > > > > > >  System.out.println(a | 0x80); // 228 - cast to int
> > > > > > > 
> > > > > > >  Is it bad style?
> > > > > > > 
> > > > > > >  2017-02-06 20:04 GMT+03:00 Igor Sapego <
> > isap...@gridgain.com
> > > > > >:
> > > > > > > 
> > > > > > > > Vyacheslav,
> > > > > > > >
> > > > > > > > Overall looks good. But why do you use -0x80 instead of
> > 0x80?
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Igor
> > > > > > > >
> > > > > > > > On Mon, Feb 6, 2017 at 5:36 PM, Vyacheslav Daradur <
> > > > > > > > daradu...@gmail.com > wrote:
> > > > > > > >
> > > > > > > >> Igor,
> > > > > > > >>
> > > > > > > >> I didn't change the CPP code 

Re: IGNITE-3196 - ready for review

[Vyacheslav Daradur]

I meant what I need to do with opened PR?
I need to close it or to leave it open for future merge?

2017-02-10 17:42 GMT+03:00 Pavel Tupitsyn :

> The ticket is targeted for 2.0 because this change may affect existing
> code.
> 1.9 is planned in the near future, and minor versions should not break
> existing code.
>
> Pavel
>
> On Fri, Feb 10, 2017 at 5:24 PM, Vyacheslav Daradur 
> wrote:
>
> > Pavel, thanks.
> >
> > What about PR to master-branch?
> >
> > 2017-02-10 16:55 GMT+03:00 Pavel Tupitsyn :
> >
> > > Merged to ignite-2.0.
> > >
> > > Thank you for the contribution, Vyacheslav!
> > >
> > > On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda 
> wrote:
> > >
> > > > + Vladimir Ozerov
> > > >
> > > > It would be better if Vladimir Ozerov does the final review
> considering
> > > > all the changes in .NET, C++ and Java.
> > > >
> > > > Vladimir, could you do that?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
> > > > wrote:
> > > > >
> > > > > +Denis
> > > > >
> > > > > >>Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > > reviewed.
> > > > > >>Denis, can you have a look?
> > > > >
> > > > > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn  > >  > > > ptupit...@apache.org>>:
> > > > > Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > reviewed.
> > > > >
> > > > > Denis, can you have a look?
> > > > >
> > > > > Pavel
> > > > >
> > > > > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego  > > > > wrote:
> > > > >
> > > > > > Looks good to me.
> > > > > >
> > > > > > Best Regards,
> > > > > > Igor
> > > > > >
> > > > > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> > > > daradu...@gmail.com >
> > > > > > wrote:
> > > > > >
> > > > > > > Ok, thanks for explanations.
> > > > > > >
> > > > > > > What about this task?
> > > > > > >
> > > > > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego  > > > >:
> > > > > > >
> > > > > > >> But that's Ok. Since we use int8_t for bytes in C++ as well I
> > > guess
> > > > > > >> your -0x80 may have more sense than 0x80.
> > > > > > >>
> > > > > > >> Best Regards,
> > > > > > >> Igor
> > > > > > >>
> > > > > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego <
> > isap...@gridgain.com
> > > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >>> I was just curious.
> > > > > > >>>
> > > > > > >>> In C++ both constants 0x80 and -0x80 are of type 'int' and
> have
> > > the
> > > > > > same
> > > > > > >>> lower byte, so they give the same result. Though first number
> > is
> > > > > > actually
> > > > > > >>> 0x0080 when the second one is 0xFF80.
> > > > > > >>>
> > > > > > >>> So it's just made a minus sign look a little redundant and
> > > > pointless to
> > > > > > >>> me
> > > > > > >>> in C++ code.
> > > > > > >>>
> > > > > > >>> Best Regards,
> > > > > > >>> Igor
> > > > > > >>>
> > > > > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > > > > daradu...@gmail.com 
> > > > > > >>> > wrote:
> > > > > > >>>
> > > > > >  Byte.MIN_VALUE = -128 = -0x80
> > > > > >  Byte.MAX_VALUE = 127 = 0x7F
> > > > > > 
> > > > > >  It is just more evident for me.
> > > > > > 
> > > > > >  Maybe, I just have the Java programming style.
> > > > > > 
> > > > > >  In Java:
> > > > > >  byte a = 100 | -0x80;  // compiled
> > > > > >  byte b = 100 | 0x80;  // doesn't compile, explicit type
> > casting
> > > is
> > > > > >  neccessary (byte)(100 | 0x80)
> > > > > >  System.out.println(a | -0x80); // -28
> > > > > >  System.out.println(a | 0x80); // 228 - cast to int
> > > > > > 
> > > > > >  Is it bad style?
> > > > > > 
> > > > > >  2017-02-06 20:04 GMT+03:00 Igor Sapego <
> isap...@gridgain.com
> > > > >:
> > > > > > 
> > > > > > > Vyacheslav,
> > > > > > >
> > > > > > > Overall looks good. But why do you use -0x80 instead of
> 0x80?
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Igor
> > > > > > >
> > > > > > > On Mon, Feb 6, 2017 at 5:36 PM, Vyacheslav Daradur <
> > > > > > > daradu...@gmail.com > wrote:
> > > > > > >
> > > > > > >> Igor,
> > > > > > >>
> > > > > > >> I didn't change the CPP code before approval approach.
> > > > > > >> I shall write directly, sorry.
> > > > > > >>
> > > > > > >> But I made CPP changes already.
> > > > > > >>
> > > > > > >> > TestEscConvertFunctionFloat
> > > > > > >> > TestEscConvertFunctionDouble.
> > > > > > >> These tests were passed
> > > > > > >>  viewQueued.html?itemId=445824
> > <
> > > > 

Re: IGNITE-817: [Test] Disabled tests of GridCacheOffHeapTest

[Александр Меньшиков]

Denis, thank you for information. I see JMH in pom file in benchmarks
module. So this must be removed or it's okay for micro-benchmarks?

2017-02-10 0:04 GMT+03:00 Denis Magda :

> Alexander,
>
> We use Yardstick benchmarks for performance measurements of Ignite:
> https://github.com/apacheignite/yardstick-ignite  apacheignite/yardstick-ignite>
>
> In 1.9 it will be much easier to execute the benchmarks directly from
> Apache Ignite bundle:
> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4212-Ignite-
> Benchmarking-Simplification-and-Automation-td13079.html <
> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4212-Ignite-
> Benchmarking-Simplification-and-Automation-td13079.html>
>
> As for GridCacheOffHeapTest it looks like a candidate for removal. *Yakov,
> Sam*, do we still need this test?
>
> > On Feb 9, 2017, at 6:25 AM, Александр Меньшиков 
> wrote:
> >
> > Hello, everyone.
> >
> > GridCacheOffHeapTest  is about the Off-heap benchmarks.
> >
> > As I understand it right now we haven't clear vision how better make
> > benchmarks in Ignite.
> >
> > If it's not true, please make me know.
> >
> > I think it's a good idea to make benchmarks with JMH framework. It can be
> > load with maven, right now it's part of OpenJDK project. But it will be
> > included in Java 9 like standart framework for benchmarks.
> >
> > So what do you think, Igniters?
>
>

Re: IGNITE-3196 - ready for review

[Pavel Tupitsyn]

The ticket is targeted for 2.0 because this change may affect existing code.
1.9 is planned in the near future, and minor versions should not break
existing code.

Pavel

On Fri, Feb 10, 2017 at 5:24 PM, Vyacheslav Daradur 
wrote:

> Pavel, thanks.
>
> What about PR to master-branch?
>
> 2017-02-10 16:55 GMT+03:00 Pavel Tupitsyn :
>
> > Merged to ignite-2.0.
> >
> > Thank you for the contribution, Vyacheslav!
> >
> > On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda  wrote:
> >
> > > + Vladimir Ozerov
> > >
> > > It would be better if Vladimir Ozerov does the final review considering
> > > all the changes in .NET, C++ and Java.
> > >
> > > Vladimir, could you do that?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur 
> > > wrote:
> > > >
> > > > +Denis
> > > >
> > > > >>Ok, so we agree on .NET and C++ parts, only Java part is to be
> > > reviewed.
> > > > >>Denis, can you have a look?
> > > >
> > > > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn  >  > > ptupit...@apache.org>>:
> > > > Ok, so we agree on .NET and C++ parts, only Java part is to be
> > reviewed.
> > > >
> > > > Denis, can you have a look?
> > > >
> > > > Pavel
> > > >
> > > > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego  > > > wrote:
> > > >
> > > > > Looks good to me.
> > > > >
> > > > > Best Regards,
> > > > > Igor
> > > > >
> > > > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> > > daradu...@gmail.com >
> > > > > wrote:
> > > > >
> > > > > > Ok, thanks for explanations.
> > > > > >
> > > > > > What about this task?
> > > > > >
> > > > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego  > > >:
> > > > > >
> > > > > >> But that's Ok. Since we use int8_t for bytes in C++ as well I
> > guess
> > > > > >> your -0x80 may have more sense than 0x80.
> > > > > >>
> > > > > >> Best Regards,
> > > > > >> Igor
> > > > > >>
> > > > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego <
> isap...@gridgain.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >>> I was just curious.
> > > > > >>>
> > > > > >>> In C++ both constants 0x80 and -0x80 are of type 'int' and have
> > the
> > > > > same
> > > > > >>> lower byte, so they give the same result. Though first number
> is
> > > > > actually
> > > > > >>> 0x0080 when the second one is 0xFF80.
> > > > > >>>
> > > > > >>> So it's just made a minus sign look a little redundant and
> > > pointless to
> > > > > >>> me
> > > > > >>> in C++ code.
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>> Igor
> > > > > >>>
> > > > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > > > daradu...@gmail.com 
> > > > > >>> > wrote:
> > > > > >>>
> > > > >  Byte.MIN_VALUE = -128 = -0x80
> > > > >  Byte.MAX_VALUE = 127 = 0x7F
> > > > > 
> > > > >  It is just more evident for me.
> > > > > 
> > > > >  Maybe, I just have the Java programming style.
> > > > > 
> > > > >  In Java:
> > > > >  byte a = 100 | -0x80;  // compiled
> > > > >  byte b = 100 | 0x80;  // doesn't compile, explicit type
> casting
> > is
> > > > >  neccessary (byte)(100 | 0x80)
> > > > >  System.out.println(a | -0x80); // -28
> > > > >  System.out.println(a | 0x80); // 228 - cast to int
> > > > > 
> > > > >  Is it bad style?
> > > > > 
> > > > >  2017-02-06 20:04 GMT+03:00 Igor Sapego  > > >:
> > > > > 
> > > > > > Vyacheslav,
> > > > > >
> > > > > > Overall looks good. But why do you use -0x80 instead of 0x80?
> > > > > >
> > > > > > Best Regards,
> > > > > > Igor
> > > > > >
> > > > > > On Mon, Feb 6, 2017 at 5:36 PM, Vyacheslav Daradur <
> > > > > > daradu...@gmail.com > wrote:
> > > > > >
> > > > > >> Igor,
> > > > > >>
> > > > > >> I didn't change the CPP code before approval approach.
> > > > > >> I shall write directly, sorry.
> > > > > >>
> > > > > >> But I made CPP changes already.
> > > > > >>
> > > > > >> > TestEscConvertFunctionFloat
> > > > > >> > TestEscConvertFunctionDouble.
> > > > > >> These tests were passed
> > > > > >>  <
> > > http://ci.ignite.apache.org/viewQueued.html?itemId=445824>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> 2017-02-06 13:20 GMT+03:00 Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >:
> > > > > >>
> > > > > >>> .NET changes look good to me.
> > > > > >>>
> > > > > >>> Pavel
> > > > > >>>
> > > > > >>> On Mon, Feb 6, 2017 at 1:10 PM, Igor Sapego <
> > > isap...@gridgain.com >
> > 

[jira] [Created] (IGNITE-4684) Local cache.get(K) should not block

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4684:
-

 Summary: Local cache.get(K) should not block
 Key: IGNITE-4684
 URL: https://issues.apache.org/jira/browse/IGNITE-4684
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov
 Fix For: 2.0


Currently this is highly contented over synchronized section of a single cache 
entry



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: IGNITE-3196 - ready for review

[Vyacheslav Daradur]

Pavel, thanks.

What about PR to master-branch?

2017-02-10 16:55 GMT+03:00 Pavel Tupitsyn :

> Merged to ignite-2.0.
>
> Thank you for the contribution, Vyacheslav!
>
> On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda  wrote:
>
> > + Vladimir Ozerov
> >
> > It would be better if Vladimir Ozerov does the final review considering
> > all the changes in .NET, C++ and Java.
> >
> > Vladimir, could you do that?
> >
> > —
> > Denis
> >
> > > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur 
> > wrote:
> > >
> > > +Denis
> > >
> > > >>Ok, so we agree on .NET and C++ parts, only Java part is to be
> > reviewed.
> > > >>Denis, can you have a look?
> > >
> > > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn   > ptupit...@apache.org>>:
> > > Ok, so we agree on .NET and C++ parts, only Java part is to be
> reviewed.
> > >
> > > Denis, can you have a look?
> > >
> > > Pavel
> > >
> > > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego  > > wrote:
> > >
> > > > Looks good to me.
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> > daradu...@gmail.com >
> > > > wrote:
> > > >
> > > > > Ok, thanks for explanations.
> > > > >
> > > > > What about this task?
> > > > >
> > > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego  > >:
> > > > >
> > > > >> But that's Ok. Since we use int8_t for bytes in C++ as well I
> guess
> > > > >> your -0x80 may have more sense than 0x80.
> > > > >>
> > > > >> Best Regards,
> > > > >> Igor
> > > > >>
> > > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego  > >
> > > > wrote:
> > > > >>
> > > > >>> I was just curious.
> > > > >>>
> > > > >>> In C++ both constants 0x80 and -0x80 are of type 'int' and have
> the
> > > > same
> > > > >>> lower byte, so they give the same result. Though first number is
> > > > actually
> > > > >>> 0x0080 when the second one is 0xFF80.
> > > > >>>
> > > > >>> So it's just made a minus sign look a little redundant and
> > pointless to
> > > > >>> me
> > > > >>> in C++ code.
> > > > >>>
> > > > >>> Best Regards,
> > > > >>> Igor
> > > > >>>
> > > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > > daradu...@gmail.com 
> > > > >>> > wrote:
> > > > >>>
> > > >  Byte.MIN_VALUE = -128 = -0x80
> > > >  Byte.MAX_VALUE = 127 = 0x7F
> > > > 
> > > >  It is just more evident for me.
> > > > 
> > > >  Maybe, I just have the Java programming style.
> > > > 
> > > >  In Java:
> > > >  byte a = 100 | -0x80;  // compiled
> > > >  byte b = 100 | 0x80;  // doesn't compile, explicit type casting
> is
> > > >  neccessary (byte)(100 | 0x80)
> > > >  System.out.println(a | -0x80); // -28
> > > >  System.out.println(a | 0x80); // 228 - cast to int
> > > > 
> > > >  Is it bad style?
> > > > 
> > > >  2017-02-06 20:04 GMT+03:00 Igor Sapego  > >:
> > > > 
> > > > > Vyacheslav,
> > > > >
> > > > > Overall looks good. But why do you use -0x80 instead of 0x80?
> > > > >
> > > > > Best Regards,
> > > > > Igor
> > > > >
> > > > > On Mon, Feb 6, 2017 at 5:36 PM, Vyacheslav Daradur <
> > > > > daradu...@gmail.com > wrote:
> > > > >
> > > > >> Igor,
> > > > >>
> > > > >> I didn't change the CPP code before approval approach.
> > > > >> I shall write directly, sorry.
> > > > >>
> > > > >> But I made CPP changes already.
> > > > >>
> > > > >> > TestEscConvertFunctionFloat
> > > > >> > TestEscConvertFunctionDouble.
> > > > >> These tests were passed
> > > > >>  > http://ci.ignite.apache.org/viewQueued.html?itemId=445824>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> 2017-02-06 13:20 GMT+03:00 Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > >>
> > > > >>> .NET changes look good to me.
> > > > >>>
> > > > >>> Pavel
> > > > >>>
> > > > >>> On Mon, Feb 6, 2017 at 1:10 PM, Igor Sapego <
> > isap...@gridgain.com >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Vyacheslav, I can see two ODBC tests fail in C++ test suits
> > that
> > > > >>> should
> > > > >>> > not:
> > > > >>> > - TestEscConvertFunctionFloat
> > > > >>> >  tab
> > 
> > > > >>> =buildResultsDiv=IgniteTests_IgnitePlatformCppLi
> > > > >>> nux#testNameId-9178617718508801660>
> > > > >>> > - TestEscConvertFunctionDouble
> > > > >>> > 

[jira] [Created] (IGNITE-4683) Need to avoid extra-copy to byte array when marhsalling to cache object

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4683:
-

 Summary: Need to avoid extra-copy to byte array when marhsalling 
to cache object
 Key: IGNITE-4683
 URL: https://issues.apache.org/jira/browse/IGNITE-4683
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Yakov Zhdanov
Assignee: Yakov Zhdanov
 Fix For: 2.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4682) Need to finish transition to thread-per-partition model

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4682:
-

 Summary: Need to finish transition to thread-per-partition model
 Key: IGNITE-4682
 URL: https://issues.apache.org/jira/browse/IGNITE-4682
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Yakov Zhdanov
Assignee: Yakov Zhdanov
 Fix For: 2.0


Need to create sub-tasks with proper description
-atomic cache is almost done
-tx cache - need to start working
-rebalancing seems to be pretty easy to move to this approach
-then we can remove deleted entries buffer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4681) Apply new future adapter

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4681:
-

 Summary: Apply new future adapter
 Key: IGNITE-4681
 URL: https://issues.apache.org/jira/browse/IGNITE-4681
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov
Assignee: Yakov Zhdanov


Attached is reference future adapter implementation. It is proven to consume 
less memory and it does not require explicit locking on listen(). We need to 
apply it.

Known threats:
# if future is completed normally, but with Throwable as result, get() throws 
exception. This can be fixed with internal wrapper class
# listener notification order changes - this is known to cause problems 
org.apache.ignite.internal.processors.rest.ClientMemcachedProtocolSelfTest#testGetBulk
 which is minor but still.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: IGNITE-3196 - ready for review

[Pavel Tupitsyn]

Merged to ignite-2.0.

Thank you for the contribution, Vyacheslav!

On Tue, Feb 7, 2017 at 10:15 PM, Denis Magda  wrote:

> + Vladimir Ozerov
>
> It would be better if Vladimir Ozerov does the final review considering
> all the changes in .NET, C++ and Java.
>
> Vladimir, could you do that?
>
> —
> Denis
>
> > On Feb 7, 2017, at 5:04 AM, Vyacheslav Daradur 
> wrote:
> >
> > +Denis
> >
> > >>Ok, so we agree on .NET and C++ parts, only Java part is to be
> reviewed.
> > >>Denis, can you have a look?
> >
> > 2017-02-07 15:27 GMT+03:00 Pavel Tupitsyn >:
> > Ok, so we agree on .NET and C++ parts, only Java part is to be reviewed.
> >
> > Denis, can you have a look?
> >
> > Pavel
> >
> > On Tue, Feb 7, 2017 at 3:23 PM, Igor Sapego  > wrote:
> >
> > > Looks good to me.
> > >
> > > Best Regards,
> > > Igor
> > >
> > > On Tue, Feb 7, 2017 at 2:55 PM, Vyacheslav Daradur <
> daradu...@gmail.com >
> > > wrote:
> > >
> > > > Ok, thanks for explanations.
> > > >
> > > > What about this task?
> > > >
> > > > 2017-02-07 13:57 GMT+03:00 Igor Sapego  >:
> > > >
> > > >> But that's Ok. Since we use int8_t for bytes in C++ as well I guess
> > > >> your -0x80 may have more sense than 0x80.
> > > >>
> > > >> Best Regards,
> > > >> Igor
> > > >>
> > > >> On Tue, Feb 7, 2017 at 1:54 PM, Igor Sapego  >
> > > wrote:
> > > >>
> > > >>> I was just curious.
> > > >>>
> > > >>> In C++ both constants 0x80 and -0x80 are of type 'int' and have the
> > > same
> > > >>> lower byte, so they give the same result. Though first number is
> > > actually
> > > >>> 0x0080 when the second one is 0xFF80.
> > > >>>
> > > >>> So it's just made a minus sign look a little redundant and
> pointless to
> > > >>> me
> > > >>> in C++ code.
> > > >>>
> > > >>> Best Regards,
> > > >>> Igor
> > > >>>
> > > >>> On Mon, Feb 6, 2017 at 10:15 PM, Vyacheslav Daradur <
> > > daradu...@gmail.com 
> > > >>> > wrote:
> > > >>>
> > >  Byte.MIN_VALUE = -128 = -0x80
> > >  Byte.MAX_VALUE = 127 = 0x7F
> > > 
> > >  It is just more evident for me.
> > > 
> > >  Maybe, I just have the Java programming style.
> > > 
> > >  In Java:
> > >  byte a = 100 | -0x80;  // compiled
> > >  byte b = 100 | 0x80;  // doesn't compile, explicit type casting is
> > >  neccessary (byte)(100 | 0x80)
> > >  System.out.println(a | -0x80); // -28
> > >  System.out.println(a | 0x80); // 228 - cast to int
> > > 
> > >  Is it bad style?
> > > 
> > >  2017-02-06 20:04 GMT+03:00 Igor Sapego  >:
> > > 
> > > > Vyacheslav,
> > > >
> > > > Overall looks good. But why do you use -0x80 instead of 0x80?
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > > On Mon, Feb 6, 2017 at 5:36 PM, Vyacheslav Daradur <
> > > > daradu...@gmail.com > wrote:
> > > >
> > > >> Igor,
> > > >>
> > > >> I didn't change the CPP code before approval approach.
> > > >> I shall write directly, sorry.
> > > >>
> > > >> But I made CPP changes already.
> > > >>
> > > >> > TestEscConvertFunctionFloat
> > > >> > TestEscConvertFunctionDouble.
> > > >> These tests were passed
> > > >>  http://ci.ignite.apache.org/viewQueued.html?itemId=445824>>
> > > >>
> > > >>
> > > >>
> > > >> 2017-02-06 13:20 GMT+03:00 Pavel Tupitsyn  >:
> > > >>
> > > >>> .NET changes look good to me.
> > > >>>
> > > >>> Pavel
> > > >>>
> > > >>> On Mon, Feb 6, 2017 at 1:10 PM, Igor Sapego <
> isap...@gridgain.com >
> > > >>> wrote:
> > > >>>
> > > >>> > Vyacheslav, I can see two ODBC tests fail in C++ test suits
> that
> > > >>> should
> > > >>> > not:
> > > >>> > - TestEscConvertFunctionFloat
> > > >>> >  
> > > >>> =buildResultsDiv=IgniteTests_IgnitePlatformCppLi
> > > >>> nux#testNameId-9178617718508801660>
> > > >>> > - TestEscConvertFunctionDouble
> > > >>> >  
> > > >>> =buildResultsDiv=IgniteTests_IgnitePlatformCppLi
> > > >>> nux#testNameId5432107083822590090>
> > > >>> > .
> > > >>> >
> > > >>> > I believe, this is because I can't see any changes in C++
> Decimal
> > > >>> > marshaling code.
> > > >>> > Please, pay attention to file 

Re: Apache Ignite 1.9

[Sergey Kalashnikov]

Anton,

Here is the status for IGNITE-4523.
The work is completed, tests are OK.
Most of the changes were made by Alexey and were reviewed.
I only authored few last commits for which I would appreciate a review.
The PR is https://github.com/apache/ignite/pull/1508

Thank you
--
Sergey



2017-02-09 23:58 GMT+03:00 Denis Magda :

> Anton,
>
> You can expect Kubernetes support from me in time.
>
> —
> Denis
>
> > On Feb 9, 2017, at 1:52 AM, Anton Vinogradov  wrote:
> >
> > Igniters,
> >
> > We're on a half way to Ignite 1.9.
> > Development phase will be finished 17th of February.
> >
> > Here  Apache+Ignite+1.9> is
> > a status page for upcoming release.
> >
> > According to this page we have some Issues we *especially *expect in 1.9:
> >
> > IGNITE-4106 SQL: parallelize sql queries over cache local partitions
> Andrew
> > Mashenkov
> > IGNITE-3860 Improve distributed SQL support. Sergi Vladykin
> > IGNITE-4539 RocketMQ data streamer Roman Shtykh
> > IGNITE-4212 Ignite Benchmarking Simplification and Automation Oleg
> Ostanin
> > IGNITE-4523 Allow distributed SQL query execution over explicit set of
> > partitions Alexei Scherbakov
> > IGNITE-3013 Support sorted merge index for SQL Sergi Vladykin
> > IGNITE-4169 Data streamer mode for DML Alexander Paschenko
> > IGNITE-4670 CPP: Implement LoadCache method Unassigned (Igor Sapego,
> seems
> > you're the best person)
> > IGNITE-4159 Cloud Native Deployment of Apache Ignite using Kubernetes
> Denis
> > Magda
> > IGNITE-4526 Add Spark Shared RDD examples Manish Mishra
> > IGNITE-4674 Ignite Web Console: Support DML queries Alexey Kuznetsov
> > IGNITE-1794 Ignite should support Hibernate 5 Mykola Pereyma
> >
> > Guys, please confirm that you will be able to complete listed tasks by
> 17th
> > of February.
> >
> >
> > On Fri, Feb 3, 2017 at 5:32 AM, Denis Magda  wrote:
> >
> >> Excellent news Roman! Please keep us posted.
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 2, 2017, at 6:26 PM, Roman Shtykh 
> >> wrote:
> >>>
> >>> Denis,
> >>> RocketMQ data streamer implementation is almost ready. But there are no
> >> binaries in maven repositories now (it will be their 1st release under
> >> Apache license). The release is very close to starting voting for, so
> >> hopefully it can be done in the middle of February.
> >>> Roman
> >>>
> >>>
> >>>   On Wednesday, February 1, 2017 4:13 AM, Denis Magda <
> >> dma...@apache.org> wrote:
> >>>
> >>>
> >>> Igniters,
> >>> Looks like 2.0 will be slightly delayed since the community requires
> >> more time to make the new page memory architecture faster. At the same
> time
> >> we have a plenty of bug fixes and performance improvements that were
> merged
> >> or close to be merged into the master and ready to be released.
> >>> I propose to release all the changes we have in 1.9 close to the end of
> >> February.
> >>> According to the dev list discussions, we will be ready to release
> >> performance optimizations done by Yakov and Sam close to the end of 2016
> >> and many SQL improvements driven by multiple folks.
> >>> In addition, this is what we might expect in 1.9:- SQL execution over
> >> specific partitions (https://issues.apache.org/jira/browse/IGNITE-4523
> ).
> >> Alexey Scherbakov.- Spark version upgrade (https://issues.apache.org/
> >> jira/browse/IGNITE-3710). Anton V.- Spark RDD Examples (
> >> https://issues.apache.org/jira/browse/IGNITE-4526). Manish Mishra.-
> >> Rocket MQ data streamer (https://issues.apache.org/
> jira/browse/IGNITE-4539).
> >> Roman Shtykh.- Hibernate 5 support (https://issues.apache.org/
> >> jira/browse/IGNITE-1794). Mykola Pereyma.- .NET DML (
> >> https://issues.apache.org/jira/browse/IGNITE-4045). Pavel Tupitsyn.-
> .NET
> >> Transaction Scope API (https://issues.apache.org/
> jira/browse/IGNITE-3430).
> >> Pavel Tupitsyn.- C++ Continuous queries (https://issues.apache.org/
> >> jira/browse/IGNITE-1443). Igor Sapego.- Kubernetes Support (
> >> https://issues.apache.org/jira/browse/IGNITE-4159). Denis Magda.- JDBC
> >> batch mode for DML (https://issues.apache.org/jira/browse/IGNITE-4269).
> >> Alexander Paschenko.- Benchmarks automation (https://issues.apache.org/
> >> jira/browse/IGNITE-4212). Oleg Ostanin.
> >>> Guys, please confirm that you will be able to complete listed tasks by
> >> the middle or by 20th of February. Anything else that is not in the
> list?
> >>> BTW, is there anyone who can take a role of the release manager for
> 1.9?
> >>> —Denis
> >>>
> >>
> >>
>
>

[jira] [Created] (IGNITE-4680) Properly split batch atomic cache operations between stripes (putAll, removeAll, etc)

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4680:
-

 Summary: Properly split batch atomic cache operations between 
stripes (putAll, removeAll, etc)
 Key: IGNITE-4680
 URL: https://issues.apache.org/jira/browse/IGNITE-4680
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Yakov Zhdanov
Assignee: Yakov Zhdanov
 Fix For: 2.0


Need to revisit keys/requests mapping procedure and map the update directly to 
stripe on remote node.
Here are some points:
# Abovementioned will require adding stripes count attribute to node's 
attribute list
# Need to make sure we take all benefits of biased locking and stripes never 
get mutually blocked
# locking all entries before processing request can be removed since it does 
not make too much sense to atomic cache



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Contributing to Ignite.NET

[Guru Stron]

Hi, Pavel

Great, thank you.

Was thinking about taking IGNITE-4636
 too)

On 10 February 2017 at 15:44, Pavel Tupitsyn  wrote:

> Sergey, looks good! Merged to master branch.
>
> Thank you for the contribution.
>
> Would be great if you pick up
> https://issues.apache.org/jira/browse/IGNITE-4636
> so that we'll have comprehensive collection support.
>
> Pavel
>
> On Fri, Feb 10, 2017 at 2:41 PM, Guru Stron 
> wrote:
>
> > Hi Pavel,
> >
> > I've fixed the issues you've pointed out and committed changes. The
> > inspections build
> >  > tab=queuedBuildOverviewTab>
> > is
> > passing, also I've queued the platform build
> >  > tab=queuedBuildOverviewTab>
> > .
> >
> > On 7 February 2017 at 19:25, Pavel Tupitsyn 
> wrote:
> >
> > > Sergey, good job! I've replied in the ticket.
> > >
> > > Pavel
> > >
> > > On Tue, Feb 7, 2017 at 6:20 PM, Guru Stron 
> > > wrote:
> > >
> > > > Can you please review the changes in pull request #1502
> > > > .
> > > >
> > > > Thank you.
> > > >
> > > > On 7 February 2017 at 18:18, Guru Stron 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've finished developing the changes for the .NET: Support
> > > > > "ICollection.Contains" in LINQ
> > > > >  ticket. Local
> > LINQ
> > > > > tests are passing, also I've triggered build on TC
> > > > >  > > > tab=queuedBuildOverviewTab>,
> > > > > but it is expected to run in 6 hours, so I had not waited it to
> > finish.
> > > > >
> > > > > On 16 January 2017 at 14:54, Pavel Tupitsyn 
> > > > wrote:
> > > > >
> > > > >> Sergey, sure, go ahead and take it!
> > > > >> Please assign to yourself and click "Start Progress".
> > > > >>
> > > > >> Let me know if you have any questions.
> > > > >>
> > > > >> Thanks,
> > > > >> Pavel
> > > > >>
> > > > >> On Mon, Jan 16, 2017 at 2:49 PM, Guru Stron <
> > > gurustronpub...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi igniters,
> > > > >> >
> > > > >> > I want to improve LINQ support in IGNITE.NET.
> > > > >> >
> > > > >> > Is it ok if i take this task: IGNITE-4425 .NET: Support
> > > > >> "Array.Contains" in
> > > > >> > LINQ 
> > > > >> >
> > > > >> > On 9 January 2017 at 15:40, Pavel Tupitsyn <
> ptupit...@apache.org>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Sergey, I've merged your fix for IGNITE-4412.
> > > > >> > >
> > > > >> > > Looking forward to more contributions from you!
> > > > >> > >
> > > > >> > > Pavel
> > > > >> > >
> > > > >> > > On Thu, Jan 5, 2017 at 8:15 PM, Pavel Tupitsyn <
> > > > ptupit...@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Sergey,
> > > > >> > > >
> > > > >> > > > I've added you to project contributors in Jira.
> > > > >> > > >
> > > > >> > > > I can suggest the following .NET tickets to get familiar
> with
> > > the
> > > > >> > > project:
> > > > >> > > >
> > > > >> > > > IGNITE-4412 .NET: NLogLoggerTest is not thread safe
> > > > >> > > > 
> > > > >> > > > IGNITE-3170 .NET: Add user-friendly ToString overrides for
> > > public
> > > > >> types
> > > > >> > > > 
> > > > >> > > > IGNITE-1508 .NET: Create common ToString() method
> > > > >> > > > 
> > > > >> > > >
> > > > >> > > > Let me know if you have any questions.
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > >
> > > > >> > > > Pavel.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Jan 5, 2017 at 8:10 PM, Guru Stron <
> > > > >> gurustronpub...@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > >> Hi,
> > > > >> > > >>
> > > > >> > > >> My name is Sergey, i would like to contribute to
> Ignite.NET.
> > > > >> > > >> Can you please suggest a ticket for me to start. My Jira
> > > username
> > > > >> is
> > > > >> > > >> GuruStron.
> > > > >> > > >>
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Contributing to Ignite.NET

[Pavel Tupitsyn]

Sergey, looks good! Merged to master branch.

Thank you for the contribution.

Would be great if you pick up
https://issues.apache.org/jira/browse/IGNITE-4636
so that we'll have comprehensive collection support.

Pavel

On Fri, Feb 10, 2017 at 2:41 PM, Guru Stron 
wrote:

> Hi Pavel,
>
> I've fixed the issues you've pointed out and committed changes. The
> inspections build
>  tab=queuedBuildOverviewTab>
> is
> passing, also I've queued the platform build
>  tab=queuedBuildOverviewTab>
> .
>
> On 7 February 2017 at 19:25, Pavel Tupitsyn  wrote:
>
> > Sergey, good job! I've replied in the ticket.
> >
> > Pavel
> >
> > On Tue, Feb 7, 2017 at 6:20 PM, Guru Stron 
> > wrote:
> >
> > > Can you please review the changes in pull request #1502
> > > .
> > >
> > > Thank you.
> > >
> > > On 7 February 2017 at 18:18, Guru Stron 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've finished developing the changes for the .NET: Support
> > > > "ICollection.Contains" in LINQ
> > > >  ticket. Local
> LINQ
> > > > tests are passing, also I've triggered build on TC
> > > >  > > tab=queuedBuildOverviewTab>,
> > > > but it is expected to run in 6 hours, so I had not waited it to
> finish.
> > > >
> > > > On 16 January 2017 at 14:54, Pavel Tupitsyn 
> > > wrote:
> > > >
> > > >> Sergey, sure, go ahead and take it!
> > > >> Please assign to yourself and click "Start Progress".
> > > >>
> > > >> Let me know if you have any questions.
> > > >>
> > > >> Thanks,
> > > >> Pavel
> > > >>
> > > >> On Mon, Jan 16, 2017 at 2:49 PM, Guru Stron <
> > gurustronpub...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi igniters,
> > > >> >
> > > >> > I want to improve LINQ support in IGNITE.NET.
> > > >> >
> > > >> > Is it ok if i take this task: IGNITE-4425 .NET: Support
> > > >> "Array.Contains" in
> > > >> > LINQ 
> > > >> >
> > > >> > On 9 January 2017 at 15:40, Pavel Tupitsyn 
> > > >> wrote:
> > > >> >
> > > >> > > Sergey, I've merged your fix for IGNITE-4412.
> > > >> > >
> > > >> > > Looking forward to more contributions from you!
> > > >> > >
> > > >> > > Pavel
> > > >> > >
> > > >> > > On Thu, Jan 5, 2017 at 8:15 PM, Pavel Tupitsyn <
> > > ptupit...@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Sergey,
> > > >> > > >
> > > >> > > > I've added you to project contributors in Jira.
> > > >> > > >
> > > >> > > > I can suggest the following .NET tickets to get familiar with
> > the
> > > >> > > project:
> > > >> > > >
> > > >> > > > IGNITE-4412 .NET: NLogLoggerTest is not thread safe
> > > >> > > > 
> > > >> > > > IGNITE-3170 .NET: Add user-friendly ToString overrides for
> > public
> > > >> types
> > > >> > > > 
> > > >> > > > IGNITE-1508 .NET: Create common ToString() method
> > > >> > > > 
> > > >> > > >
> > > >> > > > Let me know if you have any questions.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > >
> > > >> > > > Pavel.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Jan 5, 2017 at 8:10 PM, Guru Stron <
> > > >> gurustronpub...@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Hi,
> > > >> > > >>
> > > >> > > >> My name is Sergey, i would like to contribute to Ignite.NET.
> > > >> > > >> Can you please suggest a ticket for me to start. My Jira
> > username
> > > >> is
> > > >> > > >> GuruStron.
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

[GitHub] ignite pull request #1502: IGNITE-4425 .NET: Support "ICollection.Contains" ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1502


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1496: IGNITE-3994

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1518: IGNITE-2703 .NET: Dynamic type registration

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/1518

IGNITE-2703 .NET: Dynamic type registration



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2703

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1518


commit 792506a63572b0b50e47b11c3a6e54065768ac98
Author: Pavel Tupitsyn 
Date:   2017-02-10T11:56:55Z

IGNITE-2703 .NET: Dynamic type registration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IGNITE-4158 - ready for review

[Vyacheslav Daradur]

Hello. I did it.

Please, review.

https://issues.apache.org/jira/browse/IGNITE-4158 - Implement Time type for
binary protocol.

latest ci.tests 

Re: Contributing to Ignite.NET

[Guru Stron]

Hi Pavel,

I've fixed the issues you've pointed out and committed changes. The
inspections build

is
passing, also I've queued the platform build

.

On 7 February 2017 at 19:25, Pavel Tupitsyn  wrote:

> Sergey, good job! I've replied in the ticket.
>
> Pavel
>
> On Tue, Feb 7, 2017 at 6:20 PM, Guru Stron 
> wrote:
>
> > Can you please review the changes in pull request #1502
> > .
> >
> > Thank you.
> >
> > On 7 February 2017 at 18:18, Guru Stron 
> wrote:
> >
> > > Hi,
> > >
> > > I've finished developing the changes for the .NET: Support
> > > "ICollection.Contains" in LINQ
> > >  ticket. Local LINQ
> > > tests are passing, also I've triggered build on TC
> > >  > tab=queuedBuildOverviewTab>,
> > > but it is expected to run in 6 hours, so I had not waited it to finish.
> > >
> > > On 16 January 2017 at 14:54, Pavel Tupitsyn 
> > wrote:
> > >
> > >> Sergey, sure, go ahead and take it!
> > >> Please assign to yourself and click "Start Progress".
> > >>
> > >> Let me know if you have any questions.
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >> On Mon, Jan 16, 2017 at 2:49 PM, Guru Stron <
> gurustronpub...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi igniters,
> > >> >
> > >> > I want to improve LINQ support in IGNITE.NET.
> > >> >
> > >> > Is it ok if i take this task: IGNITE-4425 .NET: Support
> > >> "Array.Contains" in
> > >> > LINQ 
> > >> >
> > >> > On 9 January 2017 at 15:40, Pavel Tupitsyn 
> > >> wrote:
> > >> >
> > >> > > Sergey, I've merged your fix for IGNITE-4412.
> > >> > >
> > >> > > Looking forward to more contributions from you!
> > >> > >
> > >> > > Pavel
> > >> > >
> > >> > > On Thu, Jan 5, 2017 at 8:15 PM, Pavel Tupitsyn <
> > ptupit...@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Sergey,
> > >> > > >
> > >> > > > I've added you to project contributors in Jira.
> > >> > > >
> > >> > > > I can suggest the following .NET tickets to get familiar with
> the
> > >> > > project:
> > >> > > >
> > >> > > > IGNITE-4412 .NET: NLogLoggerTest is not thread safe
> > >> > > > 
> > >> > > > IGNITE-3170 .NET: Add user-friendly ToString overrides for
> public
> > >> types
> > >> > > > 
> > >> > > > IGNITE-1508 .NET: Create common ToString() method
> > >> > > > 
> > >> > > >
> > >> > > > Let me know if you have any questions.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Pavel.
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Jan 5, 2017 at 8:10 PM, Guru Stron <
> > >> gurustronpub...@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Hi,
> > >> > > >>
> > >> > > >> My name is Sergey, i would like to contribute to Ignite.NET.
> > >> > > >> Can you please suggest a ticket for me to start. My Jira
> username
> > >> is
> > >> > > >> GuruStron.
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: DML data streaming

[Vladimir Ozerov]

I propose to ship streaming with INSERT support only for now. This is
enough for multitude cases and will add value to Ignite 1.9 immediately. We
can think about correct streaming UPDATE/DELETE architecture separately .It
is much more difficult thing, we cannot support it in a clean way right now
due to multiple "_key" and "_val" usages over the code base.

On Fri, Feb 10, 2017 at 11:55 AM, Alexander Paschenko <
alexander.a.pasche...@gmail.com> wrote:

> And to avoid further confusion: UPDATE and DELETE are simply
> impossible in streaming mode when the key is not completely defined as
> long as data streamer operates with key-value pairs and not just
> tuples of named values. That's why we can't do DELETE from Person
> WHERE id1 = 5 from prev example with streamer - the Key { id1 = 5, id2
> = 0 } that would be constructed from such query is just one key and is
> handled by streamer as such while semantically that query is not about
> ONE key but about ALL keys where id1 = 5.
>
> - Alex
>
> 2017-02-10 11:49 GMT+03:00 Alexander Paschenko
> :
> > Dima,
> >>
> >> There are several ways to handle it. I would check how other databases
> >> handle it, maybe we can borrow something. To the least, we should log
> such
> >> errors in the log for now.
> >>
> >
> > Logging errors would mean introducing some kind of stream receiver to
> > do that and thus that would be really the same performance penalty for
> > the successful operations. I think we should go with that optional
> > flag for semantics after all.
> >
> >> You don't have to use _key. Primary key is usually a field in the
> class, so
> >> you can use a normal column name. In any case, we should remove any
> usage
> >> of _key before 2.0 is released.
> >>
> >> Again, if user does not have to specify _key on INSERT, then it is very
> >> unclear to me, why user would need to specify _key for UPDATE or DELETE.
> >> Something smells here. Can you please provide an example?
> >>
> >
> > UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
> > optimized cases - i.e. those where _key (and possibly _val) are
> > explicitly specified by the user thus allowing us to map UPDATE and
> > DELETE directly to cache's replace and remove operations without
> > messing with entry processors and doing map-reduce SELECT by given
> > criteria.
> >
> > Say, we have Person { firstName, secondName } with key class Key { id1,
> id2 }
> >
> > If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
> > there's no need to do any SELECT - we can just call IgniteCache.remove
> > on that key.
> >
> > But if I say DELETE from Person WHERE id1 = 5 then there's no way to
> > avoid MR - we have to find all keys that interest us first by doing
> > SELECT as long as we know only partly about what keys the user wants
> > to be affected.
> >
> > It works in the same way for UPDATE. And I hope that it's clear how
> > it's different from INSERT - there's no MR by definition (we don't
> > allow INSERT FROM SELECT in streaming mode).
> >
> > AGAIN: this all is said only about streaming mode; non streaming mode
> > does those optimizations too, but it also allows complex conditions,
> > while streaming mode does not allow them to keep things fast and avoid
> > MR.
> >
> > That's the reason why I suggest that we drop UPDATE and DELETE from
> > DML streaming as they mean messing with those soon-hidden columns.
> >
> > Still we could optimize stuff like DELETE from Person WHERE id1 = 5
> > AND id2 = 6 - query involves ALL fields of key AND compares only for
> > equality AND has no complex expressions - we can construct key
> > unambiguously and still call remove directly.
> >
> > But to me it does not sound like a really great reason to leave UPDATE
> > and DELETE in DML - the users will have to write some specific queries
> > to use that while all other stuff will just be declined in that mode.
> > And, as I said before, UPDATE and DELETE don't probably perfectly fit
> > with primary data streamer use cases - after all, modifying existing
> > stuff is not what data streamer is about.
> >
> > And regarding hiding columns: it's unclear how things will look like
> > for caches like  when we remove _key and _val as long as
> > tables for such cases currently have nothing but those two columns.
> >
> > - Alex
> >
> >>> 8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
> >>> dsetrak...@apache.org> написал:
> >>>
> >>> > Alexander,
> >>> >
> >>> > Are you suggesting that currently to execute a simple INSERT for 1
> row we
> >>> > invoke a data streamer on Ignite API? How about an update by a
> primary
> >>> key?
> >>> > Why not execute a simple cache put in either case?
> >>> >
> >>> > I think we had a separate thread where we agreed that the streamer
> should
> >>> > only be turned on if a certain flag on a JDBC connection is set, no?
> >>> >
> >>> > D.
> >>> >
> >>> > On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
> >>> 

Re: IGNITE-4212 (Ignite Benchmarking Simplification and Automation)

[Oleg Ostanin]

I'm sorry, there wasn't any problem with the build, it was local problem on
my machine.

On Fri, Feb 10, 2017 at 11:54 AM, Oleg Ostanin 
wrote:

> Hi.
>
> I just discovered that some functionality does not work correctly. I will
> present fixed version later.
>
> On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
> wrote:
>
>> https://drive.google.com/open?id=0B9teTzJ9iIvHckl6eVQ3N0R4a3c
>>
>> On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
>> wrote:
>>
>>> I've fixed README.txt and created new DEVNOTES.txt. Also renamed sample
>>> benchmark configuration file to benchmark-sample.properties.
>>> Please look for the results:
>>>
>>>
>>> On Tue, Jan 31, 2017 at 9:18 PM, Denis Magda  wrote:
>>>
 Oleg,

 Thanks for the clarification.

 My opinion is that we should leave ‘modules/yardstick/README.txt’ and
 ‘modules/yardstick/DEVNOTES.txt’ either unchanged or have only those
 instructions there that explain how to build and run benchmarks from
 ‘modules/yardstick’. This existing files can refer to the sources and
 compiled benchmarks that are in Ignite binary releases but this should be a
 couple of statements, no more.

 As for the instructions related to the sources and binaries added to
 Ignite binaries, preferably they need to be added to auto-generated
 README.txt. *Anton*, is it feasible to do?

 Finally, when apply the reviews notes please build and share Ignite
 binaries with your patch. Want to see the new project structure and final
 version of the instructions. Presently I can’t merge your changes due to
 some conflicts.

 *Anton*, please review modifications in the build procedures.

 —
 Denis

 > On Jan 31, 2017, at 3:12 AM, Oleg Ostanin 
 wrote:
 >
 > Hi Denis,
 >
 > Yes, we have included Ignite Yardstick source files with its pom.xml
 in
 > Ignite binary release. "Building from standalone sources" is the
 building
 > instruction for these source files. "Building from Ignite Sources" is
 the
 > instruction for building Ignite Yardstick from `modules/yardstick` in
 > Ignite source files.
 >
 > On Tue, Jan 31, 2017 at 3:31 AM, Denis Magda 
 wrote:
 >
 >> Hi Oleg,
 >>
 >> Great progress, thanks for keep driving this!
 >>
 >> I’ve left some minor notes in GitHub’s pull-request. I have the
 following
 >> questions aside:
 >>
 >> - What is the difference between "Building from standalone sources"
 and
 >> "Building from Ignite Sources"? In my understanding, a user downloads
 >> Apache Ignite release that has all the sources locally.
 >>
 >> - I do remember we planned to add the benchmarks sources in a form
 of a
 >> ready to be used project with its own pom.xml (similar to examples).
 Did
 >> you put this task off?
 >>
 >> —
 >> Denis
 >>
 >>> On Jan 27, 2017, at 2:13 AM, Oleg Ostanin 
 wrote:
 >>>
 >>> Hi!
 >>>
 >>> I've changed the README.txt and DEVNOTES.txt files. Also added a
 simple
 >>> config file for quick and easy start. Please take a look at them
 and tell
 >>> me what you think.
 >>>
 >>> https://github.com/apache/ignite/pull/1471
 >>>
 >>> On Wed, Dec 28, 2016 at 8:59 AM, Ilya Suntsov <
 isunt...@gridgain.com>
 >> wrote:
 >>>
  Denis,
 
  I think we can remove all configs except:
 
  benchmark-multicast.properties
 
  benchmark.properties
 
  ignite-base-config.xml
 
  ignite-localhost-config.xml
 
  ignite-multicast-config.xml
 
  2016-12-28 2:49 GMT+03:00 Denis Magda :
 
 > I would have only those configs that are useful. Ilya Suntsov,
 basing
 >> on
 > your experience, please suggest which configs makes sense to
 include
 >> into
 > every Ignite release.
 >
 > Oleg, also please note that community decided to include not only
 the
 > benchmarking binaries but the sources as well into every Apache
 Ignite
 > release. I’ve update the ticket before. Hope you followed the
 >> discussion
  ;)
 > https://issues.apache.org/jira/browse/IGNITE-4212?
 > focusedCommentId=15765151=com.atlassian.jira.
 > plugin.system.issuetabpanels:comment-tabpanel#comment-15765151
 >
 > —
 > Denis
 >
 >> On Dec 27, 2016, at 5:35 AM, Oleg Ostanin 
  wrote:
 >>
 >> I mean removing those configs from binary assembly, not from
  repository.
 >>
 >> On Tue, Dec 27, 2016 at 4:28 PM, 

[jira] [Created] (IGNITE-4679) Consistency checks and error tracking during node joining process

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-4679:
---

 Summary: Consistency checks and error tracking during node joining 
process
 Key: IGNITE-4679
 URL: https://issues.apache.org/jira/browse/IGNITE-4679
 Project: Ignite
  Issue Type: Improvement
  Components: general
Reporter: Sergey Chugunov


h2. Notes
Right now new node requesting to join cluster may receive reject because of 
failed security checks.

But it is possible that any *GridComponent* may have wrong configuration or 
internal state that could lead to cluster instability.

It is better to identify these situations as early as possible and fail joining 
process prohibiting poison nodes to join.

h2. Acceptance Criteria
# API for consistency checks is proposed and approved by community.
# BinaryMetadata cache validation is performed by coordinator. In case of any 
conflicts new node joining request is rejected.
# [Optionally] Security checks are migrated to common mechanism.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1516: IGNITE-3429: org.hibernate.cache.spi.CacheKey not...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1516

IGNITE-3429: org.hibernate.cache.spi.CacheKey not properly serialized by 
binary marshaller

Sample comfigurations for CacheKey added. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1516.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1516


commit 630da94351e71b418b7b605eeec66ab67c1b4884
Author: AMRepo 
Date:   2017-02-09T15:21:18Z

Implemented




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: jdbc vs jdbc2 packages

[Yakov Zhdanov]

Val, can you please explain your statement. You suggest users have port
forwarding for dozens of nodes in their topologies so client can initiate
connect to any of them? I don't think this is possible and routing seems to
be the only possible option.

--Yakov

2017-02-10 0:04 GMT+03:00 Valentin Kulichenko :

> Yakov,
>
> 1. Yes, I will file a ticket.
>
> 4. I meant that server can currently initiate connection with client, and
> that's the main problem here. Is there a way to avoid this? Message routing
> you're referring to can also be useful in some cases, but much less
> critical.
>
> -Val
>
> On Wed, Feb 8, 2017 at 9:20 PM, Yakov Zhdanov 
> wrote:
>
> > Val,
> >
> > 1. Our clients should stop require persistent store implementation if
> they
> > do not need it. Can you please file a ticket? I know you fixed some
> places
> > already. As an idea I would keep everything in binary format until we
> > really need it. Will that work?
> > 2. We can try adding the very first step to fetch the configuration and
> > then proceed with normal start.
> > https://issues.apache.org/jira/browse/IGNITE-4675
> > 3. Agree, but user needs to define the closures then. I would think on
> how
> > to put this to a product.
> > 4. This needs to be implemented :) I mean we can communicate to a client
> > through server it is connected to.
> >
> > Thanks!
> > --
> > Yakov Zhdanov, Director R
> > *GridGain Systems*
> > www.gridgain.com
> >
> > 2017-02-09 5:19 GMT+07:00 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Yakov,
> > >
> > > I agree that investing in the legacy client doesn't make sense - it's
> > slow
> > > and outdated. Regarding your points:
> > >
> > > 1. This is just another build step, but the JAR is going to pretty fat
> I
> > > think (it will have to include Spring). Not ideal, but definitely
> better
> > > than what we have now. However, our clients also often require some
> user
> > > classes, like CacheStore implementations. This is also a problem.
> > >
> > > 2. That's a great idea! Actually, I'm not sure why we require to have
> > full
> > > verbose config on client that is consistent with server. Why not fetch
> > the
> > > configuration from cluster during join? Not sure how hard this change
> is,
> > > but it can be a very big usability improvement. And surely, JDBC driver
> > > should be able to config with host:port without config file.
> > >
> > > 3. This can be already achieved with Compute Grid, no? I don't think we
> > > need to add anything here.
> > >
> > > Another issue with clients is that they currently can't work behind NAT
> > > without additional config which is not very trivial (AddressResolver).
> Is
> > > it possible to avoid server->client connections in communication SPI?
> > >
> > > -Val
> > >
> > > On Tue, Feb 7, 2017 at 9:09 PM, Yakov Zhdanov 
> > wrote:
> > >
> > > > "undeprecating" - lol :D
> > > > Consider introducing @Un annotation which negates all annotations on
> > the
> > > > same level and below.
> > > >
> > > > I would probably agree with Val and Vova, but adding features to
> > > > thin-client seems questionable to me.
> > > >
> > > > Is these possible:
> > > > 1. avoid dependencies on client machine and require ignite-jdbc.jar
> > only
> > > > (e.g. gathering dependencies inside the jar).
> > > > 2. make it possible to provide just address and port to send join
> > request
> > > > to without providing the entire IgniteConfiguration. Client node
> sends
> > > join
> > > > request to the cluster with flag that this is jdbc-driver connection
> > and
> > > > server-side topology omits configuration validation and forces client
> > to
> > > > set some properties if this is necessary (e.g. CommunicationSpi
> > > > implementation class and settings)
> > > > 3. add possibility to offload complex reduce processing to server.
> > Which
> > > > may be very helpful for main client-server use cases when clients
> being
> > > run
> > > > on much weaker machines.
> > > >
> > > > ?
> > > >
> > > > --Yakov
> > > >
> > > > 2017-02-07 14:30 GMT+07:00 Vladimir Ozerov :
> > > >
> > > > > Big +1 to Val, not only about JDBC, but about our overall approach
> to
> > > > > clients. Starting a node with "client=true" is:
> > > > > + Very reach feature set, which is cool
> > > > > - Tons of dependencies
> > > > > - Tons of threads
> > > > >
> > > > > It would be very cool if we have a true thin client with small
> single
> > > > JAR.
> > > > > It should have:
> > > > > - Failover
> > > > > - Load-balance
> > > > > - Optional server "stickyness"
> > > > >
> > > > > Once all these things are in place we will be able to provide the
> > same
> > > > API
> > > > > as in current client, but with predictable behavior and memory
> > > footprint.
> > > > > For instance our current client is not well-suited for running
> > > map-reduce
> > > > > (compute or SQL) because it moves large 

Re: Rethink native SQL API in Apache Ignite 2.0

[Sergi Vladykin]

I agree with Vladimir that we have to improve our API with respect to the
latest changes in our SQL capabilities.

As for the given API proposal, it is a bit harsh right now, but it is ok
for the first draft, overall direction looks good.

Also IMO we should not drop the existing API, thus adding a new one is not
urgent for 2.0, because adding new API is not a breaking change.

Sergi

2017-02-10 4:55 GMT+03:00 Dmitriy Setrakyan :

> Vladimir,
>
> Will these changes be backward compatible?
>
> Sergi, given your SQL expertise, can you please comment here as well?
>
> D.
>
> On Tue, Feb 7, 2017 at 3:34 AM, Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > Our SQL engine becomes more and more sophisticated. Initially we had only
> > SELECTs, now we have DML, in AI 2.x we will have DDL.
> >
> > This is very cool, but it seems that we completely forgot about extending
> > our native SQL API (IgniteCache.query, SqlQuery, SqlFieldsQuery) in
> > response to these changes. For example:
> >
> > 1) How one should execute DML/DDL command and validate the result?
> > int updateCnt = IgniteCache.query(SqlQuery).getAll().get(0);
> >
> > Counter-intuitive and too verbose.
> >
> > 2) How one should perform batched operations? One by one?
> > IgniteCache.query(SqlQuery.setArgs(1));
> > IgniteCache.query(SqlQuery.setArgs(2));
> > ...
> > IgniteCache.query(SqlQuery.setArgs(N));
> >
> > I think it is time to rework our API so that it supports all recent
> > features in consistent way and is extensible enough for future
> improvements
> > (e.g. transactional SQL).
> >
> > Probably we can take ideas behind JDBC standard as starting point and
> move
> > SQL to separate API. Several very rough examples:
> >
> > 1) Getting facade:
> > IgniteSql sql = ignite.sql("MY_SCHEMA");
> >
> > 2) Running SELECT:
> > QueryCursor cursor = sql.select(SqlCommand);
> > QueryCursor cursor = sql.selectFields(SqlCommand); // No more
> need
> > for separate SqlQuery and SqlQueryFields classes.
> >
> > 3) Running DML/DDL:
> > SqlCommandResult res = sql.execute(SqlCommand);
> > or
> > int updatedCnt = sql.execute(SqlCommand);
> >
> > 4) Running batch commands:
> > SqlCommand cmd = new SqlCommand(...).addBatch(arg1).addBatch(arg2);
> > SqlCommandResult res = sql.execute(cmd);
> >
> > 5) Re-use query parsing results (PreparedStatement):
> > SqlCommand qry = sql.prepare("SELECT ...");
> >
> > Our JDBC driver is not applicable here because it is either not peformant
> > enough (V1), or starts unnecessary client inside (V2).
> >
> > Thoughts? Does anyone else think it is time to re-approach SQL API?
> >
> > Vladimir.
> >
>

Re: DML data streaming

[Alexander Paschenko]

And to avoid further confusion: UPDATE and DELETE are simply
impossible in streaming mode when the key is not completely defined as
long as data streamer operates with key-value pairs and not just
tuples of named values. That's why we can't do DELETE from Person
WHERE id1 = 5 from prev example with streamer - the Key { id1 = 5, id2
= 0 } that would be constructed from such query is just one key and is
handled by streamer as such while semantically that query is not about
ONE key but about ALL keys where id1 = 5.

- Alex

2017-02-10 11:49 GMT+03:00 Alexander Paschenko
:
> Dima,
>>
>> There are several ways to handle it. I would check how other databases
>> handle it, maybe we can borrow something. To the least, we should log such
>> errors in the log for now.
>>
>
> Logging errors would mean introducing some kind of stream receiver to
> do that and thus that would be really the same performance penalty for
> the successful operations. I think we should go with that optional
> flag for semantics after all.
>
>> You don't have to use _key. Primary key is usually a field in the class, so
>> you can use a normal column name. In any case, we should remove any usage
>> of _key before 2.0 is released.
>>
>> Again, if user does not have to specify _key on INSERT, then it is very
>> unclear to me, why user would need to specify _key for UPDATE or DELETE.
>> Something smells here. Can you please provide an example?
>>
>
> UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
> optimized cases - i.e. those where _key (and possibly _val) are
> explicitly specified by the user thus allowing us to map UPDATE and
> DELETE directly to cache's replace and remove operations without
> messing with entry processors and doing map-reduce SELECT by given
> criteria.
>
> Say, we have Person { firstName, secondName } with key class Key { id1, id2 }
>
> If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
> there's no need to do any SELECT - we can just call IgniteCache.remove
> on that key.
>
> But if I say DELETE from Person WHERE id1 = 5 then there's no way to
> avoid MR - we have to find all keys that interest us first by doing
> SELECT as long as we know only partly about what keys the user wants
> to be affected.
>
> It works in the same way for UPDATE. And I hope that it's clear how
> it's different from INSERT - there's no MR by definition (we don't
> allow INSERT FROM SELECT in streaming mode).
>
> AGAIN: this all is said only about streaming mode; non streaming mode
> does those optimizations too, but it also allows complex conditions,
> while streaming mode does not allow them to keep things fast and avoid
> MR.
>
> That's the reason why I suggest that we drop UPDATE and DELETE from
> DML streaming as they mean messing with those soon-hidden columns.
>
> Still we could optimize stuff like DELETE from Person WHERE id1 = 5
> AND id2 = 6 - query involves ALL fields of key AND compares only for
> equality AND has no complex expressions - we can construct key
> unambiguously and still call remove directly.
>
> But to me it does not sound like a really great reason to leave UPDATE
> and DELETE in DML - the users will have to write some specific queries
> to use that while all other stuff will just be declined in that mode.
> And, as I said before, UPDATE and DELETE don't probably perfectly fit
> with primary data streamer use cases - after all, modifying existing
> stuff is not what data streamer is about.
>
> And regarding hiding columns: it's unclear how things will look like
> for caches like  when we remove _key and _val as long as
> tables for such cases currently have nothing but those two columns.
>
> - Alex
>
>>> 8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
>>> dsetrak...@apache.org> написал:
>>>
>>> > Alexander,
>>> >
>>> > Are you suggesting that currently to execute a simple INSERT for 1 row we
>>> > invoke a data streamer on Ignite API? How about an update by a primary
>>> key?
>>> > Why not execute a simple cache put in either case?
>>> >
>>> > I think we had a separate thread where we agreed that the streamer should
>>> > only be turned on if a certain flag on a JDBC connection is set, no?
>>> >
>>> > D.
>>> >
>>> > On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
>>> > alexander.a.pasche...@gmail.com> wrote:
>>> >
>>> > > Hello Igniters,
>>> > >
>>> > > I'd like to raise few questions regarding data streaming via DML
>>> > > statements.
>>> > >
>>> > > Currently, all types of DML statements are supported (INSERT, UPDATE,
>>> > > DELETE, MERGE).
>>> > >
>>> > > UPDATE and DELETE are supported in streaming mode only when their
>>> > > WHERE condition is bounded with _key and/or _val columns, and UPDATE
>>> > > works only for _val column directly.
>>> > >
>>> > > Seeing some activity in direction of hiding _key and _val from the
>>> > > user as far as possible, these features seem pointless and should not
>>> > > be 

Re: IGNITE-4212 (Ignite Benchmarking Simplification and Automation)

[Oleg Ostanin]

Hi.

I just discovered that some functionality does not work correctly. I will
present fixed version later.

On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
wrote:

> https://drive.google.com/open?id=0B9teTzJ9iIvHckl6eVQ3N0R4a3c
>
> On Fri, Feb 10, 2017 at 12:43 AM, Oleg Ostanin 
> wrote:
>
>> I've fixed README.txt and created new DEVNOTES.txt. Also renamed sample
>> benchmark configuration file to benchmark-sample.properties.
>> Please look for the results:
>>
>>
>> On Tue, Jan 31, 2017 at 9:18 PM, Denis Magda  wrote:
>>
>>> Oleg,
>>>
>>> Thanks for the clarification.
>>>
>>> My opinion is that we should leave ‘modules/yardstick/README.txt’ and
>>> ‘modules/yardstick/DEVNOTES.txt’ either unchanged or have only those
>>> instructions there that explain how to build and run benchmarks from
>>> ‘modules/yardstick’. This existing files can refer to the sources and
>>> compiled benchmarks that are in Ignite binary releases but this should be a
>>> couple of statements, no more.
>>>
>>> As for the instructions related to the sources and binaries added to
>>> Ignite binaries, preferably they need to be added to auto-generated
>>> README.txt. *Anton*, is it feasible to do?
>>>
>>> Finally, when apply the reviews notes please build and share Ignite
>>> binaries with your patch. Want to see the new project structure and final
>>> version of the instructions. Presently I can’t merge your changes due to
>>> some conflicts.
>>>
>>> *Anton*, please review modifications in the build procedures.
>>>
>>> —
>>> Denis
>>>
>>> > On Jan 31, 2017, at 3:12 AM, Oleg Ostanin 
>>> wrote:
>>> >
>>> > Hi Denis,
>>> >
>>> > Yes, we have included Ignite Yardstick source files with its pom.xml in
>>> > Ignite binary release. "Building from standalone sources" is the
>>> building
>>> > instruction for these source files. "Building from Ignite Sources" is
>>> the
>>> > instruction for building Ignite Yardstick from `modules/yardstick` in
>>> > Ignite source files.
>>> >
>>> > On Tue, Jan 31, 2017 at 3:31 AM, Denis Magda 
>>> wrote:
>>> >
>>> >> Hi Oleg,
>>> >>
>>> >> Great progress, thanks for keep driving this!
>>> >>
>>> >> I’ve left some minor notes in GitHub’s pull-request. I have the
>>> following
>>> >> questions aside:
>>> >>
>>> >> - What is the difference between "Building from standalone sources"
>>> and
>>> >> "Building from Ignite Sources"? In my understanding, a user downloads
>>> >> Apache Ignite release that has all the sources locally.
>>> >>
>>> >> - I do remember we planned to add the benchmarks sources in a form of
>>> a
>>> >> ready to be used project with its own pom.xml (similar to examples).
>>> Did
>>> >> you put this task off?
>>> >>
>>> >> —
>>> >> Denis
>>> >>
>>> >>> On Jan 27, 2017, at 2:13 AM, Oleg Ostanin 
>>> wrote:
>>> >>>
>>> >>> Hi!
>>> >>>
>>> >>> I've changed the README.txt and DEVNOTES.txt files. Also added a
>>> simple
>>> >>> config file for quick and easy start. Please take a look at them and
>>> tell
>>> >>> me what you think.
>>> >>>
>>> >>> https://github.com/apache/ignite/pull/1471
>>> >>>
>>> >>> On Wed, Dec 28, 2016 at 8:59 AM, Ilya Suntsov >> >
>>> >> wrote:
>>> >>>
>>>  Denis,
>>> 
>>>  I think we can remove all configs except:
>>> 
>>>  benchmark-multicast.properties
>>> 
>>>  benchmark.properties
>>> 
>>>  ignite-base-config.xml
>>> 
>>>  ignite-localhost-config.xml
>>> 
>>>  ignite-multicast-config.xml
>>> 
>>>  2016-12-28 2:49 GMT+03:00 Denis Magda :
>>> 
>>> > I would have only those configs that are useful. Ilya Suntsov,
>>> basing
>>> >> on
>>> > your experience, please suggest which configs makes sense to
>>> include
>>> >> into
>>> > every Ignite release.
>>> >
>>> > Oleg, also please note that community decided to include not only
>>> the
>>> > benchmarking binaries but the sources as well into every Apache
>>> Ignite
>>> > release. I’ve update the ticket before. Hope you followed the
>>> >> discussion
>>>  ;)
>>> > https://issues.apache.org/jira/browse/IGNITE-4212?
>>> > focusedCommentId=15765151=com.atlassian.jira.
>>> > plugin.system.issuetabpanels:comment-tabpanel#comment-15765151
>>> >
>>> > —
>>> > Denis
>>> >
>>> >> On Dec 27, 2016, at 5:35 AM, Oleg Ostanin 
>>>  wrote:
>>> >>
>>> >> I mean removing those configs from binary assembly, not from
>>>  repository.
>>> >>
>>> >> On Tue, Dec 27, 2016 at 4:28 PM, Oleg Ostanin <
>>> oosta...@gridgain.com>
>>> > wrote:
>>> >>
>>> >>> Hello Igniters.
>>> >>> I think it would be better to remove some configuration files
>>> from
>>> >>> benchmarks/config:
>>> >>>
>>> >>> ignite-base-load-config.xml
>>> >>> ignite-cache-load-config.xml
>>> >>> 

[jira] [Created] (IGNITE-4678) Web Console: Implement demo load as service

[Andrey Novikov (JIRA)]

Andrey Novikov created IGNITE-4678:
--

 Summary: Web Console: Implement demo load as service
 Key: IGNITE-4678
 URL: https://issues.apache.org/jira/browse/IGNITE-4678
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Affects Versions: 1.8
Reporter: Andrey Novikov
Assignee: Andrey Novikov
Priority: Minor
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: DML data streaming

[Alexander Paschenko]

Dima,
>
> There are several ways to handle it. I would check how other databases
> handle it, maybe we can borrow something. To the least, we should log such
> errors in the log for now.
>

Logging errors would mean introducing some kind of stream receiver to
do that and thus that would be really the same performance penalty for
the successful operations. I think we should go with that optional
flag for semantics after all.

> You don't have to use _key. Primary key is usually a field in the class, so
> you can use a normal column name. In any case, we should remove any usage
> of _key before 2.0 is released.
>
> Again, if user does not have to specify _key on INSERT, then it is very
> unclear to me, why user would need to specify _key for UPDATE or DELETE.
> Something smells here. Can you please provide an example?
>

UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
optimized cases - i.e. those where _key (and possibly _val) are
explicitly specified by the user thus allowing us to map UPDATE and
DELETE directly to cache's replace and remove operations without
messing with entry processors and doing map-reduce SELECT by given
criteria.

Say, we have Person { firstName, secondName } with key class Key { id1, id2 }

If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
there's no need to do any SELECT - we can just call IgniteCache.remove
on that key.

But if I say DELETE from Person WHERE id1 = 5 then there's no way to
avoid MR - we have to find all keys that interest us first by doing
SELECT as long as we know only partly about what keys the user wants
to be affected.

It works in the same way for UPDATE. And I hope that it's clear how
it's different from INSERT - there's no MR by definition (we don't
allow INSERT FROM SELECT in streaming mode).

AGAIN: this all is said only about streaming mode; non streaming mode
does those optimizations too, but it also allows complex conditions,
while streaming mode does not allow them to keep things fast and avoid
MR.

That's the reason why I suggest that we drop UPDATE and DELETE from
DML streaming as they mean messing with those soon-hidden columns.

Still we could optimize stuff like DELETE from Person WHERE id1 = 5
AND id2 = 6 - query involves ALL fields of key AND compares only for
equality AND has no complex expressions - we can construct key
unambiguously and still call remove directly.

But to me it does not sound like a really great reason to leave UPDATE
and DELETE in DML - the users will have to write some specific queries
to use that while all other stuff will just be declined in that mode.
And, as I said before, UPDATE and DELETE don't probably perfectly fit
with primary data streamer use cases - after all, modifying existing
stuff is not what data streamer is about.

And regarding hiding columns: it's unclear how things will look like
for caches like  when we remove _key and _val as long as
tables for such cases currently have nothing but those two columns.

- Alex

>> 8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
>> dsetrak...@apache.org> написал:
>>
>> > Alexander,
>> >
>> > Are you suggesting that currently to execute a simple INSERT for 1 row we
>> > invoke a data streamer on Ignite API? How about an update by a primary
>> key?
>> > Why not execute a simple cache put in either case?
>> >
>> > I think we had a separate thread where we agreed that the streamer should
>> > only be turned on if a certain flag on a JDBC connection is set, no?
>> >
>> > D.
>> >
>> > On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com> wrote:
>> >
>> > > Hello Igniters,
>> > >
>> > > I'd like to raise few questions regarding data streaming via DML
>> > > statements.
>> > >
>> > > Currently, all types of DML statements are supported (INSERT, UPDATE,
>> > > DELETE, MERGE).
>> > >
>> > > UPDATE and DELETE are supported in streaming mode only when their
>> > > WHERE condition is bounded with _key and/or _val columns, and UPDATE
>> > > works only for _val column directly.
>> > >
>> > > Seeing some activity in direction of hiding _key and _val from the
>> > > user as far as possible, these features seem pointless and should not
>> > > be released, what do you think?
>> > >
>> > > Also INSERT in streaming mode currently does not throw errors on
>> > > duplicate keys and silently ignores such new records (as long as it's
>> > > faster than it would work if we'd introduced receiver that would throw
>> > > exceptions) - this can be fixed with additional flag that could
>> > > _optionally_ make INSERT slower but more accurate in semantic.
>> > >
>> > > And MERGE in streaming mode currently not totally accurate in
>> > > semantic, too - on key presence, it will just replace whole value with
>> > > new one thus potentially making values of some concrete columns/fields
>> > > lost - this is analogous to
>> > > https://issues.apache.org/jira/browse/IGNITE-4489, but hardly can be
>> > > fixed 

Re: IGNITE-1948 ClusterTopologyCheckedException can return null for retryReadyFuture()

[Alexey Goncharuk]

Alexander,

I do not see why you would need to overwrite the cause field.

What I meant in the previous email is that instead of setting a retry ready
future in the checked exception, you would set a correct affinity topology
version here. Then, during a construction of CacheException (unchecked,
which is guaranteed to be thrown locally and will not be serialized), you
would create the retry ready future based on the topology version set.

Hope this helps,
AG

2017-02-07 17:18 GMT+03:00 Александр Меньшиков :

> Alexey, I ran into some difficulties.
>
> Look at the method: GridNearTxFinishFuture.CheckBackupMiniFuture#
> onDhtFinishResponse(GridDhtTxFinishResponse res)
>
> *Throwable err* = res.checkCommittedError();
>
> if (*err* != null) {
> if (*err* instanceof IgniteCheckedException) {
> *ClusterTopologyCheckedException cause* =
> ((IgniteCheckedException)*err*).
> *getCause(ClusterTopologyCheckedException.class)*;
>
> if (*cause* != null)
> *cause.retryReadyFuture(*cctx.
> nextAffinityReadyFuture(tx.topologyVersion()));
> *   //^_Here update the readyFut in
> some first "cause". *
> }
>
> onDone(*err*);
> }
>
>
> So I can't rewrite "cause" field in exception without reflection. It means
> if I try to use two exception: one with "readyFut" and second without
> "readyFut", I see no way to do it in this place.
>
> Perhaps I misunderstood your original idea. Or maybe this code some kind
> of a crutch and should be removed. What do you think?
>
> 2017-01-30 16:54 GMT+03:00 Alexey Goncharuk :
>
>> In the example above there is no point of setting the retry future in the
>> exception because it will be serialized and sent to a remote node.
>>
>> I see the only possible way to ensure this: make this be verified at
>> compile time. This looks like a big change, but the draft may look like
>> so:
>> 1) Introduce new exception type, like CacheTopology*Checked*Exception
>> which
>> *must* take the minimum topology version to wait for
>> 2) In all cases when a cache operation fails because of topology change,
>> provide the appropriate exception
>> 3) When CacheTopologyException is being constructed, create a
>> corresponding
>> local future based on wait version and throw the exception.
>>
>> Even though this still does not give us 100% guarantee that we did not
>> miss
>> anything, it should cover a lot more cases than now.
>>
>> 2017-01-30 14:20 GMT+03:00 Александр Меньшиков :
>>
>> > Alexey,
>> >
>> > For GridCacheIoManager#sendNoRetry it's not necessary because exception
>> > will be ignored (or will cast to String). Perhaps my message was
>> unclear.
>> > I try to say that three methods can throw "ClusterTopologyCheckedExcepti
>> on"
>> > without any problem.
>> >
>> > But you are right, in some methods I can't set "retryFuture". We must
>> > ensure that exception doesn't escape away. The best option is to ignore
>> it
>> > (or cast to String).
>> >
>> > But any way, look here:
>> >
>> > IgniteTxHandler#sendReply(UUID nodeId, GridDhtTxFinishRequest req,
>> boolean
>> > committed, GridCacheVersion nearTxId)
>> >
>> > Inside you can found next lines:
>> > __
>> > ClusterTopologyCheckedException *cause* =
>> > new ClusterTopologyCheckedException("Primary node left grid.");
>> >
>> > res.checkCommittedError(new IgniteTxRollbackCheckedException("Failed to
>> > commit transaction " +
>> > "(transaction has been rolled back on backup node): " +
>> req.version(),
>> > *cause*));
>> > __
>> >
>> > How do you unsure that *"cause"* can't come to
>> GridCacheUtils#retryTopologySafe(final
>>
>> > Callable c) ?
>> >
>> >
>> > Perhaps I don't know some rules which you use to maintain it now?
>> >
>> >
>> > 2017-01-30 12:24 GMT+03:00 Alexey Goncharuk > >:
>> >
>> >> Alexander,
>> >>
>> >> Do you have a use-case in mind which results in IgniteTopologyException
>> >> thrown from public API with retryFuture unset?
>> >>
>> >> retryFuture makes sense only for certain cache operations and may be
>> set
>> >> only in certain places in the code where we already know what to wait
>> for.
>> >> I do not see how you can initialize this future, for example, in
>> >>  GridCacheIoManager#sendNoRetry(ClusterNode node, GridCacheMessage
>> msg,
>> >> byte
>> >> plc)
>> >>
>> >> --AG
>> >>
>> >> 2017-01-27 15:45 GMT+03:00 Александр Меньшиков :
>> >>
>> >> > I found a lot of using "ClusterTopologyCheckedException" exception.
>> In
>> >> > most
>> >> > use-case these exceptions were just ignored. It's easier to see if
>> >> > issue-4612 will be applied (I'm waiting for code review by my team
>> >> leader)
>> >> > where I renamed all unused exceptions in the