Re: Rename "cache group" to "tablespace"?

2018-11-19 Thread Dmitriy Setrakyan
I really like the idea.

On Mon, Nov 19, 2018, 23:45 Vladimir Ozerov  Igniters,
>
> We had several discussion about cache groups [1]. Our current consensus is
> that this is not that good design decision - bad scan performance, no API.
> A kind of shortcut to solve memory consumption problems. For this reason we
> try to avoid exposing "cache group" to API when possible. Also we
> understand that current file-per-partition approach is not ideal either -
> when there are too many partitions we have to fsync a lot of files. And as
> an idea for "bright future" we consider tablespaces - one or several files
> where all database objects are stored in dedicated "segments", which are
> allocated in smaller units called "extents".
>
> I was thinking on how to "legalize" cache groups on API on the one hand
> (product needs them currently), and to let real tablespaces to appear in
> future. Idea is simple - treat cache group as special kind of tablespace.
> We may say that current storage organization is one type of tablespace, and
> proposed "file-segment-extent" storage organization is another type of
> tablespace.
>
> With this in mind we can configure tablespaces in IgniteConfiguration, then
> assign each cache a tablespace. As advanced tasks we may allow dynamic
> table space create/alter/drop, I'll show SQL examples below.
>
> // Interface
> interface Tablespace {
> String name();
> }
>
> // Current non-shared tablespace (aka "physical cache")
> class FilePerPartitionTablespace implements Tablespace {
> // ...
> }
>
> // Shared tablespace (aka "cache group") - note that now we have a legal
> place for cache group configuration:
> class FilePerPartitionSharedTablespace implements Tablespace {
> CacheMode cacheMode;
> CacheAtomicityMode cacheAtomicityMode;
> AffinityFunction affinityFunction;
> // + Other parameters from
> ClusterCachesInfo.validateCacheGroupConfiguration
> // + Some parameters from "DataRegionConfiguration" (e.g. paths)
> }
>
> // Future "real" tablespace
> class SegmentedTablespace implements Tablespace {
> // Whatever is needed.
> }
>
> // Cache configuration
> class CacheConfiguration {
> ...
> String tablespace;
>...
> }
>
> // Managing tablespaces from SQL:
> CREATE TABLESPACE my_tbs ENGINE=FILE_PER_PARTITION_SHARED
> CACHE_MODE=PARTITIONED BACKUPS=1
> ALTER TABLESPACE my_tbs ENCRYPTED=1
> CREATE TABLE my_table (...) TABLESPACE my_tbs
>
> What do you think?
>
> Vladimir.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Remove-cache-groups-in-AI-3-0-td29184.html
>


Re: Time to remove automated messages from the devlist?

2018-11-06 Thread Dmitriy Setrakyan
I just have a filter for Jira emails and automatically go into a different
folder for me.

On Tue, Nov 6, 2018 at 6:40 AM Dmitriy Pavlov  wrote:

> Vova, I've also started such topic about GitBox messages (
>
> https://lists.apache.org/thread.html/1870ba56e0eb9e184d055ef2c84114ea43219d7c845036566f68e880@%3Cdev.ignite.apache.org%3E
> ).
> But it seems no one reacted.
>
> I agree to move out GitHub PR notifications + GitBox messages (see a
> solution in the thread). I disagree about test failures, because a
> contributor may just disappear after a fix, and someone else should pick up
> the test fix.
>
> If we will be open, positive and welcoming we will be still in top-5 dev.
> lists, because we have something to say :)
>
> Here are some thoughts about busy lists that
> - might help to be open,
> - help everyone to feel free to discuss things
> - help to notice a useful staff
>  https://grep.codeconsult.ch/2011/12/06/stefanos-mazzocchis-busy-list-
> pattern/
> 
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 6 нояб. 2018 г. в 17:31, Vladimir Ozerov :
>
> > Igniters,
> >
> > I noted that the most common action I do on the devlist is clicking "Mark
> > As Read". All I see is JIRA and GitHub notifications. I simply counted
> last
> > 100 threads and found that 89 of them are automated messages, 2 are MTCGA
> > bot messages, and 9 are real human-created threads.
> >
> > Looks like human-created threads are drown inside generated stuff. As
> > potentially new contributor you will hardly figure out what is going on.
> > Does any info about opened tickets and PRs help anyone? Don't we want
> > instead to have a list like this [1] and move all generated stuff to
> > separate lists?
> >
> > The only drawback I see from this action is that we no longer be able to
> > brag about "Top 5 active devlist" in annual ASF summary :-)
> >
> > Thoughts?
> >
> > Vladimir.
> >
> > [1] http://apache-spark-developers-list.1001551.n3.nabble.com/
> >
>


Re: Applicability of term 'cache' to Apache Ignite

2018-10-18 Thread Dmitriy Setrakyan
I am beginning to like IgniteTable as well. How would something like this
be introduced to Ignite? Would we have IgniteTable extend IgniteCache? What
would happen to cache groups?

D.

On Thu, Oct 18, 2018 at 7:58 AM Павлухин Иван  wrote:

> HI all,
>
> +1 for "table" from me. For me "table" has several benefits:
> 1. It's common and consequently easy to explain and understand.
> 2. It's quite universal. One can worry that "table" does not describes
> key-value storage well.
> I don't see any problem here, because Hash Table data structure
> contains "table" word it it's name.
> Also DHT comes to mind. Internally we have GridDhtCache class. So it's
> already a "table".
>
> Regarding multiple QueryEntities in single cache. Correct me if I am wrong,
> but currently we do not recommend to use them.
>
> чт, 18 окт. 2018 г. в 15:18, David Harvey :
>
> > We had a terminology agreement early on where we agreed to call them
> > caches, but we still call them tables anyway.
> >
> > When I finally understood how you could have multiple tables in a single
> > cache,  I tried to find example use cases, but couldn't.  Is there even a
> > test with multiple queryEntities?
> >
> > On Thu, Oct 18, 2018, 8:10 AM Alexey Zinoviev 
> > wrote:
> >
> > > From my perspective (ML module), it will be very easy to talk about
> > Ignite
> > > in SQL terms like table (with additional information about ability to
> > make
> > > key-value CRUD operations, not only SELECT * FROM Table)
> > > Also we could look on PostgreSQL with different plugins for SQL
> extension
> > > like PostGIS or support of JSON-B and ability to store not only planar
> > data
> > > with strict schema (I agrre here with Vladimir).
> > >
> > > чт, 18 окт. 2018 г. в 14:33, Ilya Lantukh :
> > >
> > > > I thought that current "caches" and "tables" have 1-to-N relation. If
> > > > that's not a problem, than I also think that "table" is the best
> term.
> > > >
> > > > On Thu, Oct 18, 2018 at 9:29 AM Vladimir Ozerov <
> voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Well, I never thought about term “table” as a replacement for
> > “cache”,
> > > > but
> > > > > it appears to be good candidate.
> > > > >
> > > > > This is used by many some major vendors whose underlying storage is
> > > > indeed
> > > > > a kind of key-value data structure. Most well-known example is
> MySQL
> > > with
> > > > > its MyISAM engine. Table can be used for both fixed and flexible
> > (e.g.
> > > > > JSON) schemas, as well as key-value access (hash map -> hash table,
> > > both
> > > > > are good).
> > > > >
> > > > > Another important thing - we already use term “table”, and it is
> > always
> > > > > hard to explain our users how it relates to “cache”. If “cache” is
> > > > dropped,
> > > > > then a single term “table” will be used everywhere.
> > > > >
> > > > > Last, but not least - “table” works well for both in-memory and
> > > > persistent
> > > > > modes.
> > > > >
> > > > > So if we are really aim to rename “cache”, then “table” is the best
> > > > > candidate I’ve heard so far.
> > > > >
> > > > > чт, 18 окт. 2018 г. в 8:40, Alexey Zinoviev <
> zaleslaw@gmail.com
> > >:
> > > > >
> > > > > > Or we could extend our SQL commands by "GET BY KEY = X" and "PUT
> > (x1,
> > > > x2,
> > > > > > x3) BY KEY = X" and the IgniteTable could be correct.
> > > > > > Agree with Denis that each table in the 3rd normal form is like
> > > > key-value
> > > > > > store. Key-value operations are only subset of rich SQL commands.
> > > > > >
> > > > > > The problem with IgniteData that it's too common. Also, it's
> > > difficult
> > > > to
> > > > > > understand is it a plural or single object? For instance, the
> bunch
> > > of
> > > > > > IgniteTables could be IgniteData. But the set of IgniteData?
> > > > IgniteDatum?
> > > > > >
> > > > > >
> > > > > >
> > > > > > чт, 18 окт. 2018 г. в 4:

Re: Applicability of term 'cache' to Apache Ignite

2018-10-18 Thread Dmitriy Setrakyan
On Thu, Oct 18, 2018 at 5:18 AM David Harvey  wrote:

> We had a terminology agreement early on where we agreed to call them
> caches, but we still call them tables anyway.
>
> When I finally understood how you could have multiple tables in a single
> cache,  I tried to find example use cases, but couldn't.  Is there even a
> test with multiple queryEntities?


Multiple types per cache should still be supported, but I would definitely
advise against using it. Table per cache is much cleaner. On top of that,
Ignite has a notion of cache-groups, so you should combine all caches that
have identical configurations into one group. This will save you space and
make things faster.

D.


Re: Applicability of term 'cache' to Apache Ignite

2018-10-17 Thread Dmitriy Setrakyan
On Wed, Oct 17, 2018 at 4:58 PM Denis Magda  wrote:

> I've been calling everything "tables" instead of "caches" for a while. The
> main reason is the maturity of our SQL engine - seeing more SQL users and
> deployments which talk "tables" language.
>
>
I think "IgniteTable" only implies SQL, not key-value. We need both.


Re: Applicability of term 'cache' to Apache Ignite

2018-10-17 Thread Dmitriy Setrakyan
If dataset cannot be used, can we still consider "IgniteData"?

D.

On Wed, Oct 17, 2018 at 5:06 AM Ilya Lantukh  wrote:

> As I see, many people agree that the term *"cache"* is outdated, but
> consider these changes too disruptive.
>
> For me, keeping terminology up-to-date is important part of project
> development. If we change some of our core terms with more relevant ones,
> it indeed might cause confusion for current users, but in long term it will
> help new users to understand what Ignite is and what it isn't. And most
> short-term problems can easily be avoided by keeping @Deprecated
> IgniteCache.
>
> On Wed, Oct 17, 2018 at 2:59 PM Ilya Lantukh 
> wrote:
>
> > Unfortunately, we already use the word *"store"* for many other concepts,
> > like CacheStore and PageStore. I'd prefer to avoid giving it one more
> > meaning.
> >
> > As already mentioned, *"dataset"* has special meaning for ML folks.
> >
> > *"Bucket" *might give wrong association with bucket in a hash table.
> >
> > On Wed, Oct 17, 2018 at 1:49 PM Igor Sapego  wrote:
> >
> >> Well, the obvious term for me is a "Store" or "MemoryStore", as we
> already
> >> have persistence store.
> >>
> >> Best Regards,
> >> Igor
> >>
> >>
> >> On Wed, Oct 17, 2018 at 1:19 PM Andrey Kuznetsov 
> >> wrote:
> >>
> >> > I'm not an ML expert, so 'dataset' term just reminds me of various
> >> client
> >> > drivers to access tables from RDBM servers. For me, the only common
> >> trait
> >> > of all kinds of Ignite caches is their asociativity. So if we rename
> >> them
> >> > I'd suggest something like KVStore.
> >> >
> >> > ср, 17 окт. 2018 г. в 12:56, Alexey Zinoviev  >:
> >> >
> >> > > From my perspective, the main goal is to make easy the explanation
> >> what
> >> > is
> >> > > Ignite on conferences, marketing deals, in papers, in documentation.
> >> And
> >> > > the
> >> > > /cache/ term really reduces the area of Ignite usage in users minds.
> >> > >
> >> > > I don't support the critical changes in code base, but I support all
> >> > > changes
> >> > > that helps the goal described above in this letter.
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >> > >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >   Andrey Kuznetsov.
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Ilya
> >
>
>
> --
> Best regards,
> Ilya
>


Re: SQL Engine

2018-10-17 Thread Dmitriy Setrakyan
I do not understand - why are we copying values on-heap? If we must copy
something, why not copy rows into some off-heap space and read what we have
from there? At least this way we will not burden the JVM with extra GC
pressure.

D.

On Wed, Oct 17, 2018 at 10:33 AM Igor Tanackovic 
wrote:

> Vladimir,
>
> Thanks for explanation... that’s true - ignite never deserialize values
> during query execution. The point here is why copying fields to heap that
> sql engine could not use (not annotated as QuerySqlField)?
> SQL count is a perfect example where you can benefit (heap space and gc)
> copying only field sql engine could use.
>
> Regards,
> Igor
>
> On Wed, Oct 17, 2018 at 18:47 Vladimir Ozerov 
> wrote:
>
> > Hi Igor,
> >
> > We never deserialize values during query execution. Instead, we copy the
> > row to heap and extract fields as needed. In general case it is
> impossible
> > to avoid reading the whole row because we do not know whether this is
> > COUNT(*) or COUNT(*) WHERE  or COUNT() WHERE
> > .
> > We do handle plain COUNT(*) as speical case and iterate over index only,
> > but event this simple query cannot avoid row reads in general case when
> new
> > snapshot mode is enabled, because an entry in the index may be not
> visible
> > to current transaction.
> >
> > On Wed, Oct 17, 2018 at 7:19 PM igor.tanackovic <
> igor.tanacko...@gmail.com
> > >
> > wrote:
> >
> > > Hello,
> > >
> > > Seems that SQL engine always deserialize whole objects instead of using
> > > just
> > > SQL enabled fields (annotated with @QuerySqlField). This may have a
> huge
> > > impact on Ignite heap usage and GC overhead as well.
> > >
> > > For example, we have a cache holding big objects but with only two sql
> > > query
> > > fields which for each query execution (SELECT COUNT(*) FROM 'cache')
> > > consumes large amount on heap memory (~300MB). As a proof of concept,
> we
> > > divided the same cache to *index* cache with only sql query field and a
> > > *data* holding whole object for materialization. The same query (SELECT
> > > COUNT(*) FROM 'index-cache') consumes ~25 time less memory! The same is
> > > true
> > > for all other queries.
> > >
> > > The obvious workaround would be to always have separated regions for
> > > indexes
> > > (sql query enabled region) and a data/value region for materialization,
> > but
> > > it might be a good idea to fix this in a systematic way during off heap
> > > deserialization.
> > >
> > > Regards,
> > > Igor
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>


[DISCUSS] Dropping hadoop-accelerator download

2018-10-16 Thread Dmitriy Setrakyan
Igniters,

I would like to suggest dropping the hadoop accelerator distribution of
Apache Ignite. Note, that it does not mean dropping any features. We are
still going to support IGFS, in-memory map reduce, spark integration, etc.
However, the hadoop-accelerator downloadable is putting extra pressure on
the community to test and document, which can be avoided. All the features
will still be supported through regular Ignite distribution.

Any objections?

D.


Re: Applicability of term 'cache' to Apache Ignite

2018-10-16 Thread Dmitriy Setrakyan
Hm... How about IgniteData or IgniteDataset?

D.

On Tue, Oct 16, 2018 at 11:52 AM Stanislav Lukyanov 
wrote:

> I had an idea of “IgniteBucket” as in “a place to put things in”.
>
> But I think I like “space” since it sounds like and conceptually very
> close (if not identical) to “tablespace”.
>
> I have to say I never heard of JavaSpaces :) Don’t think many people will
> recall that.
>
> Stan
>
> From: Dmitriy Setrakyan
> Sent: 16 октября 2018 г. 20:21
> To: dev
> Subject: Re: Applicability of term 'cache' to Apache Ignite
>
> Although I agree that this change is disruptive, can we just entertain
> Ilya's idea for a bit? What if we were designing Ignite from scratch, what
> different name would we give to the IgniteCache abstraction? Ilya suggested
> "IgniteSpace", but I do not like it as it sounds too similar to JavaSpaces
> [1], which is an obsolete technology at this point.
>
> Any other ideas?
>
> [1]
> https://www.oracle.com/technetwork/articles/java/javaspaces-140665.html
>
> D.
>
> On Tue, Oct 16, 2018 at 6:27 AM Ivan Rakov  wrote:
>
> > Agree with Vladimir here.
> >
> > Let's stick to the "principle of least astonishment" - all current users
> > will be surprised if we'll rename IgniteCache, new users won't be
> > greatly surprised due to compliance with JCache.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 16.10.2018 15:53, Vladimir Ozerov wrote:
> > > What is the ultimate goal of all these changes? While I agree that term
> > > "cache" might be a bit outdated at the moment, there is nothing
> > > fundamentally wrong with - data is still being cached in memory with an
> > > option to persist it on disk. We should remember, that legacy and
> > previous
> > > user experience is of great importance for users. And disruptive
> changes
> > > such as rename of a basic concept may make adoption of a new versions
> > > harder for users, with very questionable benefits on the other side.
> > >
> > > As far as wrappers, personally I do not support this idea. Both "cache"
> > and
> > > "sql" are access methods to some information ("space"), rather than
> > > wrappers around it. Moreover, it is hard to say whether we will have
> SQL
> > > API at all, because this is big effort with not very clear value,
> > provided
> > > that there are industrial interfaces (JDBC, ODBC).
> > >
> > > On Tue, Oct 16, 2018 at 3:23 PM Stanislav Lukyanov <
> > stanlukya...@gmail.com>
> > > wrote:
> > >
> > >> How about separating our JCache implementation from the core of the
> > >> probuct.
> > >>
> > >> Currently IgniteCache is the heart of Ignite. It is the basic storage
> > unit.
> > >> At the same time, it is the direct implementation of the JCache API,
> > >> and some of the JCache features align somewhat awkwardly with Ignite
> > >> concepts.
> > >>
> > >> Would be nice to have something like IgniteSpace as our core
> component,
> > >> and have other components built on top of it as wrappers providing
> > various
> > >> APIs.
> > >> For example
> > >> - IgniteSpace itself is a distributed storage unit, that is
> partitioned,
> > >> that has affinity, etc;
> > >> note that it doesn’t have to have ANY particular API to add data, even
> > >> key-value
> > >> - IgniteCache is a wrapper around IgniteSpace that allows to store
> > >> key-value pairs and implements JCache API
> > >> - IgniteSql (we’re doing it eventually, right?) is a wrapper around
> > >> IgniteSpace that allows to store SQL tables and implements ANSI SQL
> > >> - IgniteQueue is a wrapper that implements Queue
> > >> and so on.
> > >>
> > >> WDYT?
> > >>
> > >> Stan
> > >>
> > >> From: Ilya Lantukh
> > >> Sent: 15 октября 2018 г. 14:49
> > >> To: dev@ignite.apache.org
> > >> Subject: Applicability of term 'cache' to Apache Ignite
> > >>
> > >> Hi Igniters,
> > >>
> > >> I would like to rise a question how we use the term *'cache'* in
> Ignite
> > and
> > >> how it corresponds to terminology in IT industry in general.
> > >>
> > >>  From wikipedia:
> > >> In computing <https://en.wikipedia.org/wiki/Computing>, a *cache*
> /kæʃ/
> > >> <https://en.wikipedi

Re: Applicability of term 'cache' to Apache Ignite

2018-10-16 Thread Dmitriy Setrakyan
Although I agree that this change is disruptive, can we just entertain
Ilya's idea for a bit? What if we were designing Ignite from scratch, what
different name would we give to the IgniteCache abstraction? Ilya suggested
"IgniteSpace", but I do not like it as it sounds too similar to JavaSpaces
[1], which is an obsolete technology at this point.

Any other ideas?

[1] https://www.oracle.com/technetwork/articles/java/javaspaces-140665.html

D.

On Tue, Oct 16, 2018 at 6:27 AM Ivan Rakov  wrote:

> Agree with Vladimir here.
>
> Let's stick to the "principle of least astonishment" - all current users
> will be surprised if we'll rename IgniteCache, new users won't be
> greatly surprised due to compliance with JCache.
>
> Best Regards,
> Ivan Rakov
>
> On 16.10.2018 15:53, Vladimir Ozerov wrote:
> > What is the ultimate goal of all these changes? While I agree that term
> > "cache" might be a bit outdated at the moment, there is nothing
> > fundamentally wrong with - data is still being cached in memory with an
> > option to persist it on disk. We should remember, that legacy and
> previous
> > user experience is of great importance for users. And disruptive changes
> > such as rename of a basic concept may make adoption of a new versions
> > harder for users, with very questionable benefits on the other side.
> >
> > As far as wrappers, personally I do not support this idea. Both "cache"
> and
> > "sql" are access methods to some information ("space"), rather than
> > wrappers around it. Moreover, it is hard to say whether we will have SQL
> > API at all, because this is big effort with not very clear value,
> provided
> > that there are industrial interfaces (JDBC, ODBC).
> >
> > On Tue, Oct 16, 2018 at 3:23 PM Stanislav Lukyanov <
> stanlukya...@gmail.com>
> > wrote:
> >
> >> How about separating our JCache implementation from the core of the
> >> probuct.
> >>
> >> Currently IgniteCache is the heart of Ignite. It is the basic storage
> unit.
> >> At the same time, it is the direct implementation of the JCache API,
> >> and some of the JCache features align somewhat awkwardly with Ignite
> >> concepts.
> >>
> >> Would be nice to have something like IgniteSpace as our core component,
> >> and have other components built on top of it as wrappers providing
> various
> >> APIs.
> >> For example
> >> - IgniteSpace itself is a distributed storage unit, that is partitioned,
> >> that has affinity, etc;
> >> note that it doesn’t have to have ANY particular API to add data, even
> >> key-value
> >> - IgniteCache is a wrapper around IgniteSpace that allows to store
> >> key-value pairs and implements JCache API
> >> - IgniteSql (we’re doing it eventually, right?) is a wrapper around
> >> IgniteSpace that allows to store SQL tables and implements ANSI SQL
> >> - IgniteQueue is a wrapper that implements Queue
> >> and so on.
> >>
> >> WDYT?
> >>
> >> Stan
> >>
> >> From: Ilya Lantukh
> >> Sent: 15 октября 2018 г. 14:49
> >> To: dev@ignite.apache.org
> >> Subject: Applicability of term 'cache' to Apache Ignite
> >>
> >> Hi Igniters,
> >>
> >> I would like to rise a question how we use the term *'cache'* in Ignite
> and
> >> how it corresponds to terminology in IT industry in general.
> >>
> >>  From wikipedia:
> >> In computing , a *cache* /kæʃ/
> >>  *kash*
> >> , is a
> >> hardware or software component that stores data so that future requests
> for
> >> that data can be served faster; the data stored in a cache might be the
> >> result of an earlier computation or a copy of data stored elsewhere. [1]
> >>
> >> When the first version of Ignite was released, this term was correct. We
> >> positioned Ignite mostly as an intermediate storage layer between
> >> application and a database, designed to make data access faster.
> >>
> >> However, since addition of native persistence we started to call Ignite
> a
> >> "memory-centric database", and as far as I know, some organizations now
> use
> >> it as a primary data storage, without underlying database. In this case,
> >> calling our storage unit a *'cache'* causes unnecessary confusion.
> >>
> >> Thus, I suggest to rename IgniteCache in Ignite 3.0 to something that
> would
> >> fit both use-cases.
> >> Personally I like the term IgniteSpace.
> >>
> >> [1] https://en.wikipedia.org/wiki/Cache_(computing)
> >> --
> >> Best regards,
> >> Ilya
> >>
> >>
>
>


Re: Apache Ignite 2.7. Last Mile

2018-10-10 Thread Dmitriy Setrakyan
Vladimir Ozerov,

Can you help with the unassigned MVCC tickets?

D.

On Wed, Oct 10, 2018 at 3:32 AM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> I list all contributors that assigned to the 2.7 tickets.
> If you can help them to finish that tickets - please, do.
> Assigners, if you need any help - please, respond to this thread.
>
> NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on it!
>
> Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685, IGNITE-9823
> Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> Taras Ledkov   - IGNITE-9171
> Alexey Goncharuk   - IGNITE-9784
> Dmitriy Govorukhin - IGNITE-9550
> Igor Seliverstov   - IGNITE-9749
> Dmitry Melnichuk   - IGNITE-7782
> Alexey Platonov- IGNITE-9726
> Ivan Pavlukhin - IGNITE-5935
> Yury Babak - IGNITE-8670
> Roman Kondakov - IGNITE-7953, IGNITE-9446
> Maxim Pudov- IGNITE-9126
> Alexey Stelmak - IGNITE-9776
> Alexey Kuznetsov   - IGNITE-7926
>
> Unassigned tickets:
>
> IGNITE-9781 - JDK11: SSL handshake is failed
> IGNITE-9620 - MVCC: select throwing `Transaction is already completed`
> exception after mvcc missmatch
> IGNITE-9292 - MVCC SQL: Unexpected state exception when updating backup
> IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> IGNITE-9724 - MVCC SQL: Test
> CacheMvccSelectForUpdateQueryAbstractTest.testSelectForUpdateDistributed
> hangs sporadically.
> IGNITE-9133 - MVCC: Proper empty DHT transactions handling.
>


Re: Absence of unsigned integer read/write support in IBinaryRawReader/Writer

2018-10-10 Thread Dmitriy Setrakyan
On Tue, Oct 9, 2018 at 9:55 PM Raymond Wilson 
wrote:

>
> I am curious that unsigned ints would not be a universally supported type
> on all platforms. Is that really the case?
>

I am not aware of any unsigned ints in Java, at least yet. Here is a Stack
Overflow thread on this:

https://stackoverflow.com/questions/25556017/how-to-use-the-unsigned-integer-in-java-8-and-java-9/32837775


Re: Absence of unsigned integer read/write support in IBinaryRawReader/Writer

2018-10-09 Thread Dmitriy Setrakyan
This is not about what "Java" supports. This is about the protocol working
seamlessly across Java, .NET, and C++ environments, so we had to remove the
types that are not supported on some platforms.

D.

On Tue, Oct 9, 2018 at 7:57 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> I think this is because Ignite marshallers stick to what Java supports.
> Java only supports signed numbers and it only supports nullable composite
> values (no structs).
>
> Thus on the C# side you can use those types which intersect between Java
> and .Net runtimes.
>
> I can see how this can be inconvenient, unfortunately we don't have that
> strong C# lobby to make the difference currently.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 9 окт. 2018 г. в 0:04, Raymond Wilson :
>
> > I’m using Ignite IBinarizable raw serialization in Ignite v2.6 with C#
> > client.
> >
> >
> >
> > I notice there is no support for unsigned short, int and long integer
> types
> > (both single values and arrays).
> >
> >
> >
> > I also noticed that Decimal, DateTime and Guid read/write methods only
> > support nullable values.
> >
> >
> >
> > Currently I have code that writes a ushort value into an into to preserve
> > its value range. Similarly I have non-nullable Guid values and need to do
> > the nullable dance to on the read side to transform them back to
> > non-nullable.
> >
> >
> >
> > Is there a particular reason these are not supported?
> >
> >
> >
> > Thanks,
> >
> > Raymond.
> >
>


Re: TDE. Phase-1 Merged to master

2018-10-09 Thread Dmitriy Setrakyan
Congrats! Great effort by everyone involved.

On Tue, Oct 9, 2018 at 9:18 AM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> I merged to master TDE.Phase 1 that has been actively discussed on
> dev-list.
>
> I want to thank all guys that helped me with the implementation:
>
> 1. Dmitriy Ryabov   - initial design.
> 2. Anton Vinogradov - an initial review. wisdom sharing.
> 3. Vladimir Ozerov  - final review. priceless advices and knowledge of
> Ignite codebase and trends.
> 4. Dmitriy Pavlov   - security and process help.
> 5. Petr Ivanov  - Team City configuration.
>
> Guys, this improvement wouldn't be done without your help.
> Thank you!
>


Re: Should we make the interface ClusterNode to extend Serializable?

2018-10-06 Thread Dmitriy Setrakyan
Got it. I did not know about DiscoveryEvent.

On Sat, Oct 6, 2018 at 7:04 AM Vyacheslav Daradur 
wrote:

> Hi Dmitriy,
>
> It's not about users, it's about some kind of mismatch when
> serializable objects like 'DiscoveryEvent' contains non-serializable
> fields.
>
> I'm not seeing a big problem for the project just want to point this
> out and to resolve if needed.
> On Fri, Oct 5, 2018 at 8:55 PM Dmitriy Setrakyan 
> wrote:
> >
> > I think I would be against it. Why would anyone serialize ClusterNode
> > outside of Ignite? Did we get any complaints from users?
> >
> > D.
> >
> > On Fri, Oct 5, 2018 at 3:55 AM Vyacheslav Daradur 
> > wrote:
> >
> > > Hi Igniters!
> > >
> > > I've noticed that interface 'ClusterNode' doesn't extend
> > > 'Serializable', but at the same time its implementations are being
> > > transferred across the network widely.
> > >
> > > We have not faced the problem because of the most used implementation
> > > 'TcpDiscoveryNode' implemented 'Externalizable' that allows JVM to
> > > delegate the serialization to the implementation.
> > >
> > > I'd suggest making the interface ClusterNode to extend Serializable.
> > >
> > > What do you think?
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
>


Re: Should we make the interface ClusterNode to extend Serializable?

2018-10-05 Thread Dmitriy Setrakyan
I think I would be against it. Why would anyone serialize ClusterNode
outside of Ignite? Did we get any complaints from users?

D.

On Fri, Oct 5, 2018 at 3:55 AM Vyacheslav Daradur 
wrote:

> Hi Igniters!
>
> I've noticed that interface 'ClusterNode' doesn't extend
> 'Serializable', but at the same time its implementations are being
> transferred across the network widely.
>
> We have not faced the problem because of the most used implementation
> 'TcpDiscoveryNode' implemented 'Externalizable' that allows JVM to
> delegate the serialization to the implementation.
>
> I'd suggest making the interface ClusterNode to extend Serializable.
>
> What do you think?
>
> --
> Best Regards, Vyacheslav D.
>


Re: Apache Ignite 2.7 release

2018-10-01 Thread Dmitriy Setrakyan
Thanks, got it.

On Mon, Oct 1, 2018 at 1:14 PM Dmitriy Pavlov  wrote:

> Here I agree with Vladimir. Furthermore, I do my absolute best to finalize
> all reviews in all 2.7 tickets I'm related to. I think most of the
> contributors doing the same.
>
> пн, 1 окт. 2018 г. в 23:03, Vladimir Ozerov :
>
> > This is precisely the scope we have at the moment. All these tickets were
> > considered carefully on whether to include them into AI 2.7 scope. I
> would
> > say that 10-15% of current tickets may be moved furhter.
> >
> > Third of current tickets are features on their final review stages (e.g.
> > TDE, MVCC invoke, TensorFlow, Thin Clients), another big part is
> > stabilization tickets (mainly - various test failures), and another big
> > part is infrastructure (adopting new modules, Java 9+ support, etc.). So
> > despite big absolute number, most of these tickets are grouped around
> > several big areas, and overall progress over this week should be very
> good.
> >
> > On Mon, Oct 1, 2018 at 9:50 PM Dmitriy Setrakyan 
> > wrote:
> >
> > > If this filter is for 2.7 release, then I do not believe all these
> > tickets
> > > will be closed. It would be nice to leave only "must-have" tickets in
> 2.7
> > > and move the rest to 2.8.
> > >
> > > D.
> > >
> > > On Mon, Oct 1, 2018 at 11:02 AM Vladimir Ozerov 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Please use this filter, as it properly handles tickets without
> > > components:
> > > >
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20not%20in%20(documentation)))%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> > > >
> > > > On Mon, Oct 1, 2018 at 6:18 PM Nikolay Izhikov 
> > > > wrote:
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > I announce scope freeze for an Apache Ignite 2.7 release.
> > > > >
> > > > > It means:
> > > > >
> > > > > 1. We add to 2.7 only critical bugs.
> > > > > 2. We merge to 2.7 branch only previously announces features.
> > > > > 3. I expect we should exclude or *MERGE ALL TASKS FOR 2.7 DUE TO
> > > OCTOBER
> > > > > 10*.
> > > > > So the *October 10 is DEADLINE* for new features.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > For now we have 34 In Progress tickets planned to 2.7 version [1].
> > > > > So is you assigned to any of this ticker friendly reminder #1, *the
> > > > > deadline is near :)*.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IGNITE-8803?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20component%20!%3D%20documentation%20)%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> > > > >
> > > > >
> > > > >
> > > > > В Пн, 01/10/2018 в 16:13 +0300, Andrey Gura пишет:
> > > > > > Agree with Andrey.
> > > > > >
> > > > > > IGNITE-9723 will be merged to ignite-2.7 branch soon.
> > > > > > On Mon, Oct 1, 2018 at 3:56 PM Andrey Kuznetsov <
> stku...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > There is an inaccuracy in critical worker termination
> detection,
> > > and
> > > > > I'm
> > > > > > > working on a fix right now, see [1]. Also, we have trivial yet
> > > > > important
> > > > > > > fix [2], this one is ready to merge.
> > > > > > >
> > > > > > > I deem both should get to 2.7. Any objections?
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9744
> > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-9723
> > > > > > >
> > >

Re: Published articles on Data streaming in Dzone and Tech@Target

2018-10-01 Thread Dmitriy Setrakyan
Thanks, Saikat, great blog!

On Sat, Sep 29, 2018 at 7:13 AM Saikat Maitra 
wrote:

> Hi,
>
> We have published an article on Data streaming at Dzone. Thank you Prachi
> for help for publishing the article
>
>
> https://dzone.com/articles/data-streaming-using-apache-flink-and-apache-ignit
>
> Also wrote an post at Tech@Target about how I started with Apache Ignite
>
>
> https://tech.target.com/2018/09/14/data-streaming-using-apache-flink-and-apache-ignite.html
>
> Please let me know if you have any feedback.
>
> Regards,
> Saikat
>


Re: Apache Ignite 2.7 release

2018-10-01 Thread Dmitriy Setrakyan
> > > >
> > > > > > put
> > > > > > > into project stabilization. This is the sole purpose of that
> > > > >
> > > > > pre-release
> > > > > > > phase. Accepting a patch with deep rework of Ignite internals
> in
> > the
> > > > > >
> > > > > > middle
> > > > > > > of that process, means that our effrots will be lost. This is
> > simply a
> > > > > > > matter of respect to contributor's time.
> > > > > > >
> > > > > > > Another problem is that this false hope puts us at rush. Rush
> > during
> > > > > > > design, development, review, testing. Result is known - bad
> > features,
> > > > > >
> > > > > > which
> > > > > > > makes damage to the project.
> > > > > > >
> > > > > > > So my question is - why don't we just want to move it to AI 2.8
> > right
> > > > > >
> > > > > > now?
> > > > > > > Feature is big, feature is very far from being ready. This
> simple
> > > > >
> > > > > action
> > > > > > > immediately shifts the focus from dates to quality of the
> > product, and
> > > > > > > remove any risks that potential merge will defeat stabilization
> > effrots
> > > > > >
> > > > > > of
> > > > > > > other contributors.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 29, 2018 at 8:32 AM Vyacheslav Daradur <
> > > > >
> > > > > daradu...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dmitriy,
> > > > > > > >
> > > > > > > > Hot redeployment and versioning will not be implemented in
> > phase 1,
> > > > > > > > but it is scheduled once it is finished.
> > > > > > > >
> > > > > > > > Here is an umbrella ticket [1] to track phase 1 scope.
> > > > > > > >
> > > > > > > > It includes very few new features, but we completely rework
> > component
> > > > > > > > to improve guarantees to be more reliable and we build the
> > base for
> > > > > > > > new features.
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > > > > > On Fri, Sep 28, 2018 at 9:38 PM Dmitriy Setrakyan <
> > > > > >
> > > > > > dsetrak...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I am not sure I can agree. SG redesign includes:
> > > > > > > > > - hot redeployment
> > > > > > > > > - versioning
> > > > > > > > >
> > > > > > > > > In my experience, features like these take about 1 month to
> > test
> > > > > >
> > > > > > properly
> > > > > > > > > and fix all the bugs, including redeployment tests and
> > restart
> > > > >
> > > > > tests
> > > > > > on
> > > > > > > > > larger topologies, together with overnight runs. If this
> > type of
> > > > > >
> > > > > > testing
> > > > > > > > > has not been performed, I think it would be unreasonable to
> > expect
> > > > > >
> > > > > > this
> > > > > > > > > feature making it into the release.
> > > > > > > > >
> > > > > > > > > Can someone comment on the testing?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Sep 28, 2018 at 10:38 AM Dmitriy Pavlov <
> > > > > >
> > > > > > dpavlov@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > 

Re: Apache Ignite 2.7 release

2018-09-28 Thread Dmitriy Setrakyan
I am not sure I can agree. SG redesign includes:
- hot redeployment
- versioning

In my experience, features like these take about 1 month to test properly
and fix all the bugs, including redeployment tests and restart tests on
larger topologies, together with overnight runs. If this type of testing
has not been performed, I think it would be unreasonable to expect this
feature making it into the release.

Can someone comment on the testing?

D.


On Fri, Sep 28, 2018 at 10:38 AM Dmitriy Pavlov 
wrote:

> Nikolay, because I think you're a do'er, but not a commenter, like me, for
> example, I can trust your opinion.
>
> I will join review if I have spare cycles.
>
> пт, 28 сент. 2018 г. в 20:34, Denis Magda :
>
> > Nikolay,
> >
> > Thanks for stepping in and giving more context. In general, I'm fully for
> > your proposal below:
> >
> > My vote goes to option *a*.
> > > I think we should release 2.7 with the bunch of new cool features.
> > > *AND* we should plan 2.8 release at the end of the year with SG
> redesign
> > > and MVCC stabilization tasks.
> >
> >
> > --
> > Denis
> >
> > On Fri, Sep 28, 2018 at 10:30 AM Nikolay Izhikov 
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I think we shouldn't put so many emotions in discussion of any
> > > contribution.
> > > Even so big and important as SG redesign.
> > >
> > > The crucial point we all agreed about: Service Grid redesign a big
> > feature
> > > that can significally improve Ignite.
> > > We all want to have it in the product.
> > >
> > > Let me write my vision of the current task state:
> > >
> > > 1. Design of SG is discussed *and aligned* both: privately with Ignite
> > > experts(Vladimir Ozerov, Alexey Goncharyuk, Anton Vinogradov, Denis
> > > Mekhanikov)
> > > and publicly on dev-list. This task is done.
> > >
> > > 2. Current PR state - *on my review*.
> > > I spend some time with this contribution and gave Vyacheslav a
> feedback.
> > > I expect he can fix my comments in a day or two.
> > > Seem we can ask of Anton Vinogradov review on the beginning of next
> week.
> > >
> > > If Dmitriy or any community member wants to help *by doing things, not
> > > discussing them on dev-list*.
> > > Please, join to the review - you are welcome. PR is here [1]
> > >
> > > 3. I think, we have two mutually exclusive options regarding of release
> > > 2.7
> > > a. We release 2.7 in planned dates.
> > > b. We include SG in release scope.
> > >
> > > My vote goes to option *a*.
> > > I think we should release 2.7 with the bunch of new cool features.
> > > *AND* we should plan 2.8 release at the end of the year with SG
> redesign
> > > and MVCC stabilization tasks.
> > >
> > > Anyway, while we preparing release a lot of things can happen.
> > > Let's come back to discussion of SG release version *when it will be
> > ready
> > > to be merged to master*.
> > >
> > > Guys, does it makes sense for you?
> > >
> > > [1] https://github.com/apache/ignite/pull/4434
> > >
> > > В Пт, 28/09/2018 в 19:47 +0300, Dmitriy Pavlov пишет:
> > > > Hi Dmitriy,
> > > >
> > > > The design is aligned totally. The thread you mention was not named
> > > > properly.
> > > >
> > > > It seems to me some community members are trying to take over the
> > > community
> > > > and lead it instead of doing.
> > > >
> > > > As a member of the Apache community, I value Do-ocracy and power of
> > those
> > > > who do, but not just disagree. There are no leaders in open source,
> > just
> > > > do'ers.
> > > >
> > > > By do'ers here I mean Nikolay and Vyacheslav. For me, their
> conclusion
> > > has
> > > > more weight here. If Vladimir is ready to lead an additional release
> > for
> > > SG
> > > > and SG developers agree, it works for me.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > >
> > > > пт, 28 сент. 2018 г. в 19:39, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > We agreed in the beginning of this thread that Service Grid changes
> > > are not
> > > > > going

Re: Apache Ignite 2.7 release

2018-09-28 Thread Dmitriy Setrakyan
Dmitriy,

We agreed in the beginning of this thread that Service Grid changes are not
going to make the release because the community still did not approve the
design. Nothing has changed since. I have not seen any design discussions.
At this point, I have no confidence that the Service Grid changes will make
it into the 2.8 release. The 2.7 release seems out of question altogether.

Also, Nikolay is a release manager. We should let him drive the release. To
my knowledge, he is doing a great job and the release is going according to
the schedule he proposed.

D.

On Fri, Sep 28, 2018 at 4:31 AM Dmitriy Pavlov 
wrote:

> Hi Dmitriy, Vladimir,
>
> I suggest we stop this nonsense with release dates-pushing because of some
> open question.
>
> Just because you disagreed with any include/exclude something into scope,
> does not mean that whole community disagreed.
>
> If folks will start a separate discussion with results of the review, I see
> no reasons to reject their contribution, even if we need to revisit our
> agreements and wait for a couple of days more.
>
> Am I missing some reason why dates are so fundamentally important to you?
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 28 сент. 2018 г. в 12:20, Dmitriy Setrakyan :
>
> > If services is not ready, which it is not, then we should include them
> into
> > the next release. There is no need to force them into 2.7. I suggest we
> > move according to the schedule we all agreed on.
> >
> > D.
> >
> > On Fri, Sep 28, 2018 at 1:22 AM Dmitriy Pavlov 
> > wrote:
> >
> > > Yes, so correct statement is "community did not make any decisions
> about
> > > services not go to 2.7/ services are out of scope".
> > >
> > > If so, please forgive me my confusion.
> > >
> > > пт, 28 сент. 2018 г. в 11:19, Vladimir Ozerov :
> > >
> > > > Exactly. So correct statement is “it is up to *community* to decide
> > > whether
> > > > something goes to 2.7”.
> > > >
> > > > пт, 28 сент. 2018 г. в 11:11, Dmitriy Pavlov  >:
> > > >
> > > > > No, it is up to the community to discuss after their review
> results.
> > > > >
> > > > > пт, 28 сент. 2018 г. в 11:09, Vladimir Ozerov <
> voze...@gridgain.com
> > >:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > Did I read your words correctly that it is up to implementor of a
> > > > single
> > > > > > feature to decide whether release of all other features and fixes
> > to
> > > be
> > > > > > delayed?
> > > > > >
> > > > > > пт, 28 сент. 2018 г. в 11:00, Dmitriy Pavlov <
> > dpavlov@gmail.com
> > > >:
> > > > > >
> > > > > > > My point we can wait a bit for services because
> > > > > > > 1  we are open-minded and we don't have outside pressure to do
> > > > release
> > > > > in
> > > > > > > October
> > > > > > > 2  and services it is not some new feature, which suddenly
> > appeared
> > > > in
> > > > > > > autumn, it is a well known and important feature.
> > > > > > >
> > > > > > > So it is up to Vyacheslav, Anton and Nikolay to decide.
> > > > > > >
> > > > > > > Decisions can be services are not ready/ready to merge only to
> > > > > > master/ready
> > > > > > > to merge to master and to 2.7.
> > > > > > >
> > > > > > >
> > > > > > > пт, 28 сент. 2018 г. в 10:46, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >:
> > > > > > >
> > > > > > > > Dmitry,
> > > > > > > >
> > > > > > > > Community agreement was to perform the release in October. Of
> > > > course
> > > > > we
> > > > > > > can
> > > > > > > > wait a bit for services. Then we wait a bit for other cool
> > > features
> > > > > > ready
> > > > > > > > by that time, then again and again, and release will never
> > > happen.
> > > > > And
> > > > > > > > while we are waiting for new features to come, already
> > completerd
> > > > > > > features
> > > > > > > > cannot be used 

Re: [Discussion] revert of commit MVCC, ignite-9320

2018-09-28 Thread Dmitriy Setrakyan
Guys, let's just fix the tests without reverting commits. Reverting a
commit may trigger a time machine, where all following commits may be
broken because of it. Fixing that scenario will be much harder.

Going forward, I would agree that we should not merge anything that breaks
tests. This is about following a basic engineering discipline. We should
all do it.

D.


On Fri, Sep 28, 2018 at 12:47 AM Dmitriy Pavlov 
wrote:

> Yep, we're humans and we constantly make mistakes. It is a very human thing
> to do mistakes.
>
> So I suggest we will be under the control and protection of robot to avoid
> mistakes, I suggest robot will revert such commits in 72h without its own
> personal attitudes, emotions, etc.
>
> Someone who is interested in contribution usually can find time to make
> contribution perfect.
>
> I'm not aware of project priorities, please share it. I believe different
> priorities can co-exist. A number of contributors are fixing tests, so it
> is a priority for them, isn't it? So why to add work to that guys because
> of you have other priorities?
>
> пт, 28 сент. 2018 г. в 10:39, Vladimir Ozerov :
>
> > Because a lot of other activities depended on configuration in Java, and
> we
> > didn't have expertise to fix .NET immediately.
> >
> > If you want to revert it - please go ahead. But I'd better suggest you to
> > think about the impact and project priorities first, instead of trying to
> > apply the some sort rules blindly. We are not robots.
> >
> > On Fri, Sep 28, 2018 at 10:19 AM Dmitriy Pavlov 
> > wrote:
> >
> > > Hi Vladimir,
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-9320 is named
> configuration
> > > finalization.
> > >
> > > Why finalization was considered as done without tests passing?
> > >
> > > Why can't ve revert finalization change, re-do finalization with
> passing
> > > tests and merge changes?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пт, 28 сент. 2018 г. в 8:16, Vladimir Ozerov :
> > >
> > > > Test is going to be fixed in the scope of AI 2.7 [1]. This is not
> > > > one-minute fix as there are multiple places where configuration
> should
> > be
> > > > passed, and changes should be covered with tests. I muted the test
> for
> > > now.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9390
> > > >
> > > > On Fri, Sep 28, 2018 at 2:40 AM Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Let's not revert any commits yet. Can we find out who did the
> commit
> > > and
> > > > > why he/she is not fixing the test?
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Sep 27, 2018 at 4:21 PM Vyacheslav Daradur <
> > > daradu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Are you talking about
> > > > > > 'IgniteConfigurationParityTest#TestIgniteConfiguration'?
> > > > > >
> > > > > > Seems it's not hard to fix this test, it's necessary just to
> > > implement
> > > > > > missing members (at least as stubs) on .NET side in
> > > > > > IgniteConfiguration class.
> > > > > >
> > > > > > Is there a Jira issue?
> > > > > >
> > > > > > On Fri, Sep 28, 2018 at 2:12 AM Dmitriy Pavlov <
> > > dpavlov@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm grateful for contributions made in that area, but it seems
> > > folks
> > > > > > don't
> > > > > > > have time to fix the test.
> > > > > > >
> > > > > > >
> > > > > > > Tomorrow I'm going to revert commit.
> > > > > > >
> > > > > > > It seems it is the only way we can keep master more or less
> > green.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=1888723=buildChangesDiv=IgniteTests24Java8_PlatformNet
> > > > > > >
> > > > > > >
> > > > > > > Sincerely
> > > > > > > Dmitry Pavlov
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav D.
> > > > > >
> > > > >
> > > >
> > >
> >
>


Re: Apache Ignite 2.7 release

2018-09-28 Thread Dmitriy Setrakyan
If services is not ready, which it is not, then we should include them into
the next release. There is no need to force them into 2.7. I suggest we
move according to the schedule we all agreed on.

D.

On Fri, Sep 28, 2018 at 1:22 AM Dmitriy Pavlov 
wrote:

> Yes, so correct statement is "community did not make any decisions about
> services not go to 2.7/ services are out of scope".
>
> If so, please forgive me my confusion.
>
> пт, 28 сент. 2018 г. в 11:19, Vladimir Ozerov :
>
> > Exactly. So correct statement is “it is up to *community* to decide
> whether
> > something goes to 2.7”.
> >
> > пт, 28 сент. 2018 г. в 11:11, Dmitriy Pavlov :
> >
> > > No, it is up to the community to discuss after their review results.
> > >
> > > пт, 28 сент. 2018 г. в 11:09, Vladimir Ozerov :
> > >
> > > > Dmitriy,
> > > >
> > > > Did I read your words correctly that it is up to implementor of a
> > single
> > > > feature to decide whether release of all other features and fixes to
> be
> > > > delayed?
> > > >
> > > > пт, 28 сент. 2018 г. в 11:00, Dmitriy Pavlov  >:
> > > >
> > > > > My point we can wait a bit for services because
> > > > > 1  we are open-minded and we don't have outside pressure to do
> > release
> > > in
> > > > > October
> > > > > 2  and services it is not some new feature, which suddenly appeared
> > in
> > > > > autumn, it is a well known and important feature.
> > > > >
> > > > > So it is up to Vyacheslav, Anton and Nikolay to decide.
> > > > >
> > > > > Decisions can be services are not ready/ready to merge only to
> > > > master/ready
> > > > > to merge to master and to 2.7.
> > > > >
> > > > >
> > > > > пт, 28 сент. 2018 г. в 10:46, Vladimir Ozerov <
> voze...@gridgain.com
> > >:
> > > > >
> > > > > > Dmitry,
> > > > > >
> > > > > > Community agreement was to perform the release in October. Of
> > course
> > > we
> > > > > can
> > > > > > wait a bit for services. Then we wait a bit for other cool
> features
> > > > ready
> > > > > > by that time, then again and again, and release will never
> happen.
> > > And
> > > > > > while we are waiting for new features to come, already completerd
> > > > > features
> > > > > > cannot be used by anyone.
> > > > > >
> > > > > > This is why we have an agreement that if feature is not ready, it
> > > > should
> > > > > be
> > > > > > moved to future release, instead of shifting release. The sole
> > reason
> > > > to
> > > > > > have strict dates when decisions are made is to let release
> happen.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 28, 2018 at 2:22 AM Dmitriy Pavlov <
> > > dpavlov@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Vladimir,  I'm not searching for enemy, and not fighting with
> > you.
> > > > I'm
> > > > > > not
> > > > > > > happy about cases when we are hurrying.
> > > > > > >
> > > > > > > We can't fix test, fill ticket details, can't wait for
> > > contributions
> > > > to
> > > > > > > finish their tasks.  It is not best idea to use experience from
> > > > > > commercial
> > > > > > > companies in open source. Are there any pressure outside
> > community?
> > > > Did
> > > > > > > someone promised rest of features to be released at 30
> September?
> > > > > > >
> > > > > > > Let's remember principle do-orcracy, power of those who do. If
> > > > > contribor
> > > > > > > does change and reviewer does review, let's give right of
> making
> > > > > decision
> > > > > > > to them, but not to some closed club of people who privately
> > > discuss
> > > > > > > something.
> > > > > > >
> > > > > > > Sincerely
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > чт, 27 сент. 2018 г., 23:42 Vyacheslav Daradur <
> > > daradu...@gmail.com
> > > > >:
> > > > > > >
> > > > > > > > Hi Igniters!
> > > > > > > >
> > > > > > > > As I have written about Service Grid before [1] I'm
> finalizing
> > > the
> > > > > > > > solution to be sure that implementation is reliable.
> > > > > > > >
> > > > > > > > About including it in 2.7, if we talk that code freeze
> tomorrow
> > > > then
> > > > > > > > the solution is not ready to merge yet.
> > > > > > > > I hope that prereviewers Anton Vinogradov and Nikolay Izhikov
> > > will
> > > > be
> > > > > > > > able to answer if solution out of scope or not in a couple of
> > > days.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-7-release-td34076.html#a34485
> > > > > > > > On Thu, Sep 27, 2018 at 11:14 PM Dmitriy Pavlov <
> > > > > dpavlov@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Yes, I agree, NPE during WAL flush is definitely a blocker.
> > > > > > > > >
> > > > > > > > > It is strange why the current test set did not fail after
> > > commit.
> > > > > > > > >
> > > > > > > > > чт, 27 сент. 2018 г. в 21:45, Andrey Kuznetsov <
> > > > stku...@gmail.com
> > > > > >:
> > > > > > > > >
> > > > > > > > > > 

Re: [Discussion] revert of commit MVCC, ignite-9320

2018-09-27 Thread Dmitriy Setrakyan
Let's not revert any commits yet. Can we find out who did the commit and
why he/she is not fixing the test?

D.

On Thu, Sep 27, 2018 at 4:21 PM Vyacheslav Daradur 
wrote:

> Hi,
>
> Are you talking about
> 'IgniteConfigurationParityTest#TestIgniteConfiguration'?
>
> Seems it's not hard to fix this test, it's necessary just to implement
> missing members (at least as stubs) on .NET side in
> IgniteConfiguration class.
>
> Is there a Jira issue?
>
> On Fri, Sep 28, 2018 at 2:12 AM Dmitriy Pavlov 
> wrote:
> >
> > Hi,
> >
> > I'm grateful for contributions made in that area, but it seems folks
> don't
> > have time to fix the test.
> >
> >
> > Tomorrow I'm going to revert commit.
> >
> > It seems it is the only way we can keep master more or less green.
> >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=1888723=buildChangesDiv=IgniteTests24Java8_PlatformNet
> >
> >
> > Sincerely
> > Dmitry Pavlov
>
>
>
> --
> Best Regards, Vyacheslav D.
>


Ignite and Spark on Yarn

2018-09-26 Thread Dmitriy Setrakyan
Can anyone familiar with the subject respond here?

https://stackoverflow.com/questions/52488073/ignite-yarn-and-hortonworks

D.


Re: [ML] New features and improvement of ML module for 2.7 release

2018-09-26 Thread Dmitriy Setrakyan
Great stuff! Would be nice to see a series of blogs outlining Ignite ML
capabilities.This would give us a better momentum.

D.

On Wed, Sep 26, 2018 at 10:31 AM Юрий Бабак  wrote:

> Hello Igniters,
>
> I want to make up some overview of all features and major improvement of ML
> module for this release.
>
> So let me start from the one of our main feature for this release:
>
> *TensorFlow integration* <
> https://issues.apache.org/jira/browse/IGNITE-8670>
>
> This integration allows us to use Apache Ignite as a data source for
> TensorFlow. Also, this integration will allow creating and maintaining
> TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters. More details in the related umbrella ticket.
>
> Also, for this release we have some new algorithms:
>
> * Random forest  
> * Gradient boosted trees <
> https://issues.apache.org/jira/browse/IGNITE-7149>
> * Logistic regression[binary
> ][multi-class
> ]
> * ANN 
>
> New features related with data preprocessing:
>
> * Pipeline 
> * L1,L2 normalization 
> * Data filtering for new datasets
> 
> * Encoding categorical features [OneHotEncoder
> ][OneOfKEncoder
> ]
> * Imputer and Binarizer  >
> * MaxAbsScaler 
> * Dataset splitting 
>
> New features for a model validation:
>
> * Model estimator 
> * k-fold cross-validation
> 
> * Param grid for tuning hyper-parameters in cross-validation
> 
>
> Other features and improvements:
>
> * Model updating 
> * ML tutorial 
> * Optional indexing for decision trees
> 
> * Learning context for trainers(local parallelizing and logging of training
> process) 
> * Unification of API for feature extractor
> 
> * Several tickets for removing old unused classes and improvements for code
> coverage and examples [1 <
> https://issues.apache.org/jira/browse/IGNITE-9124>
> ][2 ][3
> ][4
> ][5
> ][6
> ]
>
> Sincerely,
> Yuriy Babak
>


Re: [ML] TensorFlow intergration module release

2018-09-25 Thread Dmitriy Setrakyan
Distributed TensorFlow, awesome!

On Tue, Sep 25, 2018 at 12:53 PM Юрий Бабак  wrote:

> Hello, Igniters.
>
> For release 2.7 we will introduce integration between TensorFlow and Apache
> Ignite. This integration contains changes on Apache Ignite side and on
> TensorFlow side.
>
> Apache Ignite part is the command line tool which allows create and
> maintain TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters.
>
> For TensorFlow we implemented "ignite dataset". More details in related PR
> [1]
>
> As Apache Ignite part is done and TensorFlow part is ready for the merge I
> suggest to add module "ignite-tensorflow" to other Ignite deliverables. So
> I've created the ticket in JIRA for this [2]. In that case, we will be able
> to release this feature with Apache Ignite binary release includes deb/rpm
> packages.
>
> [1] https://github.com/tensorflow/tensorflow/pull/22210
> [2] https://issues.apache.org/jira/browse/IGNITE-9685
>
> Regards,
> Yury
>


Re: [DISCUSSION] Add reviewer field to Apache Ignite JIRA project

2018-09-25 Thread Dmitriy Setrakyan
I like the idea.

On Tue, Sep 25, 2018 at 8:25 AM Dmitriy Pavlov 
wrote:

> Hi Ignite Enthusiasts,
>
> During the planning of release 2.7, I've faced with the situation when it
> is completely not clear who is going to review ticket.
>
> Usually, we do not reassign tickets to a reviewer, but info about planned
> reviewer can be very useful for all reviewers, who select some contribution
> to pick up into a review.
>
> Please share your vision about the idea of adding a reviewer field (type:
> user) in addition to the assignee field.
>
> If we agree I will try to ask the Infra team on Friday 28.09.
>
> Sincerely,
> Dmitriy Pavlov
>


Re: The future of Affinity / Topology concepts and possible PME optimizations.

2018-09-18 Thread Dmitriy Setrakyan
Ilya, I would suggest that you discuss everything here on the dev@ list,
including suggestions how to split the work.

D.

On Tue, Sep 18, 2018 at 6:34 PM Ilya Lantukh  wrote:

> Thanks for the feedback!
>
> I agree that we should start with the simplest optimizations, but it seems
> that decoupling affinity/topology versions is necessary before we can make
> any progress here, and this is a rather complex change all over the code.
>
> If anyone wants to help, please contact me privately and we will discuss
> how this work can be split.
>
> Denis Magda, do you think we should create IEP for these optimizations?
>
> On Tue, Sep 18, 2018 at 5:59 PM, Maxim Muzafarov 
> wrote:
>
> > Ilya,
> >
> >
> > > 3. Start node in baseline: both affinity and topology versions should
> be
> > incremented, but it might be possible to optimize PME for such case and
> > avoid cluster-wide freeze. Partition assignments for such node are
> already
> > calculated, so we can simply put them all into MOVING state. However, it
> > might take significant effort to avoid race conditions and redesign our
> > architecture.
> >
> > As you mentioned all assignments are already calculated. So as another
> > suggestion,
> > we can introduce a new `intermediate` state of such joined nodes. Being
> in
> > this state
> > node recovers all data from their local storage, preloads whole missed
> > partition
> > data from the cluster (probably on some point in time), creates and
> > preloads missed
> > in-memory and persistent caches. And only after these recovery such node
> > will fire
> > discovery join event and affinity\topology version may be incremented. I
> > think this
> > approach can help to reduce the further rebalance time.
> > WDYT?
> >
> >
> >
> > On Tue, 18 Sep 2018 at 16:31 Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> > wrote:
> >
> > > Ilya,
> > >
> > > This is a great idea, but before we can ultimately decouple the
> affinity
> > > version from the topology version, we need to fix a few things with
> > > baseline topology first. Currently the in-memory caches are not using
> the
> > > baseline topology. We are going to fix this as a part of IEP-4 Phase II
> > > (baseline auto-adjust). Once fixed, we can safely assume that
> > > out-of-baseline node does not affect affinity distribution.
> > >
> > > Agree with Dmitriy that we should start with simpler optimizations
> first.
> > >
> > > чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh :
> > >
> > > > Igniters,
> > > >
> > > > As most of you know, Ignite has a concept of AffinityTopologyVersion,
> > > which
> > > > is associated with nodes that are currently present in topology and a
> > > > global cluster state (active/inactive, baseline topology, started
> > > caches).
> > > > Modification of either of them involves process called Partition Map
> > > > Exchange (PME) and results in new AffinityTopologyVersion. At that
> > moment
> > > > all new cache and compute grid operations are globally "frozen". This
> > > might
> > > > lead to indeterminate cache downtimes.
> > > >
> > > > However, our recent changes (esp. introduction of Baseline Topology)
> > > caused
> > > > me to re-think those concept. Currently there are many cases when we
> > > > trigger PME, but it isn't necessary. For example, adding/removing
> > client
> > > > node or server node not in BLT should never cause partition map
> > > > modifications. Those events modify the *topology*, but *affinity* in
> > > > unaffected. On the other hand, there are events that affect only
> > > *affinity*
> > > > - most straightforward example is CacheAffinityChange event, which is
> > > > triggered after rebalance is finished to assign new primary/backup
> > nodes.
> > > > So the term *AffinityTopologyVersion* now looks weird - it tries to
> > > "merge"
> > > > two entities that aren't always related. To me it makes sense to
> > > introduce
> > > > separate *AffinityVersion *and *TopologyVersion*, review all events
> > that
> > > > currently modify AffinityTopologyVersion and split them into 3
> > > categories:
> > > > those that modify only AffinityVersion, only TopologyVersion and
> both.
> > It
> > > > will allow us to process such events using different mechanics and
> > avoid
> > > > redundant steps, and also reconsider mapping of operations - some
> will
> > be
> > > > mapped to topology, others - to affinity.
> > > >
> > > > Here is my view about how different event types theoretically can be
> > > > optimized:
> > > > 1. Client node start / stop: as stated above, no PME is needed,
> ticket
> > > > https://issues.apache.org/jira/browse/IGNITE-9558 is already in
> > > progress.
> > > > 2. Server node start / stop not from baseline: should be similar to
> the
> > > > previous case, since nodes outside of baseline cannot be partition
> > > owners.
> > > > 3. Start node in baseline: both affinity and topology versions should
> > be
> > > > incremented, but it might be possible to optimize PME for such case
> and
> > > > avoid 

Re: Cache scan efficiency

2018-09-18 Thread Dmitriy Setrakyan
On Tue, Sep 18, 2018 at 1:58 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Summing up, I suggest adding new public
> method IgniteCache.preloadPartition(partId).
>
> I will start preparing PR for IGNITE-8873
>  if no more objections
> follow.
>


Alexey, let's make sure we document this feature very well in Javadoc, as
well as in public readme.io documentation. Also, all cache iterator methods
and SCAN queries should be documented, suggesting when the partitions
should be preloaded to achieve better performance.

D.


Re: Apache Ignite 2.7 release

2018-09-18 Thread Dmitriy Setrakyan
If it is an Ignite release, then it has to pass through the vote. If not,
then you can do the test without publishing or uploading the release.

D.

On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov  wrote:

> Ok.
>
> In case of TC questions — ask me.
>
>
>
> > On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> >
> > Hello, Petr.
> >
> > I want to make ignite-2.7 branch today.
> > And execute release procedure based on this branch.
> >
> > However, ignite-2.7 branch will be copy of master until code freeze date.
> >
> > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> >> Will it be just a test or there is already ignite-2.7 branch?
> >>
> >> Fabric removal related TC modifications are not ready yet, and code is
> not in master.
> >>
> >>
> >>
> >>> On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
> >>>
> >>> Hello, Igniters.
> >>>
> >>> I want to start and release procedures and make an RC1 build.
> >>>
> >>> It has a 2 intention:
> >>>
> >>> 1. I want to walk through all release steps to make sure they all
> works for me.
> >>> So I will be fully ready on release date.
> >>>
> >>> 2. We have updated some dependencies in 2.7 and we need to make sure
> binary build is still workable.
> >>>
> >>> Any objections?
> >>>
> >>>
> >>> В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
>  We already have all the mechanics in place to work with properties -
> we use
>  ignite.build and ignite.revision from ignite.properties which are
> adjusted
>  during the build in the binary package.
> 
>  Should I create the ticket if there are no objections?
> 
>  пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>:
> 
> > Hello!
> >
> > So now there's an issue that this script makes source change after
> every
> > build, show up in git status.
> >
> > What we could do to it:
> > - Commit the changes after the build, once. In hopes that it won't
> change
> > very often. With benefit that we could do that right now, before the
> code
> > freeze.
> > - Move these values to a properties file from both pom.xml and
> > IgniteProvider.java. Any problems with this approach? We'll just
> read them
> > from classpath properties file.
> > - Update the links in the file once and remove them from build
> process. Why
> > were they added to build process in the first place - to make them
> > configurable during build?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> >
> >> Ilya,
> >>
> >> The "latest" version is the default, and resolved by
> >> https://ignite.apache.org/latest which is used by our web site
> when a
> >> user download the latest Ignite version. And I think this is the
> >
> > authority
> >> to judge of the latest official release (pom.xml you suggest can
> have
> >> SNAPSHOTs etc.).
> >> Also, as I explained during our review sessions, ignite-mesos-2.6.0
> is a
> >> driver and doesn't mean you need to have Ignite 2.6.0. User can run
> any
> >> version of Ignite he/she specifies. By default, it's "latest" but a
> user
> >> can specify any version needed, even from a non-archive URL.
> >>
> >> In short, what we have now
> >> 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by
> default
> >> -> it will try to resolve the latest officially releases version of
> >
> > Apache
> >> Ignite, find the closest mirror and download Ignite in a minute. If
> the
> >> version resolution fails, we fall back to the slow apache archive
> (as you
> >> suggest; in my opinion we better fail-fast instead of waiting for
> hours
> >
> > to
> >> download, so the user can choose another download option (3))
> >> 2. If the user specifies the version explicitly, it goes to the slow
> >> apache archive.
> >> 3. The user can put ignite zip file on his/her http server and
> provide
> >
> > the
> >> URL as a parameter to the driver, if options 1 and 2 don't work.
> >>
> >> As you see, there are 3 options. And I just fix the 1st one with
> >> https://issues.apache.org/jira/browse/IGNITE-9388 and don't change
> the
> >> original logic (which I find reasonable) documented on our site -- I
> >
> > don't
> >> see how it blocks anything.
> >>
> >> Roman Shtykh
> >>
> >>
> >> On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
> >> ilya.kasnach...@gmail.com> wrote:
> >>
> >>
> >> Hello!
> >>
> >> There's still two issues with the submission.
> >>
> >> The first one is that we're downloading "latest" version from
> preferred
> >> mirror but a specified version, such as "2.6", we're also going to
> >
> > download
> >> from "slow" archive.apache.org/dist.
> >> That's a great limitation for this change, since 

Re: Cache scan efficiency

2018-09-17 Thread Dmitriy Setrakyan
I would rather fix the scan than hack the scan. Is there any technical
reason for hacking it now instead of fixing it properly? Can some of the
experts in this thread provide an estimate of complexity and difference in
work that would be required for each approach?

D.

On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk 
wrote:

> I think it would be beneficial for some Ignite users if we added such a
> partition warmup method to the public API. The method should be
> well-documented and state that it may invalidate existing page cache. It
> will be a very effective instrument until we add the proper scan ability
> that Vladimir was referring to.
>
> пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :
>
> > Folks,
> >
> > Such warming up can be an effective technique for performing calculations
> > which required large cache
> > data reads, but I think it's the single narrow use case of all over
> Ignite
> > store usages. Like all other
> > powerfull techniques, we should use it wisely. In the general case, I
> think
> > we should consider other
> > techniques mentioned by Vladimir and may create something like `global
> > statistics of cache data usage`
> > to choose the best technique in each case.
> >
> > For instance, it's not obvious what would take longer: multi-block reads
> or
> > 50 single-block reads issues
> > sequentially. It strongly depends on used hardware under the hood and
> might
> > depend on workload system
> > resources (CPU-intensive calculations and I\O access) as well. But
> > `statistics` will help us to choose
> > the right way.
> >
> >
> > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov 
> wrote:
> >
> > > Hi Alexei,
> > >
> > > I did not find any PRs associated with the ticket for check code
> changes
> > > behind this idea. Are there any PRs?
> > >
> > > If we create some forwards scan of pages, it should be a very
> > intellectual
> > > algorithm including a lot of parameters (how much RAM is free, how
> > probably
> > > we will need next page, etc). We had the private talk about such idea
> > some
> > > time ago.
> > >
> > > By my experience, Linux systems already do such forward reading of file
> > > data (for corresponding sequential flagged file descriptors), but some
> > > prefetching of data at the level of application may be useful for
> > O_DIRECT
> > > file descriptors.
> > >
> > > And one more concern from me is about selecting a right place in the
> > system
> > > to do such prefetch.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov :
> > >
> > > > HI Alex,
> > > >
> > > > This is good that you observed speedup. But I do not think this
> > solution
> > > > works for the product in general case. Amount of RAM is limited, and
> > > even a
> > > > single partition may need more space than RAM available. Moving a lot
> > of
> > > > pages to page memory for scan means that you evict a lot of other
> > pages,
> > > > what will ultimately lead to bad performance of subsequent queries
> and
> > > > defeat LRU algorithms, which are of great improtance for good
> database
> > > > performance.
> > > >
> > > > Database vendors choose another approach - skip BTrees, iterate
> > direclty
> > > > over data pages, read them in multi-block fashion, use separate scan
> > > buffer
> > > > to avoid excessive evictions of other hot pages. Corresponding ticket
> > for
> > > > SQL exists [1], but idea is common for all parts of the system,
> > requiring
> > > > scans.
> > > >
> > > > As far as proposed solution, it might be good idea to add special API
> > to
> > > > "warmup" partition with clear explanation of pros (fast scan after
> > > warmup)
> > > > and cons (slowdown of any other operations). But I think we should
> not
> > > make
> > > > this approach part of normal scans.
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6057
> > > >
> > > >
> > > > On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
> > > > alexey.scherbak...@gmail.com> wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > My use case involves scenario where it's necessary to iterate over
> > > > > large(many TBs) persistent cache doing some calculation on read
> data.
> > > > >
> > > > > The basic solution is to iterate cache using ScanQuery.
> > > > >
> > > > > This turns out to be slow because iteration over cache involves a
> lot
> > > of
> > > > > random disk access for reading data pages referenced from leaf
> pages
> > by
> > > > > links.
> > > > >
> > > > > This is especially true when data is stored on disks with slow
> random
> > > > > access, like SAS disks. In my case on modern SAS disks array
> reading
> > > > speed
> > > > > was like several MB/sec while sequential read speed in perf test
> was
> > > > about
> > > > > GB/sec.
> > > > >
> > > > > I was able to fix the issue by using ScanQuery with explicit
> > partition
> > > > set
> > > > > and running simple warmup code before each partition scan.
> > > > >
> > > 

Re: Using materialised views for queries with joins

2018-09-17 Thread Dmitriy Setrakyan
Hi Sven,

I will let others comment on the materialized view suggestion, but I have
another question.

*As we all know, joins are a nightmare in a distributed system*


Have you considered collocation or replication? If a table is replicated,
then it will be present on all the nodes and all joins will be fast. If two
partitioned tables are colocated based on some affinity key, then joins on
that affinity key will be fast as well.

Both, colocation and replication are supported by Ignite. Will any of these
approaches work for you?

D.

On Mon, Sep 17, 2018 at 11:04 AM Sven Beauprez 
wrote:

> All,
>
>
>
> We are in a situation where we have to query data over several caches. As
> we all know, joins are a nightmare in a distributed system and I know there
> are other means like denormalisation, but it is not sufficient anymore in
> some cases we have and we need the joins.
>
>
>
> We mainly work in an OLTP context, where queries are known in advance (ie
> dev time) and inpsired by following blog of several years ago, I was
> wondering if the concept of “materialized views” could make it into Apache
> Ignite.
>
> (
> https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
> )
>
>
>
> It would work as follows:
>
>- A query must register itself in Ignite at startup time (eg. via
>configuration) or during run time (eg. API call)
>- The registered query is parsed and a new “view” cache is created
>which will ‘cache’ the resultset of the query (could take a while, but
>intermediate status can be “warming up” and “hot” when ready)
>- All caches involved in the joins are now monitored for CUD
>operations and relevant data is stored in the new “view” cache so the view
>gets updated in real time
>- All operations must be ACID compliant
>- The view is queried via a very trivial select statement
>
>
>
> Would that be feasible as a new feature?
>
>
>
>
>
> Regards,
>
>
>
> Sven
>
>
>
>
>
>
>
> [image: cid:image001.png@01D3007B.4D007790]
>
>
>
> SVEN BEAUPREZ
>
>
>
> L e a d   A r c h i t e c t
>
>
>
> De Kleetlaan 5, B-1831 Diegem
>
> www.theglue.com
>


Re: Cache scan efficiency

2018-09-16 Thread Dmitriy Setrakyan
Alexey, this is a great feature. Can you explain what you meant by
"warm-up" when iterating through pages? Do you have this feature already
implemented?

D.

On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Igniters,
>
> My use case involves scenario where it's necessary to iterate over
> large(many TBs) persistent cache doing some calculation on read data.
>
> The basic solution is to iterate cache using ScanQuery.
>
> This turns out to be slow because iteration over cache involves a lot of
> random disk access for reading data pages referenced from leaf pages by
> links.
>
> This is especially true when data is stored on disks with slow random
> access, like SAS disks. In my case on modern SAS disks array reading speed
> was like several MB/sec while sequential read speed in perf test was about
> GB/sec.
>
> I was able to fix the issue by using ScanQuery with explicit partition set
> and running simple warmup code before each partition scan.
>
> The code pins cold pages in memory in sequential order thus eliminating
> random disk access. Speedup was like x100 magnitude.
>
> I suggest adding the improvement to the product's core  by always
> sequentially preloading pages for all internal partition iterations (cache
> iterators, scan queries, sql queries with scan plan) if partition is cold
> (low number of pinned pages).
>
> This also should speed up rebalancing from cold partitions.
>
> Ignite JIRA ticket [1]
>
> Thoughts ?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8873
>
> --
>
> Best regards,
> Alexei Scherbakov
>


Re: The future of Affinity / Topology concepts and possible PME optimizations.

2018-09-13 Thread Dmitriy Setrakyan
Ilya,

Thanks for the detailed explanation. Everything you suggested makes sense.
Needless to say, PME effect on the grid should be minimized. Let's start
with the simpler and less risky changes first.

D.

On Thu, Sep 13, 2018 at 5:58 AM Ilya Lantukh  wrote:

> Igniters,
>
> As most of you know, Ignite has a concept of AffinityTopologyVersion, which
> is associated with nodes that are currently present in topology and a
> global cluster state (active/inactive, baseline topology, started caches).
> Modification of either of them involves process called Partition Map
> Exchange (PME) and results in new AffinityTopologyVersion. At that moment
> all new cache and compute grid operations are globally "frozen". This might
> lead to indeterminate cache downtimes.
>
> However, our recent changes (esp. introduction of Baseline Topology) caused
> me to re-think those concept. Currently there are many cases when we
> trigger PME, but it isn't necessary. For example, adding/removing client
> node or server node not in BLT should never cause partition map
> modifications. Those events modify the *topology*, but *affinity* in
> unaffected. On the other hand, there are events that affect only *affinity*
> - most straightforward example is CacheAffinityChange event, which is
> triggered after rebalance is finished to assign new primary/backup nodes.
> So the term *AffinityTopologyVersion* now looks weird - it tries to "merge"
> two entities that aren't always related. To me it makes sense to introduce
> separate *AffinityVersion *and *TopologyVersion*, review all events that
> currently modify AffinityTopologyVersion and split them into 3 categories:
> those that modify only AffinityVersion, only TopologyVersion and both. It
> will allow us to process such events using different mechanics and avoid
> redundant steps, and also reconsider mapping of operations - some will be
> mapped to topology, others - to affinity.
>
> Here is my view about how different event types theoretically can be
> optimized:
> 1. Client node start / stop: as stated above, no PME is needed, ticket
> https://issues.apache.org/jira/browse/IGNITE-9558 is already in progress.
> 2. Server node start / stop not from baseline: should be similar to the
> previous case, since nodes outside of baseline cannot be partition owners.
> 3. Start node in baseline: both affinity and topology versions should be
> incremented, but it might be possible to optimize PME for such case and
> avoid cluster-wide freeze. Partition assignments for such node are already
> calculated, so we can simply put them all into MOVING state. However, it
> might take significant effort to avoid race conditions and redesign our
> architecture.
> 4. Cache start / stop: starting or stopping one cache doesn't modify
> partition maps for other caches. It should be possible to change this
> procedure to skip PME and perform all necessary actions (compute affinity,
> start/stop cache contexts on each node) in background, but it looks like a
> very complex modification too.
> 5. Rebalance finish: it seems possible to design a "lightweight" PME for
> this case as well. If there were no node failures (and if there were, PME
> should be triggered and rebalance should be cancelled anyways) all
> partition states are already known by coordinator. Furthermore, no new
> MOVING or OWNING node for any partition is introduced, so all previous
> mappings should still be valid.
>
> For the latter complex cases in might be necessary to introduce "is
> compatible" relationship between affinity versions. Operation needs to be
> remapped only if new version isn't compatible with the previous one.
>
> Please share your thoughts.
>
> --
> Best regards,
> Ilya
>


Re: Compression prototype

2018-09-04 Thread Dmitriy Setrakyan
In my view, dictionary of 1024 bytes is not going to be nearly enough.

On Tue, Sep 4, 2018 at 8:06 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> In case of Apache Ignite, most of savings is due to BinaryObject format,
> which encodes types and fields with byte sequences. Any enum/string flags
> will also get in dictionary. And then as it processes a record it fills up
> its individual dictionary.
>
> But, in one cache, most if not all entries have identical BinaryObject
> layout so a tiny dictionary covers that case. Compression algorithms are
> not very keen on large dictionaries, preferring to work with local
> regularities in byte stream.
>
> E.g. if we have large entries in cache with low BinaryObject overhead,
> they're served just fine by "generic" compression.
>
> All of the above is my speculations, actually. I just observe that on a
> large data set, compression ratio is around 0.4 (2.5x) with a dictionary of
> 1024 bytes. The rest is black box.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 4 сент. 2018 г. в 17:16, Dmitriy Setrakyan :
>
> > On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > Each node has a local dictionary (per node currently, per cache
> planned).
> > > Dictionary is never shared between nodes. As data patterns shift,
> > > dictionary rotation is also planned.
> > >
> > > With Zstd, the best dictionary size seems to be 1024 bytes. I imagine
> It
> > is
> > > enough to store common BinaryObject boilerplate, and everything else is
> > > compressed on the fly. The source sample is 16k records.
> > >
> > >
> > Thanks, Ilya, understood. I think per-cache is a better idea. However, I
> > have a question about dictionary size. Ignite stores TBs of data. How do
> > you plan the dictionary to fit in 1K bytes?
> >
> > D.
> >
>


Re: Python thin client

2018-09-04 Thread Dmitriy Setrakyan
Got it, sounds good!

On Tue, Sep 4, 2018 at 10:54 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Dmitriy,
>
> It would be quite messy to implement with Python modular system.
>
> First of all, Python 2 and Python 3 are different languages with a small
> common subset of syntax rules. That's why what we see in a stack trace is a
> syntax error, and not a “missing feature” error.
>
> Second, there is no single entry point in Python code. User is allowed to
> import any name, from any module, in any order. In fact the module is run
> when it first discovered by CPython during any `import` operation, and that
> is how the imported entities are created and initialized: by the chain of
> imports.
>
> It means for us, that implementing even a simple compatibility message in
> Python 2 requires any module in our program (or at least all the modules,
> that represent the public API of our library) to consist entirely of a
> valid Python 2 code.
>
> It can be achieved by writing stubs with a version check, putting said
> stubs in place of real modules, and proxying all the calls through the
> conditional imports. It would take a small effort once, but make code less
> readable and harder to maintain for the rest of its life cycle. Should we,
> for example, provide two testing environments − for the main program and
> for Python 2-compatible stubs?
>
> As far as I know, no Python developer is making such efforts nowadays.
> There are some projects with a long history, that achieve 2/3-compatibility
> through the use of restricted syntax and external packages like `six`, or
> simply support two separate versions. Most of the new projects are creating
> with the latest Python 3, pip and virtualenv in mind.
>
> I took the idea of my `setup.py` solution from the Django project, which
> is dropped Python 2 support not long ago. Its setup relies on `setuptools`
> 9+ option, or otherwise displays a simple banner; the banner is likely to
> be buried under the further cryptic output of old setuptools, but it is
> better than nothing.
>
>
> On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote:
>
>> Dmitriy,
>>
>> setuptools sounds like an installation step. Does it make sense to add a
>> check during startup of a client?
>>
>> D.
>>
>> On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
>> dmitry.melnic...@nobitlost.com> wrote:
>>
>> Nikolay,
>>>
>>> There is indeed a feature in `setuptools` I just learned about, which
>>> would help in this case (and I believe the case you demonstrated can be
>>> quite typical, thank you for pointing it out). It gives user a clever
>>> message in the end of a stack trace like
>>>
>>> UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
>>>> running Python is 2.7.15
>>>>
>>>
>>> This feature is not 100% bullet-proof, and it will not help users who
>>> will
>>> try to install my client other way than with `pip` or `setup.py`.
>>> It will also be less helpful with old versions of `pip` (before 9.0).
>>> However, it should cover most situations.
>>>
>>> On 9/4/18 7:15 PM, Nikolay Izhikov wrote:
>>>
>>> Hello, Dmitry.
>>>>
>>>> I understand that for experienced Python developer it obvious from
>>>> stack trace I send.
>>>>
>>>> But can we check python version on startup? And print big fat error
>>>> message "You are using wrong python version".
>>>>
>>>>
>>>>
>>>  From my experience, there are some tickets in Ignite that should be
>>>> implemented in various thin clients. It a very trivial changes, but
>>>> lack of testability makes this task harder then steel.
>>>>
>>>> I think a .Net DEVNOTES are very good example.
>>>>
>>>> Please, be gentle with your fellow contributors and make DEVNOTES as
>>>> clear as possible.
>>>>
>>>>
>>>
>>
>


Re: Hi... trying to write a test case for a pacth ... need help....

2018-09-04 Thread Dmitriy Setrakyan
Hi Paul, what Jira ticket are you working on?

On Tue, Sep 4, 2018 at 7:02 AM, Paul Anderson  wrote:

> ... The code is done ready to go, but the test suite looks huge, I know the
> two test classes I have to augment... but I am unsure of a couple of
> things.
>
> 1) how do I run a specific test (command line)
> 2) how do I start (make sure is started) an ignite instance
>


Re: Python thin client

2018-09-04 Thread Dmitriy Setrakyan
Dmitriy,

setuptools sounds like an installation step. Does it make sense to add a
check during startup of a client?

D.

On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Nikolay,
>
> There is indeed a feature in `setuptools` I just learned about, which
> would help in this case (and I believe the case you demonstrated can be
> quite typical, thank you for pointing it out). It gives user a clever
> message in the end of a stack trace like
>
> > UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
> > running Python is 2.7.15
>
> This feature is not 100% bullet-proof, and it will not help users who will
> try to install my client other way than with `pip` or `setup.py`.
> It will also be less helpful with old versions of `pip` (before 9.0).
> However, it should cover most situations.
>
> On 9/4/18 7:15 PM, Nikolay Izhikov wrote:
>
>> Hello, Dmitry.
>>
>> I understand that for experienced Python developer it obvious from
>> stack trace I send.
>>
>> But can we check python version on startup? And print big fat error
>> message "You are using wrong python version".
>>
> >
>
>> From my experience, there are some tickets in Ignite that should be
>> implemented in various thin clients. It a very trivial changes, but
>> lack of testability makes this task harder then steel.
>>
>> I think a .Net DEVNOTES are very good example.
>>
>> Please, be gentle with your fellow contributors and make DEVNOTES as
>> clear as possible.
>>
>


Re: Compression prototype

2018-09-04 Thread Dmitriy Setrakyan
On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> Each node has a local dictionary (per node currently, per cache planned).
> Dictionary is never shared between nodes. As data patterns shift,
> dictionary rotation is also planned.
>
> With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It is
> enough to store common BinaryObject boilerplate, and everything else is
> compressed on the fly. The source sample is 16k records.
>
>
Thanks, Ilya, understood. I think per-cache is a better idea. However, I
have a question about dictionary size. Ignite stores TBs of data. How do
you plan the dictionary to fit in 1K bytes?

D.


Re: Compression prototype

2018-09-04 Thread Dmitriy Setrakyan
On Tue, Sep 4, 2018 at 1:16 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> The compression is per-binary-object, but dictionary is external, shared
> between multiple (millions of) entries and stored alongside compressed
> data.
>

I was under a different impression. If the dictionary is for the whole data
set, then it will occupy megabytes (if not gigabytes) of data. What happens
when a new node joins and has no idea about the dictionary? What happens
when dictionary between nodes get out-of-sync?

D.


Re: Compression prototype

2018-09-03 Thread Dmitriy Setrakyan
Hi Ilya,

This is very useful. Is the compression going to be per-page, in which case
the dictionary is going to be kept inside of a page? Or do you have some
other design in mind?

D.

On Mon, Sep 3, 2018 at 10:36 AM, Ilya Kasnacheev 
wrote:

> Hello again!
>
> I've been running various compression parameters through cod dataset.
>
> It looks like the best compression level in terms of speed is either 1 or
> 2.
> The default for Zstd seems to be 3 which would almost always perform worse.
> For best performance a dictionary of 1024 is optimal, for better
> compression
> one might choose larger dictionaries, 6k looks good but I will also run a
> few benchmarks on larger dicts. Unfortunately, Zstd crashes if sample size
> is set to more than 16k entries (I guess I should probe the max buffer size
> where problems begin).
>
> I'm attaching two charts which show what's we've got. Compression rate is a
> fraction of original records size. Time to run is wall clock time the test
> run. Reasonable compression will increase the run time twofold (of a
> program
> that only does text record parsing -> creates objects -> binarylizes them
> ->
> compresses -> decompresses). Notation: s{number of bin objects used to
> train}-d{dictionary length in bytes}-l{compression level}.
>  com/file/t374/chart1.png>
> Second one is basically a zoom in on the first.
>  com/file/t374/chart2.png>
> I think that in additional to dictionary compression we should have
> dictionary-less compression. On typical data of small records it shows
> compression rate of 0.8 ~ 0.65, but I can imagine that with larger
> unstructured records it can be as good as dict-based and much less of a
> hassle dictionary-processing-wise. WDYT?
> Sorry for the fine prints. I hope my charts will visible.
>
> You can see the updated code as pull request:
> https://github.com/apache/ignite/pull/4673
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


Re: Service Grid new design overview

2018-08-30 Thread Dmitriy Setrakyan
I also hope that we have some batching API to allow deployment of multiple
services together, either on grid startup or during the call to "deploy..."
API.

D.

On Thu, Aug 30, 2018 at 5:04 AM, Vyacheslav Daradur 
wrote:

> Hi Igniters!
>
> I had a private talk about new Service Grid design with: Alexey
> Goncharuk, Vladimir Ozerov, Denis Mekhanikov, Nikolay Izhikov, Anton
> Vinogradov and I'd like to share results.
>
> Design looks good in general, but we have decided to improve the
> unified flow of requests processing as follows - when any node
> received request on deploying, a node should calculate assignments
> yourself and deploys it if needed, then sends the result to the
> coordinator instead of waiting for assignments from the coordinator.
>
> For this change, we should make our service's assignments function
> *determined*, that means the function will return the same results for
> the same arguments at any node.
>
> We all agreed with this change because it allows us to reduce messages
> for handling each request and making the solution more flexible.
>
> On Tue, Aug 28, 2018 at 12:26 AM Dmitriy Setrakyan
>  wrote:
> >
> > Agree with Val. I think all users would expect that a service is
> restarted
> > upon a node or cluster restart. Let's make sure we preserve this
> behavior.
> >
> > D.
> >
> > On Fri, Aug 24, 2018 at 4:17 PM, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Guys,
> > >
> > > I believe we should preserve the behavior that we have now. What
> happens to
> > > services if we restart a persistent cluster running 2.6? Are services
> > > recreated or not? If YES, we should make sure the same happens after
> > > redesign. Would be even better if we preserve compatibility, i.e. allow
> > > seamless upgrade from older version that uses system cache to newer
> version
> > > that uses disco messages for service deployment. If NO, it's much
> easier
> > > and we can leave it as is for now. However, eventually would be great
> to
> > > have an option to persist services and redeploy them after cluster
> restart.
> > >
> > > -Val
> > >
> > > On Fri, Aug 24, 2018 at 2:51 PM Dmitriy Pavlov 
> > > wrote:
> > >
> > > > Denis M. & Val please share your vision about this topic.
> > > >
> > > > пт, 24 авг. 2018 г. в 15:52, Vyacheslav Daradur  >:
> > > >
> > > > > Nick, Antron, thank you for stepping in.
> > > > >
> > > > > AFAIK, Ignite cluster can move its state to a new version of Ignite
> > > > > using persistence only.
> > > > >
> > > > > Since Ignite v.2.3 persistence is configured on a memory region and
> > > > > system memory region is not persistence, that means the system
> > > > > (utility) cache will not be recovered on cluster restart.
> > > > >
> > > > > Here is a ticket which describes the same issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-6629
> > > > >
> > > > > > BTW, Is proposed solution provides the guarantee that services
> will
> > > be
> > > > > > redeployed after each cluster restart since now we're not using
> the
> > > > > cache?
> > > > >
> > > > > No, only services described in IgniteConfiguration will be
> deployed at
> > > > > node startup as well as now.
> > > > >
> > > > > Am I wrong in something?
> > > > > On Thu, Aug 23, 2018 at 5:59 PM Anton Vinogradov 
> > > wrote:
> > > > > >
> > > > > > Vyacheslav.
> > > > > >
> > > > > > It looks like we able to restart all services on grid startup
> from
> > > old
> > > > > > definitions (inside cache) in case persistence turned on.
> > > > > > Se no problems to provide such automated migration case.
> > > > > > Also, we can test it using compatibility framework.
> > > > > >
> > > > > > BTW, Is proposed solution provides the guarantee that services
> will
> > > be
> > > > > > redeployed after each cluster restart since now we're not using
> the
> > > > > cache?
> > > > > >
> > > > > > чт, 23 авг. 2018 г. в 15:21, Nikolay Izhikov <
> nizhi...@apache.org>:
> > > > > >
> > > > > > > Hello, Vya

Re: MVCC and transactional SQL is merged to master

2018-08-30 Thread Dmitriy Setrakyan
Very nice! Looking forward to seeing this functionality in 2.7 release.

On Thu, Aug 30, 2018 at 5:22 AM, Yakov Zhdanov  wrote:

> Great news, Vladimir! Congratulations!
>
> --Yakov
>
> 2018-08-30 15:15 GMT+03:00 Vladimir Ozerov :
>
> > Igniters,
> >
> > I am glad to announce that we finally merged MVCC and transactional SQL
> > support to master branch.
> >
> > This long journey started more than a year ago with multiple design
> > brainstorm sessions, conducted by Apache Ignite fellows - Semen Boikov,
> > Alexey Goncharuk, Sergi Vladykin.
> >
> > As things had became more clear, we gradually switched to active
> > development phase in November 2017. Since then we implemented new
> > transactional model based on multi-version approach and snapshot
> isolation,
> > and almost fully reworked SQL engine to support transactions.
> >
> > But this is not the end of the story. In Apache Ignite 2.7 we expect to
> > release transactional SQL as "release candidate". To achieve this we
> still
> > need to implement a number of things, such as new transactional protocol
> > for key-value API, historical rebalance, continuous queries. Between AI
> 2.7
> > and AI 2.8 we will work on several not-yet-supported cache operations,
> and
> > also will focus on performance and stability.
> >
> > I would like to thank all community members, who worked hard to make MVCC
> > happen: Igor Seliverstov, Alexander Paschenko, Sergey Kalashnikov, igor
> > Sapego, Roman Kondakov, Pavel Kuznetsov, Ivan Pavlukihn, Andrey
> Mashenkov,
> > and many other contributors who helped us with design, testing and
> > benchmarking.
> >
> > Release notes and documentation will be prepared by AI 2.7 release.
> >
> > Please feel free to ask any questions about the feature here.
> >
> > Vladimir.
> >
>


Re: Bots on dev list

2018-08-29 Thread Dmitriy Setrakyan
Is anyone in the community using or was using Nabble for the dev list
communication? Personally, I am subscribed to the dev list and use filters
in my email client.

D.

On Wed, Aug 29, 2018 at 3:40 AM, Denis Mekhanikov 
wrote:

> Guys,
>
> Yep, I use filters on my mail account. But the portal is impossible to use.
> When you subscribe to the dev list for the first time, you don't have any
> history on your email,
> and the archive is polluted with messages, sent by bots.
>
> Some view on Nabble, that doesn't contain any automatically generated
> messages, would help here.
>
> Denis
>
> ср, 29 авг. 2018 г. в 12:26, Dmitrii Ryabov :
>
> >  Modern mail services allow users to filter messages. You can easily
> filter
> > out bot messages.
> >
> > 2018-08-29 11:48 GMT+03:00 Denis Mekhanikov :
> >
> > > Igniters,
> > >
> > > We have a lot of threads, created by bots on the dev list.
> > > Currently messages are sent by JIRA, GitHub and MTCGA bots. Maybe, some
> > > others too, but these are the most active.
> > >
> > > Take a look at this page:
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > > com/Apache-Ignite-Developers-f1i35.html
> > > It's hard to find actual discussions in this mess. I'd like to see
> > > something like what we have on the users list:
> > > http://apache-ignite-users.70518.x6.nabble.com/
> > >
> > > It doesn't seem necessary to me to mix discussions with this cryptic
> flow
> > > of information.
> > > Can we create a separate mailing list for bots?
> > >
> > > Denis
> > >
> >
>


Re: New committer: Dmitriy Govorukhin

2018-08-29 Thread Dmitriy Setrakyan
Dmitriy, congrats! Looking forward to many contributions to come.

On Wed, Aug 29, 2018 at 12:35 PM, Denis Magda  wrote:

> The Project Management Committee (PMC) for Apache Ignite
> has invited Dmitriy Govorukhin to become a committer and we are pleased
> to announce that he has accepted.
>
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
>
> Congrats, Dmitriy! Look forward to your contributions.
>
> --
> Denis
>


Re: Apache Ignite 2.7 release

2018-08-28 Thread Dmitriy Setrakyan
Hi Nikolai,

Generally looks OK, however, It is hard to comment on your schedule without
seeing a full list of all must-have features we plan to add to this
release. I am hoping that the community will see this list at some point.

D.

On Tue, Aug 28, 2018 at 8:23 AM, Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I think we should discuss the release schedule.
>
> Current dates are following:
>
> * Code Freeze: September 30, 2018
> * Voting Date: October 1, 2018
> * Release Date: October 15, 2018
>
> We discussed it privately with Vladimir Ozerov.
>
> Is seems better to reschedule a bit:
>
> * Scope freeze - September 17 - We should have a full list of
> tickets for 2.7 here.
> * Code freeze - October 01 - We should merge all 2.7 tickets to
> master here.
> * Vote - October 08.
>
> What do you think?
>
>
> В Сб, 25/08/2018 в 00:57 +0300, Dmitriy Pavlov пишет:
> > I hope Vyacheslav can comment better than me. I suppose it is, more or
> > less, rectifications and clarifications of design aspects. Not overall
> > redesign.
> >
> > I also hope Igniters, especially Services experts, will join the
> discussion
> > in the separate topic. Now after a couple of days there is no reaction.
> >
> > сб, 25 авг. 2018 г. в 0:53, Dmitriy Setrakyan :
> >
> > > On Fri, Aug 24, 2018 at 2:50 PM, Dmitriy Pavlov  >
> > > wrote:
> > >
> > > > Hi Dmitriy, I suppose it highly depends on how fast community will
> come
> > >
> > > to
> > > > a consensus about design. So it is up to us to make this happen
> > > >
> > >
> > > I am confused then. If we are still discussing design, then we are
> miles
> > > away. Do you know if there anything in service grid that has already
> been
> > > implemented and can be released?
> > >
>


Re: Service Grid new design overview

2018-08-27 Thread Dmitriy Setrakyan
Agree with Val. I think all users would expect that a service is restarted
upon a node or cluster restart. Let's make sure we preserve this behavior.

D.

On Fri, Aug 24, 2018 at 4:17 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Guys,
>
> I believe we should preserve the behavior that we have now. What happens to
> services if we restart a persistent cluster running 2.6? Are services
> recreated or not? If YES, we should make sure the same happens after
> redesign. Would be even better if we preserve compatibility, i.e. allow
> seamless upgrade from older version that uses system cache to newer version
> that uses disco messages for service deployment. If NO, it's much easier
> and we can leave it as is for now. However, eventually would be great to
> have an option to persist services and redeploy them after cluster restart.
>
> -Val
>
> On Fri, Aug 24, 2018 at 2:51 PM Dmitriy Pavlov 
> wrote:
>
> > Denis M. & Val please share your vision about this topic.
> >
> > пт, 24 авг. 2018 г. в 15:52, Vyacheslav Daradur :
> >
> > > Nick, Antron, thank you for stepping in.
> > >
> > > AFAIK, Ignite cluster can move its state to a new version of Ignite
> > > using persistence only.
> > >
> > > Since Ignite v.2.3 persistence is configured on a memory region and
> > > system memory region is not persistence, that means the system
> > > (utility) cache will not be recovered on cluster restart.
> > >
> > > Here is a ticket which describes the same issue:
> > > https://issues.apache.org/jira/browse/IGNITE-6629
> > >
> > > > BTW, Is proposed solution provides the guarantee that services will
> be
> > > > redeployed after each cluster restart since now we're not using the
> > > cache?
> > >
> > > No, only services described in IgniteConfiguration will be deployed at
> > > node startup as well as now.
> > >
> > > Am I wrong in something?
> > > On Thu, Aug 23, 2018 at 5:59 PM Anton Vinogradov 
> wrote:
> > > >
> > > > Vyacheslav.
> > > >
> > > > It looks like we able to restart all services on grid startup from
> old
> > > > definitions (inside cache) in case persistence turned on.
> > > > Se no problems to provide such automated migration case.
> > > > Also, we can test it using compatibility framework.
> > > >
> > > > BTW, Is proposed solution provides the guarantee that services will
> be
> > > > redeployed after each cluster restart since now we're not using the
> > > cache?
> > > >
> > > > чт, 23 авг. 2018 г. в 15:21, Nikolay Izhikov :
> > > >
> > > > > Hello, Vyacheslav.
> > > > >
> > > > > Thanks, for sharing your design.
> > > > >
> > > > > > I have a question about services migration from AI 2.6 to a new
> > > solution
> > > > >
> > > > > Can you describe consequences of not having migration solution?
> > > > > What will happen on the user side?
> > > > >
> > > > >
> > > > > В Чт, 23/08/2018 в 14:44 +0300, Vyacheslav Daradur пишет:
> > > > > > Hi, Igniters!
> > > > > >
> > > > > > I’m working on Service Grid redesign tasks and design seems to be
> > > > > finished.
> > > > > >
> > > > > > The main goal of Service Grid redesign is to provide missed
> > > guarantees:
> > > > > > - Synchronous services deployment/undeployment;
> > > > > > - Failover on coordinator change;
> > > > > > - Propagation of deployments errors across the cluster;
> > > > > > - Introduce of a deployment failures policy;
> > > > > > - Prevention of deployments initiators hangs while deployment;
> > > > > > - etc.
> > > > > >
> > > > > > I’d like to ask the community their thoughts about the proposed
> > > design
> > > > > > to be sure that all important things have been considered.
> > > > > >
> > > > > > Also, I have a question about services migration from AI 2.6 to a
> > new
> > > > > > solution. It’s very hard to provide tools for users migration,
> > > because
> > > > > > of significant changes. We don’t use utility cache anymore.
> Should
> > we
> > > > > > spend time on this?
> > > > > >
> > > > > > I’ve prepared a definition of new Service Grid design, it’s
> > described
> > > > > below:
> > > > > >
> > > > > > *OVERVIEW*
> > > > > >
> > > > > > All nodes (servers and clients) are able to host services, but
> the
> > > > > > client nodes are excluded from service deployment by default. The
> > > only
> > > > > > way to deploy service on clients nodes is to specify node filter
> in
> > > > > > ServiceConfiguration.
> > > > > >
> > > > > > All deployed services are identified internally by “serviceId”
> > > > > > (IgniteUuid). This allows us to build a base for such features as
> > hot
> > > > > > redeployment and service’s versioning. It’s important to have the
> > > > > > ability to identify and manage services with the same name, but
> > > > > > different version.
> > > > > >
> > > > > > All actions on service’s state change are processed according to
> > > unified
> > > > > flow:
> > > > > > 1) Initiator sends over disco-spi a request to change service
> state
> > > > > > [deploy, undeploy] 

Re: Apache Ignite 2.7 release

2018-08-24 Thread Dmitriy Setrakyan
On Fri, Aug 24, 2018 at 2:50 PM, Dmitriy Pavlov 
wrote:

> Hi Dmitriy, I suppose it highly depends on how fast community will come to
> a consensus about design. So it is up to us to make this happen
>

I am confused then. If we are still discussing design, then we are miles
away. Do you know if there anything in service grid that has already been
implemented and can be released?


Re: Apache Ignite 2.7 release

2018-08-24 Thread Dmitriy Setrakyan
 to include the links to all important Jira
> > >
> > > tickets> in this thread
> > >
> > > Open:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-6980 - Automatic
> cancelling
> > > of hanging Ignite operations
> > > https://issues.apache.org/jira/browse/IGNITE-5473 - Create ignite
> > > troubleshooting logger
> > > https://issues.apache.org/jira/browse/IGNITE-6903 - Implement new JMX
> > > metrics for Indexing
> > > https://issues.apache.org/jira/browse/IGNITE-6507 - Commit can be
> lost in
> > > network split scenario
> > >
> > > In Progress:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-9349
> > >
> > > Patch Available:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-7251 - Remove term
> "fabric"
> > > from Ignite deliverables
> > >
> > >
> > > Resolved:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-8780 - File I/O
> operations
> > > must be retried if buffer hasn't read/written completely
> > > https://issues.apache.org/jira/browse/IGNITE-5059 - Implement logistic
> > > regression
> > > https://issues.apache.org/jira/browse/IGNITE-3478 - Multi-version
> > > concurrency control
> > > https://issues.apache.org/jira/browse/IGNITE-9340 - Update jetty
> version
> > > in Apache Ignite (ignite-rest-http)
> > >
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/IGNITE/
> Apache+Ignite+2.7
> > >
> > > В Пт, 24/08/2018 в 10:47 +0300, Nikolay Izhikov пишет:
> > > > Hello, Pavel.
> > > >
> > > > Please, be aware of IGNITE-6055 [1]
> > > >
> > > > I'm edit thin protocol in that ticket.
> > > > I can't support changes in Python and PHP clients, because, they are
> not
> > >
> > > merged in master, yet.
> > > > Write me, If you have any questions about new fields.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6055
> > > >
> > > > В Чт, 23/08/2018 в 18:02 -0700, Pavel Petroshenko пишет:
> > > > > Hi Nikolay,
> > > > >
> > > > > Python [1], PHP [2], and Node.js [3] thin clients will get into the
> > >
> > > release.
> > > > >
> > > > > Thanks,
> > > > > p.
> > > > >
> > > > > [1] https://jira.apache.org/jira/browse/IGNITE-7782
> > > > > [2] https://jira.apache.org/jira/browse/IGNITE-7783
> > > > > [3] https://jira.apache.org/jira/browse/IGNITE-
> > > > >
> > > > >
> > > > > On Tue, Aug 21, 2018 at 12:20 PM, Dmitriy Setrakyan <
> > >
> > > dsetrak...@apache.org> wrote:
> > > > > > Thanks, Nikolay!
> > > > > >
> > > > > > I think it is important to include the links to all important
> Jira
> > >
> > > tickets
> > > > > > in this thread, so that the community can track them.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Tue, Aug 21, 2018 at 12:06 AM, Nikolay Izhikov <
> > >
> > > nizhi...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, Dmitriy.
> > > > > > >
> > > > > > > I think Transparent Data Encryption will be available in 2.7
> > > > > > >
> > > > > > > В Пн, 20/08/2018 в 13:20 -0700, Dmitriy Setrakyan пишет:
> > > > > > > > Hi Nikolay,
> > > > > > > >
> > > > > > > > Thanks for being the release manager!
> > > > > > > >
> > > > > > > > I am getting a bit lost in all these tickets. Can we specify
> some
> > > > > > > > high-level tickets, that are not plain bug fixes, which will
> be
> > > > > > >
> > > > > > > interesting
> > > > > > > > for the community to notice?
> > > > > > > >
> > > > > > > > For example, here are some significant tasks that the
> community
> > >
> > > is either
> > > > > > > > working on or has been working on:
> > > > > > > >
> > > > > > > > - Node.JS client
> > > > > > > > - Python client
> > > > > > > > - Transactional SQL (MVCC)
> > > > > > > > - service grid stabilization
> > > > > > > > - SQL memory utilization improvements
> > > > > > > > - more?
> > > > > > > >
> > > > > > > > Can you please solicit status from the community for these
> tasks?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov <
> > >
> > > nizhi...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello, Igniters.
> > > > > > > > >
> > > > > > > > > I'm release manager of Apache Ignite 2.7.
> > > > > > > > >
> > > > > > > > > It's time to start discussion of release. [1]
> > > > > > > > >
> > > > > > > > > Current code freeze date is September, 30.
> > > > > > > > > If you have any objections - please, responsd to this
> thread.
> > > > > > > > >
> > > > > > > > > [1] https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > >
> > > > > > > Apache+Ignite+2.7
> > > > > > >
>


Re: Binary Client Protocol client hangs in case of OOM on server

2018-08-23 Thread Dmitriy Setrakyan
Hi, do you have query timeout configured?

D.

On Thu, Aug 23, 2018 at 9:09 AM, dmitrievanthony 
wrote:

> When I'm sending Scan Query request via Binary Client Protocol with very
> big
> page size I get OOM on the server node:
> java.lang.OutOfMemoryError: Java heap space at
> org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk.
> reallocate(BinaryMemoryAllocatorChunk.java:69)
> at
> org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.
> ensureCapacity(BinaryHeapOutputStream.java:65)
> at
> org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.
> writeByteArray(BinaryAbstractOutputStream.java:41)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteByteArray(
> BinaryWriterExImpl.java:530)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.
> write(BinaryClassDescriptor.java:634)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal0(BinaryWriterExImpl.java:223)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal(BinaryWriterExImpl.java:164)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal(BinaryWriterExImpl.java:151)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(
> BinaryWriterExImpl.java:1506)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.
> java:44)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.
> java:29)
> at
> org.apache.ignite.internal.processors.platform.client.
> cache.ClientCacheQueryCursor.writePage(ClientCacheQueryCursor.java:80)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheQueryResponse.encode(ClientCacheQueryResponse.java:50)
> at
> org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.encode(ClientMessageParser.java:379)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:172)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:45)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$
> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> at
> org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> body(GridNioAsyncNotifyFilter.java:97)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at
> org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> run(GridWorkerPool.java:70)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Exception in thread
> "client-connector-#61" java.lang.OutOfMemoryError: Java heap space  at
> org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk.
> reallocate(BinaryMemoryAllocatorChunk.java:69)
> at
> org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.
> ensureCapacity(BinaryHeapOutputStream.java:65)
> at
> org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.
> writeByteArray(BinaryAbstractOutputStream.java:41)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteByteArray(
> BinaryWriterExImpl.java:530)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.
> write(BinaryClassDescriptor.java:634)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal0(BinaryWriterExImpl.java:223)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal(BinaryWriterExImpl.java:164)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.
> marshal(BinaryWriterExImpl.java:151)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(
> BinaryWriterExImpl.java:1506)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.
> java:44)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.
> java:29)
> at
> org.apache.ignite.internal.processors.platform.client.
> cache.ClientCacheQueryCursor.writePage(ClientCacheQueryCursor.java:80)
> at
> org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheQueryResponse.encode(ClientCacheQueryResponse.java:50)
> at
> org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.encode(ClientMessageParser.java:379)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:172)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> 

Re: Apache Ignite 2.7 release

2018-08-21 Thread Dmitriy Setrakyan
Thanks, Nikolay!

I think it is important to include the links to all important Jira tickets
in this thread, so that the community can track them.

D.

On Tue, Aug 21, 2018 at 12:06 AM, Nikolay Izhikov 
wrote:

> Hello, Dmitriy.
>
> I think Transparent Data Encryption will be available in 2.7
>
> В Пн, 20/08/2018 в 13:20 -0700, Dmitriy Setrakyan пишет:
> > Hi Nikolay,
> >
> > Thanks for being the release manager!
> >
> > I am getting a bit lost in all these tickets. Can we specify some
> > high-level tickets, that are not plain bug fixes, which will be
> interesting
> > for the community to notice?
> >
> > For example, here are some significant tasks that the community is either
> > working on or has been working on:
> >
> > - Node.JS client
> > - Python client
> > - Transactional SQL (MVCC)
> > - service grid stabilization
> > - SQL memory utilization improvements
> > - more?
> >
> > Can you please solicit status from the community for these tasks?
> >
> > D.
> >
> > On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov 
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I'm release manager of Apache Ignite 2.7.
> > >
> > > It's time to start discussion of release. [1]
> > >
> > > Current code freeze date is September, 30.
> > > If you have any objections - please, responsd to this thread.
> > >
> > > [1] https://cwiki.apache.org/confluence/display/IGNITE/
> Apache+Ignite+2.7
>


Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

2018-08-20 Thread Dmitriy Setrakyan
Sergey, that was precisely my comment in the ticket:

Can we add this option without breaking compatibility with previous
page/storage formats? If not, then this should support both implementation.
The default should be the new fastest implementation, but if we encounter
the older, slower one, then we should print out a warning in the log and
automatically switch to the older implementation.

On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov  wrote:

> Hi Igniters
>
> I suppose that'll break compatibility for LFS (PDS).
>
> Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x to
> 3.0?
>
> On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan  >
> wrote:
>
> > I commented in the ticket: https://issues.apache.org/
> > jira/browse/IGNITE-9272
> >
> > It if can integrate it correctly, according to my comment, in 2.7
> release,
> > it would be great. Otherwise, let's plan this change for 3.0 release.
> >
> > D.
> >
> > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > eduard.shangar...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have checked the benchmark and it shows great performance boost on my
> > > laptop!
> > >
> > > +1 for this change.
> > >
> > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov 
> > > wrote:
> > >
> > > > Hi Evgeniy,
> > > >
> > > > Thank you. I see that the ticket is unassigned.
> > > >
> > > > Would you like to contribute PR to be macro-benchmarked with Ignite?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > :
> > > >
> > > > > I fill the ticket, bench code attached there.
> > > > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > > > Thanks!
> > > > >
> > > > >
> > > > > >Has anyone else run the benchmark and reproduced the performance
> > > > > >difference?
> > > > > >
> > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> > > dpavlov@gmail.com
> > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> It depends.
> > > > > >>
> > > > > >> CRC is a CPU-intensive operation, while WAL logging and page
> store
> > > > write
> > > > > >> are mostly about IO speed.
> > > > > >>
> > > > > >> In the same time, it can make the huge impact on machines with
> > fast
> > > IO
> > > > > >> and
> > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and
> Alexey
> > > it
> > > > > >> could
> > > > > >> benefit performance because we save CPU. Later we can use it's
> > power
> > > > in
> > > > > a
> > > > > >> more efficient manner (e.g. with compression).
> > > > > >>
> > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov <
> yzhda...@apache.org
> > >:
> > > > > >>
> > > > > >> > Guys, what time in % does crc calculation take in WAL logging
> > > > process?
> > > > > >> >
> > > > > >> > --Yakov
> > > > > >> >
> > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov <
> > dpavlov@gmail.com
> > > > >:
> > > > > >> >
> > > > > >> > > Hi Alex, thank you for this idea.
> > > > > >> > >
> > > > > >> > > Evgeniy, Alex, would you like to submit the patch with
> > bypassing
> > > > > >> > > implementation differences to keep compatibility?
> > > > > >> > >
> > > > > >> > > Sincerely,
> > > > > >> > > Dmitriy Pavlov
> > > > > >> > >
> > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov <
> > > > > plehanov.a...@gmail.com >:
> > > > > >> > >
> > > > > >> > > > Hello, Igniters!
> > > > > >> > > >
> > > > > >> > > > In java8 java.lang.zip.CRC32 methods become int

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

2018-08-20 Thread Dmitriy Setrakyan
I commented in the ticket: https://issues.apache.org/jira/browse/IGNITE-9272

It if can integrate it correctly, according to my comment, in 2.7 release,
it would be great. Otherwise, let's plan this change for 3.0 release.

D.

On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Hi,
>
> I have checked the benchmark and it shows great performance boost on my
> laptop!
>
> +1 for this change.
>
> On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov 
> wrote:
>
> > Hi Evgeniy,
> >
> > Thank you. I see that the ticket is unassigned.
> >
> > Would you like to contribute PR to be macro-benchmarked with Ignite?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > :
> >
> > > I fill the ticket, bench code attached there.
> > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > Thanks!
> > >
> > >
> > > >Has anyone else run the benchmark and reproduced the performance
> > > >difference?
> > > >
> > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> dpavlov@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> It depends.
> > > >>
> > > >> CRC is a CPU-intensive operation, while WAL logging and page store
> > write
> > > >> are mostly about IO speed.
> > > >>
> > > >> In the same time, it can make the huge impact on machines with fast
> IO
> > > >> and
> > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey
> it
> > > >> could
> > > >> benefit performance because we save CPU. Later we can use it's power
> > in
> > > a
> > > >> more efficient manner (e.g. with compression).
> > > >>
> > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >:
> > > >>
> > > >> > Guys, what time in % does crc calculation take in WAL logging
> > process?
> > > >> >
> > > >> > --Yakov
> > > >> >
> > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com
> > >:
> > > >> >
> > > >> > > Hi Alex, thank you for this idea.
> > > >> > >
> > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing
> > > >> > > implementation differences to keep compatibility?
> > > >> > >
> > > >> > > Sincerely,
> > > >> > > Dmitriy Pavlov
> > > >> > >
> > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov <
> > > plehanov.a...@gmail.com >:
> > > >> > >
> > > >> > > > Hello, Igniters!
> > > >> > > >
> > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic,
> moreover
> > > new
> > > >> > > > "update" method, which use ByteBuffer was introduced. Since we
> > > >> moved
> > > >> to
> > > >> > > > java8, perhaps we really can get performance boost by using
> > > >> standard
> > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32.
> > > >> > > >
> > > >> > > > About compatibility: looks like PureJavaCrc32 implements the
> > same
> > > >> > > algorithm
> > > >> > > > as java.lang.zip.CRC32. These two implementations uses the
> same
> > > >> > > polynomial
> > > >> > > > and the same initial value. The only difference is final xor
> > mask
> > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily
> convert
> > > >> from
> > > >> > > > PureJavaCrc32
> > > >> > > > to standard CRC32 and vice versa, using this expression: crc32
> > ^=
> > > >> > > > 0x
> > > >> > > >
> > > >> > > >
> > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev <
> > > >> >  eduard.shangar...@gmail.com
> > > >> > > >:
> > > >> > > >
> > > >> > > > > Evgeniy,
> > > >> > > > >
> > > >> > > > > Could you share benchmark code? And please share what
> version
> > of
> > > >> JVM
> > > >> > > > > you have used.
> > > >> > > > >
> > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya
> > > >> < arzamas...@mail.ru.invalid
> > > >> >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > I think it would break backward compatibility, as Nikolay
> > > >> mentioned
> > > >> > > > above
> > > >> > > > > > we would take exception here:
> > > >> > > > > >
> > > >> > > > > > [1]
> > > >> > > > > >
> > > >> > > > > >  https://github.com/apache/ignite/blob/master/modules/
> > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/
> > > >> > > > > cache/persistence/file/FilePageStore.java#L372
> > > >> > > > > >
> > > >> > > > > > thats why i question for community thoughts here.
> > > >> > > > > >
> > > >> > > > > > > Hi Evgeniy,
> > > >> > > > > > >
> > > >> > > > > > > would you like to submit a patch with CRC32
> implementation
> > > >> > change?
> > > >> > > > > > >
> > > >> > > > > > > Sincerely,
> > > >> > > > > > > Dmitriy Pavlov
> > > >> > > > > > >
> > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский
> > > >> > > > > > > < arzamas...@mail.ru.invalid >:
> > > >> > > > > > >
> > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like
> > > >> PureJavaCrc32
> > > >> > has
> > > >> > > > > > >> performance problems in compatible with zip.CRC32.
> > > >> > > > > > >>
> > > >> > > > > > >> Benchmark Mode Cnt Score Error Units
> > > >> > > > > > >> 

Re: Apache Ignite 2.7 release

2018-08-20 Thread Dmitriy Setrakyan
Hi Nikolay,

Thanks for being the release manager!

I am getting a bit lost in all these tickets. Can we specify some
high-level tickets, that are not plain bug fixes, which will be interesting
for the community to notice?

For example, here are some significant tasks that the community is either
working on or has been working on:

- Node.JS client
- Python client
- Transactional SQL (MVCC)
- service grid stabilization
- SQL memory utilization improvements
- more?

Can you please solicit status from the community for these tasks?

D.

On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I'm release manager of Apache Ignite 2.7.
>
> It's time to start discussion of release. [1]
>
> Current code freeze date is September, 30.
> If you have any objections - please, responsd to this thread.
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7


Re: Continuous queries and MVCC

2018-08-17 Thread Dmitriy Setrakyan
On Fri, Aug 17, 2018 at 3:18 AM, Vladimir Ozerov 
wrote:

> Folks,
>
> As you know we are developing multi-version concurrency control for Ignite
> caches, which is essentially new mode for transactional cache with snapshot
> semantics. One of the most important applications of this mode would be
> fully transactional SQL The question is how to implement continuous query
> semantics with MVCC. Several interesting things here.
>
> 1) *Event ordering*. Currently we guarantee order of updates for specific
> key. But for fair transactional mode user may want to get transactional
> ordering instead of key ordering. If there are several dependent
> transactions, I may want to receive their updates in order. E.g.:
> TX1: transfer(A -> B)
> TX2: transfer(B -> C)
> TX3: transfer(C -> D)
>
> If user receives update on key D (TX3), should we guarantee that he already
> received updates for all keys in TX1 and TX2? My opinion is that we'd
> better to leave things as is and guarantee only per-key ordering. Only one
> reason - implementation complexity.
>

I would preserve at least the current guarantees. However, many users
expressed interested in receiving events in proper order. If it is possible
to do, I would definitely do it, or at lease provide a configuration option
to have such behavior.


>
> 2) *Initial query*. We implemented it so that user can get some initial
> data snapshot and then start receiving events. Without MVCC we have no
> guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> possible to see V2 in initial query and in event. With MVCC it is now
> technically possible to query data on certain snapshot and then receive
> only events happened after this snapshot. So that we never see V2 twice. Do
> you think we this feature will be interesting for our users?
>

Absolutely! It will CQ usage much cleaner and less error-prone.


Re: QueryDetailMetrics for cache-less SQL queries

2018-08-16 Thread Dmitriy Setrakyan
But internally the SQL query still runs on some cache, no? What happens to
the metrics accumulated on that cache?

D.

On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov  wrote:

> Dima,
>
> "cache-less" means that SQL executed directly on SQL engine.
>
> In previous version of Ignite we execute queries via cache:
>
> ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
>
> In current Ignite we can execute query directly without using cache as
> "gateway".
>
> And if we execute query directly, metrics not update.
>
>
>
>
> On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan 
> wrote:
>
> > Evgeny, what is a "cache-less" SQL query?
> >
> > D.
> >
> > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > e.zhuravlev...@gmail.com
> > > wrote:
> >
> > > Hi Igniters,
> > >
> > > I've started to work on adding QueryDetailMetrics for cache-less SQL
> > > queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and
> > found
> > > that it's required to change API. I don't think that adding methods
> like
> > > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to
> Ignite
> > > class is a good idea. So, I see 2 possible solutions here:
> > >
> > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> > > metric "queryDetailMetrics" to it. Of course, old methods will be
> > > deprecated.
> > >
> > > 2. Finally create Ignite.sql() API, which was already discussed here:
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > and place "queryDetailMetrics" metric there. Here is the ticket for
> this
> > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > >
> > > Personally, I think that the second solution looks better in this case,
> > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > ignite.matrics() is still a good idea - IMO, Ignite class is not the
> > right
> > > place for them - we shouldn't change our main API class so often.
> > >
> > > What do you think?
> > >
> > > Thank you,
> > > Evgenii
> > >
> >
> > --
> > Alexey Kuznetsov
> >
> >
>


Re: Wrong off-heap size is reported for a node

2018-08-16 Thread Dmitriy Setrakyan
Is there a blocker ticket for 2.7?

On Thu, Aug 16, 2018, 19:59 Denis Magda  wrote:

> Igniters,
>
> Was troubleshooting an Ignite deployment today and couldn't find out from
> the logs what was the actual off-heap space used.
>
> Those were the given memory resoures (Ignite 2.6):
>
> [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology
> snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*,
> heap=24.0GB]
>
> And that weird stuff was reported by the node (pay attention to the last
> line):
>
> [2018-08-16 15:45:50,211][INFO
>
> ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257]
> ^-- H/N/C [hosts=1, nodes=1, CPUs=64]
> ^-- CPU [cur=0.03%, avg=5.54%, GC=0%]
> ^-- PageMemory [pages=6997377]
> ^-- Heap [used=9706MB, free=61.18%, comm=22384MB]
>* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always
> the same!*
>
> Had to change the code by using dataRegion.getPhysicalMemoryPages() to find
> out that actual off-heap usage size was
> >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB
>
> Let's fix this issue in 2.7, I proposed a new format. Please review and
> share your thoughts:
> https://issues.apache.org/jira/browse/IGNITE-9305
>
> --
> Denis
>


Re: Spark SQL Table Name Resolution

2018-08-16 Thread Dmitriy Setrakyan
Stuart, can you please move the ticket into PATCH_AVAILABLE state? You need
to click "Submit Patch" button in Jira.

D.

On Wed, Aug 15, 2018 at 10:22 AM, Stuart Macdonald 
wrote:

> Here's the initial pull request for this issue, please review and let me
> know your feedback. I had to combine the two approaches to enable this to
> work for both standard .read() where we can add the schema option, and
> catalog-based selects where we use schemaName.tableName. Happy to discuss
> on a call if this isn't clear.
>
> https://github.com/apache/ignite/pull/4551
>
> On Thu, Aug 9, 2018 at 2:32 PM, Stuart Macdonald 
> wrote:
>
> > Hi Nikolay, yes would be happy to - will likely be early next week. I’ll
> > go with the approach of adding a new optional field to the Spark data
> > source provider unless there are any objections.
> >
> > Stuart.
> >
> > > On 9 Aug 2018, at 14:20, Nikolay Izhikov  wrote:
> > >
> > > Stuart, do you want to work on this ticket?
> > >
> > > В Вт, 07/08/2018 в 11:13 -0700, Stuart Macdonald пишет:
> > >> Thanks Val, here’s the ticket:
> > >>
> > >> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-9228
> > >>  > IGNITE-9228?filter=allopenissues>
> > >>
> > >> (Thanks for correcting my terminology - I work mostly with the
> > traditional
> > >> CacheConfiguration interface where I believe each cache occupies its
> own
> > >> schema.)
> > >>
> > >> Stuart.
> > >>
> > >> On 7 Aug 2018, at 18:34, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>
> > >> wrote:
> > >>
> > >> Stuart,
> > >>
> > >> Two tables can have same names only if they are located in different
> > >> schemas. Said that, sdding schema name support makes sense to me for
> > sure.
> > >> We can implement this using either separate SCHEMA_NAME parameter, or
> > >> similar to what you suggested in option 3 but with schema name instead
> > of
> > >> cache name.
> > >>
> > >> Please feel free to create a ticket.
> > >>
> > >> -Val
> > >>
> > >> On Tue, Aug 7, 2018 at 9:32 AM Stuart Macdonald 
> > wrote:
> > >>
> > >> Hello Igniters,
> > >>
> > >>
> > >> The Ignite Spark SQL interface currently takes just “table name” as a
> > >>
> > >> parameter which it uses to supply a Spark dataset with data from the
> > >>
> > >> underlying Ignite SQL table with that name.
> > >>
> > >>
> > >> To do this it loops through each cache and finds the first one with
> the
> > >>
> > >> given table name [1]. This causes issues if there are multiple tables
> > >>
> > >> registered in different caches with the same table name as you can
> only
> > >>
> > >> access one of those caches from Spark. Is the right thing to do here:
> > >>
> > >>
> > >> 1. Simply not support such a scenario and note in the Spark
> > documentation
> > >>
> > >> that table names must be unique?
> > >>
> > >> 2. Pass an extra parameter through the Ignite Spark data source which
> > >>
> > >> optionally specifies the cache name?
> > >>
> > >> 3. Support namespacing in the existing table name parameter, ie
> > >>
> > >> “cacheName.tableName”?
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Stuart.
> > >>
> > >>
> > >> [1]
> > >>
> > >>
> > >> https://github.com/apache/ignite/blob/ca973ad99c6112160a305df05be945
> > 8e29f88307/modules/spark/src/main/scala/org/apache/ignite/
> > spark/impl/package.scala#L119
> >
>


Re: QueryDetailMetrics for cache-less SQL queries

2018-08-16 Thread Dmitriy Setrakyan
Evgeny, what is a "cache-less" SQL query?

D.

On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev  wrote:

> Hi Igniters,
>
> I've started to work on adding QueryDetailMetrics for cache-less SQL
> queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and found
> that it's required to change API. I don't think that adding methods like
> queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to Ignite
> class is a good idea. So, I see 2 possible solutions here:
>
> 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> metric "queryDetailMetrics" to it. Of course, old methods will be
> deprecated.
>
> 2. Finally create Ignite.sql() API, which was already discussed here:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> and place "queryDetailMetrics" metric there. Here is the ticket for this
> change: https://issues.apache.org/jira/browse/IGNITE-4701
>
> Personally, I think that the second solution looks better in this case,
> however, moving dataRegionMetrics and dataStorageMetrics to
> ignite.matrics() is still a good idea - IMO, Ignite class is not the right
> place for them - we shouldn't change our main API class so often.
>
> What do you think?
>
> Thank you,
> Evgenii
>


Re: This morning I added ssl support to...

2018-08-16 Thread Dmitriy Setrakyan
On Thu, Aug 16, 2018 at 8:39 AM, Paul Anderson  wrote:

> Wonderful... I also have Netdata integration... (producing netdata charts
> straight from ignite) if you are interested...
>
> http://my-netdata.io


Yes, we are interested very much.


Re: Peer review: Victory over Patch Available debt

2018-08-16 Thread Dmitriy Setrakyan
Great idea!

Review is not only meant for the committers, but for everyone in the
community. Committers are only responsible for final reviews and merges.

D.

On Thu, Aug 16, 2018 at 7:29 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> For a long time review queue in the community is quite long. Ticket count
> is around 80-100 for last year. I would like to make the proposal, which
> should help the community with this aspect. Being reviewer for some tickets
> I found a number of tickets from new community members with not checked
> tests, not linked dev. list discussion, empty description, code style
> violations etc. It is not easy to finalize the review in such case.
>
> Every Igniter could make way too needed contribution to Apache Ignite by
> educating newcomers how to make ticket, patch, and discussion as perfect as
> it is humanly possible.
>
> I propose any Igniter (regardless of his role and experience: PMC,
> committer or contributor) will comment patches of other members in JIRA to
> help newcomers make contribution compliant with
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute  I
> know a number of community members already do so.
>
> Every contributor could find any ticket he or she likes in
> https://cwiki.apache.org/confluence/display/IGNITE/
> Issues+waiting+for+review
>
>
> The contributor can check compliance using
> https://cwiki.apache.org/confluence/display/IGNITE/Review+Checklist
> and copy-paste review checklist items to the ticket with his or her visa
> about each item. Each contribution can have an infinite number of visas.
>
> Looks good visa are also helpful, but I would like to go forward and
> formalize this process. Please share your vision. I would be happy to run a
> short webinar in case you have questions. I will update how to contribute
> shortly if we agree.
>
> Sincerely,
> Dmitriy Pavlov
>


Re: Page replacement policy improvements (when persistent is enabled)

2018-08-16 Thread Dmitriy Setrakyan
On Thu, Aug 16, 2018 at 2:27 AM, Vladimir Ozerov 
wrote:

> Dima,
>
> None database I know use separate regions for index pages due to the reason
> I expressed above. Instead, they split all pages into two groups - hot and
> cold. With certain rules on how to move pages inside and between these
> groups. None of these algorithms are complex enough. In fact, they are
> pretty straightforward and battle-tested. When implemented properly it
> doesn't matter whether the page is index page or data page. The only thing
> that matter is how often it is accessed. This is critical piece that we
> lack in the product - our policy is called "random *LRU*", while in reality
> is not LRU at all.
>

Aerospike keeps index pages in memory, so there is at least one database
that does that. I am sure if we research around, there will be more.


> As far as index pages replacement we do not know whether this is problem at
> all. We heard some complaints that it might be a problem. But we didn't see
> any proofs (thanks to lack of monitoring) and even if this is a problem, we
> do not understand how severe it is. May be it adds 1% overhead and can be
> ignored for years, may be it adds 1000% overhead and must be fixed
> immediately.
>

I remember one test case with a potential user where we had to change our
eviction algorithm to avoid evicting index pages and because of that
improved performance by about 10x.


> This is sensitive piece of a product. Let's use objective data, not
> assumptions.
>

I agree. The difference is that we need a solution in the mean time. I am
suggesting a very straight forward approach that can be added fairly
quickly and will solve majority performance problems associated with index
page eviction. Once we have that, we can take our time and investigate
further.


Re: Page replacement policy improvements (when persistent is enabled)

2018-08-16 Thread Dmitriy Setrakyan
On Thu, Aug 16, 2018 at 2:01 AM, Vladimir Ozerov 
wrote:

> Hi Dima,
>
> Putting index pages in separate region is wrong approach, because data
> pages may be equally important on certain workloads, especially in
> heap-organized databases, such as Ignite


Never seen a use case where the data page was more important than the index
page. I think we are getting into a hypothetical land. Most Ignite users
are having the reverse problem - index pages get evicted in the same way as
data pages.

Currently, we are solving it in a most awkward way by trying to give index
pages a different eviction policy. A right solution would be to stick them
into a different region and control the eviction policy for the index
region separately from the data region.

D.


Re: IEP-22: Direct Data Load proposal

2018-08-16 Thread Dmitriy Setrakyan
On Thu, Aug 16, 2018 at 1:24 AM, Vladimir Ozerov 
wrote:

> Hi Denis,
>
> This IEP is mostly about how we work with our own indexes and pages. So 3rd
> party DB is out of question.
>

Why? I think 3rd party DB will be supported automatically with CacheStore.
However, do we need to do something different for memory-only vs.
memory+disk?

D.


Re: StandaloneWalRecordsIterator: support iteration from custom pointer

2018-08-15 Thread Dmitriy Setrakyan
Agree, this should be a great performance boost.

On Wed, Aug 15, 2018 at 10:17 AM, Dmitriy Pavlov 
wrote:

> Hi Ivan,
>
> I agree that providing WAL pointer is the better option. Initially,
> Standalone WAL iterator was developed for debugging utility, so a set of
> files was perfectly OK.
>
> Sincerely,
> Dmitriy Palov
>
> ср, 15 авг. 2018 г. в 20:06, Ivan Rakov :
>
> > Igniters,
> >
> > Right now we are developing WAL shipping process for our internal
> > purposes and we use StandaloneWalRecordsIterator to read WAL files from
> > custom destination. We have bumped into a problem - iterator can be
> > constructed from set of files and dirs, but there's no option to pass
> > WAL pointer to the iterator factory class to start iteration with. It
> > can be worked around (by filtering all records prior to needed pointer),
> > but I think it would be handy to add such option to
> > IgniteWalIteratorFactory API.
> >
> > What do you think?
> >
> > --
> > Best Regards,
> > Ivan Rakov
> >
> >
>


Re: [GitHub] dspavlov commented on a change in pull request #1: MTCGA-002 Build trigger timeout.

2018-08-15 Thread Dmitriy Setrakyan
Agree, we should disable the comments.

On Wed, Aug 15, 2018 at 10:46 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> do you know how to disable PR comments to be forwarded to dev.list? I think
> notification about PR creation is OK to be forwarded.
>
> But I'm not sure we need each comment to be forwarded to dev.list. WDYT?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 15 авг. 2018 г. в 20:39, GitBox :
>
> > dspavlov commented on a change in pull request #1: MTCGA-002 Build
> trigger
> > timeout.
> > URL:
> > https://github.com/apache/ignite-teamcity-bot/pull/1#
> discussion_r210347861
> >
> >
> >
> >  ##
> >  File path:
> > ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/
> conf/ChainAtServer.java
> >  ##
> >  @@ -17,46 +17,62 @@
> >
> >  package org.apache.ignite.ci.conf;
> >
> > +import java.util.Objects;
> >  import javax.annotation.Nonnull;
> >  import javax.annotation.Nullable;
> >
> >  /**
> >   * Created by Дмитрий on 09.11.2017.
> >   */
> >  public class ChainAtServer {
> > -/** Server ID to access config files within helper */
> > +/** Server ID to access config files within helper. */
> >  @Nullable public String serverId;
> >
> > -/** Suite identifier by teamcity identification for root chain */
> > +/** Suite identifier by teamcity identification for root chain. */
> >  @Nonnull public String suiteId;
> >
> >  /** Automatic build triggering. */
> >  @Nullable private Boolean triggerBuild;
> >
> > +/** Automatic build triggering timeout in minutes. */
> > +@Nullable private Integer triggerBuildTimeout;
> > +
> > +/** {@inheritDoc} */
> >  @Override public boolean equals(Object o) {
> >  if (this == o)
> >  return true;
> >  if (o == null || getClass() != o.getClass())
> >  return false;
> > -
> >  ChainAtServer server = (ChainAtServer)o;
> > -
> > -if (serverId != null ? !serverId.equals(server.serverId) :
> > server.serverId != null)
> > -return false;
> > -return suiteId.equals(server.suiteId);
> > +return Objects.equals(serverId, server.serverId) &&
> > +Objects.equals(suiteId, server.suiteId) &&
> > +Objects.equals(triggerBuild, server.triggerBuild) &&
> > +Objects.equals(triggerBuildTimeout,
> > server.triggerBuildTimeout);
> >  }
> >
> > +/** {@inheritDoc} */
> >  @Override public int hashCode() {
> > -int result = serverId != null ? serverId.hashCode() : 0;
> > -result = 31 * result + suiteId.hashCode();
> > -return result;
> > +return Objects.hash(serverId, suiteId, triggerBuild,
> > triggerBuildTimeout);
> >  }
> >
> > +/**
> > + * @return Server ID to access config files within helper.
> > + */
> >  @Nullable public String getServerId() {
> >  return serverId;
> >  }
> >
> > +/**
> > + * @return {@code True} If automatic build triggering enabled.
> > + */
> >  @Nonnull public boolean isTriggerBuild() {
> >  return triggerBuild == null ? false : triggerBuild;
> >  }
> > +
> > +/**
> > + * @return Timeout in minutes between triggering builds or zero if
> > timeout is not set and should be ignored.
> > + */
> > +@Nonnull public int getTriggerBuildTimeout() {
> >
> >  Review comment:
> >I suggest naming this property as the quiet period or something like
> > that. WDYT?
> >
> > 
> > This is an automated message from the Apache Git Service.
> > To respond to the message, please log on GitHub and use the
> > URL above to go to the specific comment.
> >
> > For queries about this service, please contact Infrastructure at:
> > us...@infra.apache.org
> >
> >
> > With regards,
> > Apache Git Services
> >
>


Re: [MTCGA]: new failures in builds [1657700] needs to be handled

2018-08-15 Thread Dmitriy Setrakyan
Pavel, has a blocker for 2.7 been filed?

On Wed, Aug 15, 2018 at 6:19 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> I think it is newly introduced failure. There are at least 3 series of
> failures in recent runs. Could you please take a look?
>
> Please respond if you're already investigating these failures.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 15 авг. 2018 г. в 15:34, :
>
> > Hi Ignite Developer,
> >
> > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> > I hope you can help.
> >
> >  *New Critical Failure in master Queries 1
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_Queries1=%3Cdefault%3E=buildTypeStatusDiv
> >  No changes in build
> >
> > - If your changes can led to this failure(s), please create issue
> > with label MakeTeamCityGreenAgain and assign it to you.
> > -- If you have fix, please set ticket to PA state and write to
> dev
> > list fix is ready
> > -- For case fix will require some time please mute test and set
> > label Muted_Test to issue
> > - If you know which change caused failure please contact change
> > author directly
> > - If you don't know which change caused failure please send
> > message to dev list to find out
> > Should you have any questions please contact dpav...@apache.org or write
> > to dev.list
> > Best Regards,
> > MTCGA.Bot
> > Notification generated at Wed Aug 15 15:34:18 MSK 2018
> >
>


Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

2018-08-14 Thread Dmitriy Setrakyan
Has anyone else run the benchmark and reproduced the performance difference?

On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov 
wrote:

> It depends.
>
> CRC is a CPU-intensive operation, while WAL logging and page store write
> are mostly about IO speed.
>
> In the same time, it can make the huge impact on machines with fast IO and
> slow CPU. So if we can apply change proposed by Evgeniy and Alexey it could
> benefit performance because we save CPU. Later we can use it's power in a
> more efficient manner (e.g. with compression).
>
> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov :
>
> > Guys, what time in % does crc calculation take in WAL logging process?
> >
> > --Yakov
> >
> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov :
> >
> > > Hi Alex, thank you for this idea.
> > >
> > > Evgeniy, Alex, would you like to submit the patch with bypassing
> > > implementation differences to keep compatibility?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov :
> > >
> > > > Hello, Igniters!
> > > >
> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new
> > > > "update" method, which use ByteBuffer was introduced. Since we moved
> to
> > > > java8, perhaps we really can get performance boost by using standard
> > > > java.lang.zip.CRC32 instead of PureJavaCrc32.
> > > >
> > > > About compatibility: looks like PureJavaCrc32 implements the same
> > > algorithm
> > > > as java.lang.zip.CRC32. These two implementations uses the same
> > > polynomial
> > > > and the same initial value. The only difference is final xor mask
> > > > (0x for java.lang.zip.CRC32). So, we can easily convert from
> > > > PureJavaCrc32
> > > > to standard CRC32 and vice versa, using this expression: crc32 ^=
> > > > 0x
> > > >
> > > >
> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev <
> > eduard.shangar...@gmail.com
> > > >:
> > > >
> > > > > Evgeniy,
> > > > >
> > > > > Could you share benchmark code? And please share what version of
> JVM
> > > > > you have used.
> > > > >
> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya  >
> > > > > wrote:
> > > > >
> > > > > > I think it would break backward compatibility, as Nikolay
> mentioned
> > > > above
> > > > > > we would take exception here:
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > > https://github.com/apache/ignite/blob/master/modules/
> > > > > core/src/main/java/org/apache/ignite/internal/processors/
> > > > > cache/persistence/file/FilePageStore.java#L372
> > > > > >
> > > > > > thats why i question for community thoughts here.
> > > > > >
> > > > > > > Hi Evgeniy,
> > > > > > >
> > > > > > > would you like to submit a patch with CRC32 implementation
> > change?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский
> > > > > > > :
> > > > > > >
> > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32
> > has
> > > > > > >> performance problems in compatible with zip.CRC32.
> > > > > > >>
> > > > > > >> Benchmark Mode Cnt Score Error Units
> > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op
> > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210
> > ns/op
> > > > > > >>
> > > > > > >> thoughts?
> > > > > >
> > > > >
> > > >
> > >
> >
>


Re: Breaking change to spatial indexes to migrate to latest H2 version

2018-08-14 Thread Dmitriy Setrakyan
I tend to agree with Vladimir and Sergey. I would upgrade to the new H2
version and document this incompatibility with spatial indexes.

D.


On Tue, Aug 14, 2018 at 3:11 AM, Sergey Kozlov  wrote:

> Vladimir
>
> The spatial module is not delivering in AI binaries (and does not provided
> in AI maven repo). This the impact will be less than if it happended for
> other modules.
>
>
>
> On Tue, Aug 14, 2018 at 10:46 AM, Vladimir Ozerov 
> wrote:
>
> > Dmitriy,
> >
> > Spatial indexes is one of our features, packed as separate optional
> module
> > [1]. With proposed changes users of this feature will not be able to
> > migrate to 2.7 until they rewrite and recompile their code, because both
> > compile-time and run-time compatibility will be broken. Required changes
> to
> > the code are minimal - just change package names.
> >
> > If there are no concerns, I will merge it to master branch in the nearest
> > time.
> >
> > [1] https://apacheignite-sql.readme.io/docs/geospatial-support
> >
> > On Thu, Jun 7, 2018 at 4:00 AM Dmitriy Setrakyan 
> > wrote:
> >
> > > Vladimir,
> > >
> > > Can you please explain how our users will be affected? What does it
> mean
> > to
> > > use "spatial" indexes in Ignite?
> > >
> > > D.
> > >
> > > On Wed, Jun 6, 2018 at 12:54 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > New H2 version 1.4.197 [1] was released recently. It contain a lot of
> > > > changes (>1000 commits) and some of them are very useful for us. Of
> > most
> > > > importance is IN clause optimization which is currently one of our
> SQL
> > > pain
> > > > points.
> > > >
> > > > Unfortunately, new version use updated dependency for spatial
> indexes.
> > > > Earlier it was "org.vividsolutions", now it is "org.locationtech".
> This
> > > is
> > > > the same product, only package name was changed.
> > > >
> > > > It means that if we upgrade to newer H2 version all our users of
> > spatial
> > > > indexes feature will have compilation and/or linkage errors. This is
> a
> > > > breaking change.
> > > >
> > > > I propose to implement it still in AI 2.7. We still depend on H2
> > heavily
> > > > and cannot stop updates.
> > > >
> > > > Any objections?
> > > >
> > > > Vladimir.
> > > >
> > > > [1]
> > > https://github.com/h2database/h2database/releases/tag/version-1.4.197
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-4150
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>


Re: Service grid redesign

2018-08-14 Thread Dmitriy Setrakyan
On Tue, Aug 14, 2018 at 6:10 AM, Nikita Amelchev 
wrote:

> Hello, Igniters.
>
> I am working on task [1] that would replace serialized service's instance
> by service's class name and properties map in {ServiceConfiguration}.
>
> The task describes that we should use
> {String className} + {Map properties} instead {Service
> srvc}.
>
> I'd like to clarify the following questions:
>
> 1. What about public methods?
> I suggest to mark them as deprecated and use class name of provided
> instance.
> Also to add deploying methods with new parameters:
>
> @Deprecated
> public IgniteInternalFuture deployNodeSingleton(ClusterGroup prj,
> String
> name, Service svc)
>
> public IgniteInternalFuture deployNodeSingleton(ClusterGroup prj,
> String
> name, String srvcClsName, Map prop)
>

I think this makes sense, but I would like other committers to confirm.
Perhaps Vladimir Ozerov should comment here.


> 2. Is {Map properties} parameter mandatory when deploying a
> service?
> Is it make sense to add deploying methods without it? For example:
>
> public IgniteInternalFuture deployNodeSingleton(ClusterGroup prj,
> String
> name, String srvcClsName)
>
> public IgniteInternalFuture deployNodeSingleton(ClusterGroup prj,
> String
> name, String srvcClsName, Map prop)
>

I would always ask the user to pass the property map, but would allow it to
be null.

D.


Re: Does Ignite support java.sql.Array?

2018-08-14 Thread Dmitriy Setrakyan
On Tue, Aug 14, 2018 at 3:34 AM, Vladimir Ozerov 
wrote:

> Hi,
>
> No, we do not support ARRAY data type at the moment. Let's remove it from
> docs.
>

Vladimir, what is the reason for not supporting it? ARRAY is a basic SQL
type and not supporting it presents a serious limitation.

D.


Re: MVCC and IgniteDataStreamer

2018-08-14 Thread Dmitriy Setrakyan
On Tue, Aug 14, 2018 at 4:30 AM, Vladimir Ozerov 
wrote:

> Bypassing WAL will make the whole cache data vulnerable to complete loss in
> case of node failure. I would not do this automatically.
>

Well, in this case I would expect a log message suggesting that there is an
option to turn off WAL which could significantly improve performance. We
could just print it out once.


Re: Service grid redesign

2018-08-09 Thread Dmitriy Setrakyan
Versions will complicate the implementation and will not be done in 2.7. I
would vote for the hot redeployment for now and add versions in 2.8.

D.

On Thu, Aug 9, 2018 at 10:06 AM, Anton Vinogradov  wrote:

> Real case is A/B testing.
> When you want to allow new service usage only to 0.1% of users.
> And only when you sure it works then replace all v1 with v2.
>
> So, I vote for versions.
> Let's do this in maven way (exact version, range, RELEASE or LATEST)
>
> чт, 9 авг. 2018 г. в 17:55, Dmitriy Setrakyan :
>
> > Vyacheslav,
> >
> > For the case you are describing, I would take the same approach as we
> have
> > for compute tasks. Keep the older version around only as long as there
> are
> > active requests and then undeploy it automatically. No need to allow it
> > linger around indefinitely.
> >
> > D.
> >
> > On Thu, Aug 9, 2018 at 9:52 AM, Vyacheslav Daradur 
> > wrote:
> >
> > > Dmitry, it's not only about hot redeployment.
> > >
> > > Denis, I don't see such use case, because of the answer in a different
> > > front.
> > >
> > > It relates to the best practices of SOA versioning [1] [2].
> > >
> > > For example:
> > > * we have a cluster with service [name="MySevice", v="1"];
> > > * I want to upgrade service to [name="MySevice", v="2"], but I have
> > > clients which are using [name="MySevice", v="1"] and can't stop
> > > processing;
> > > * With service versioning, we are able to deploy new service near
> > > existing service, then switch clients and undeploy outdated service.
> > >
> > > My only question is: are we going to implement such a feature [3] or
> > > not? Maybe PMC don't see such feature in Service Grid roadmap.
> > > IMO it's a good feature for a microservices platform.
> > >
> > >
> > > [1] https://www.thbs.com/thbs-insights/soa-service-
> > > versioning-best-practices
> > > [2] https://www.ibm.com/blogs/bluemix/2017/08/rapidly-
> > > developing-applications-part-6-exposing-and-versioning-apis/
> > > [3] https://issues.apache.org/jira/browse/IGNITE-6069
> > > On Thu, Aug 9, 2018 at 5:48 PM Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > > wrote:
> > > >
> > > > Guys,
> > > >
> > > > I thought this was about automatic service redeployment, which should
> > > have
> > > > been a part of the current IEP, no? Can you please clarify?
> > > >
> > > > D.
> > > >
> > > > On Thu, Aug 9, 2018 at 9:26 AM, Denis Mekhanikov <
> > dmekhani...@gmail.com>
> > > > wrote:
> > > >
> > > > > Vyacheslav,
> > > > >
> > > > > It looks like an overcomplication to me.
> > > > > Could you describe a case, that can be solved using versioning, but
> > not
> > > > > naming?
> > > > >
> > > > > Denis
> > > > >
> > > > > чт, 9 авг. 2018 г. в 16:56, Vyacheslav Daradur <
> daradu...@gmail.com
> > >:
> > > > >
> > > > > > Denis, it's not about different users services implementations.
> > > > > >
> > > > > > A real use case is user's services API versioning which is being
> > used
> > > > > > widely t in SOAP/REST microservices infrastructure.
> > > > > >
> > > > > > In my opinion, it is about services with the same name and the
> same
> > > > > > full class name, but different classes versions for example in
> > > > > > different classloaders.
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 9, 2018 at 4:41 PM Denis Mekhanikov <
> > > dmekhani...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I don't think, that we really need this feature.
> > > > > > > It seems to me, that if you want to use a different
> > implementation
> > > of a
> > > > > > > service, you can assign a different name to it.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > чт, 9 авг. 2018 г. в 16:32, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>:
> > > > &

Re: ConcurrentLinkedHashMap works incorrectly after clear()

2018-08-09 Thread Dmitriy Setrakyan
On Thu, Aug 9, 2018 at 8:16 AM, Alexey Goncharuk  wrote:

> Guys, I think we can definitely change current implementation of CLHM with
> a more stable one, but as a temporal solution I see nothing wrong with
> throwing an UnsupportedOperationException from clear() method. Ilya already
> provided a patch which replaces all clear() calls with a new map creation,
> semantically it has the same behavior as a correct clear() method.
>
> I suggest to merge the provided PR because IgniteH2Indexing is broken
> because of this bug and then create a ticket to replace/fix/whatever it
> feels right to do with current CLHM.
>
> Thoughts?
>

I agree. This sounds like a less risky approach.

D.


Re: DataFrame integration does not support ARRAY type

2018-08-09 Thread Dmitriy Setrakyan
On Thu, Aug 9, 2018 at 8:19 AM, Nikolay Izhikov  wrote:

> Dmitriy, I will take care of this ticket in a few days.
>

Great!


Re: Data regions on client nodes

2018-08-09 Thread Dmitriy Setrakyan
Alexey,

I see your point. However, I would still argue that there are more benefits
to the dynamic region allocation than not. For example, if we have this
feature, users would be able to create more regions after the node and
cluster start, which will allow to grow the data size indefinitely in the
cloud.

D.

On Thu, Aug 9, 2018 at 8:41 AM, Alexey Goncharuk  wrote:

> Once the OS gave us the pointer, no exceptions can be raised in the user
> code. If the OS overcommitted the memory, then the process will be halted
> when OOME occurs, not much we can do here.
>
> My point was that dynamic data region allocation can lead to another
> dynamic exception that should be properly handled during cache start. When
> the data region is allocated statically, this exception is handled on node
> start, which is much easier.
>
> ср, 8 авг. 2018 г. в 18:18, Dmitriy Pavlov :
>
> > I used to think that OS allocates pages not immediately after call to
> > allocate(), but only during first touch of each page.
> >
> > I'm not sure every OS provides guaranee that 'allocated' memory will
> never
> > cause OOME. Please correct me if I'm wrong.
> >
> > ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan :
> >
> > > Alexey,
> > >
> > > I am not sure I understand. If cache creation proceeds, but memory
> region
> > > was not allocated, how can the cache operate?
> > >
> > > D.
> > >
> > > On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > wrote:
> > >
> > > > I do not mind making this change, but note that the reason for
> non-lazy
> > > > region allocation is the need to gracefully handle OOME errors during
> > > cache
> > > > start. When a region is pre-allocated, no OOME can happen.
> > > >
> > > > If a region is allocated dynamically, then all errors that happened
> > > during
> > > > the node start before should be properly handled (a client node must
> be
> > > > stopped, but cache creation should proceed).
> > > >
> > > > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <
> dpavlov@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Maxim, thank you.
> > > > > >
> > > > > > If it seems it is technically possible, we can file ticket for
> this
> > > > > change.
> > > > > >
> > > > > > I find this proposal reasonable, change makes perfectly sense to
> > me.
> > > > > >
> > > > > > We can wait Alex G. feedback on this change before starting
> actual
> > > > > > implementation. It can take for a while, because he is travelling
> > > now.
> > > > > >
> > > > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov  >:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I can miss some details, but at the first glance we have
> > everething
> > > > we
> > > > > > need
> > > > > > > to defer
> > > > > > > region memory allocation if it has no cache groups assignments.
> > And
> > > > it
> > > > > > > doesn't matter
> > > > > > > where it happens on client or server nodes.
> > > > > > >
> > > > > > > Currently region memory allocation happens at exchange future
> > init
> > > > > > method.
> > > > > > > At the
> > > > > > > node startup method initCachesOnLocalJoin executes. This method
> > > > > > resposnible
> > > > > > > for
> > > > > > > memory allocation (through initiating cache managers) and it
> also
> > > > > starts
> > > > > > > caches.
> > > > > > > So, at this point we have all existing caches descriptors and
> can
> > > > find
> > > > > > out
> > > > > > > which
> > > > > > > cache matches which region to defer some regions initialization
> 

Re: Service grid redesign

2018-08-09 Thread Dmitriy Setrakyan
Vyacheslav,

For the case you are describing, I would take the same approach as we have
for compute tasks. Keep the older version around only as long as there are
active requests and then undeploy it automatically. No need to allow it
linger around indefinitely.

D.

On Thu, Aug 9, 2018 at 9:52 AM, Vyacheslav Daradur 
wrote:

> Dmitry, it's not only about hot redeployment.
>
> Denis, I don't see such use case, because of the answer in a different
> front.
>
> It relates to the best practices of SOA versioning [1] [2].
>
> For example:
> * we have a cluster with service [name="MySevice", v="1"];
> * I want to upgrade service to [name="MySevice", v="2"], but I have
> clients which are using [name="MySevice", v="1"] and can't stop
> processing;
> * With service versioning, we are able to deploy new service near
> existing service, then switch clients and undeploy outdated service.
>
> My only question is: are we going to implement such a feature [3] or
> not? Maybe PMC don't see such feature in Service Grid roadmap.
> IMO it's a good feature for a microservices platform.
>
>
> [1] https://www.thbs.com/thbs-insights/soa-service-
> versioning-best-practices
> [2] https://www.ibm.com/blogs/bluemix/2017/08/rapidly-
> developing-applications-part-6-exposing-and-versioning-apis/
> [3] https://issues.apache.org/jira/browse/IGNITE-6069
> On Thu, Aug 9, 2018 at 5:48 PM Dmitriy Setrakyan 
> wrote:
> >
> > Guys,
> >
> > I thought this was about automatic service redeployment, which should
> have
> > been a part of the current IEP, no? Can you please clarify?
> >
> > D.
> >
> > On Thu, Aug 9, 2018 at 9:26 AM, Denis Mekhanikov 
> > wrote:
> >
> > > Vyacheslav,
> > >
> > > It looks like an overcomplication to me.
> > > Could you describe a case, that can be solved using versioning, but not
> > > naming?
> > >
> > > Denis
> > >
> > > чт, 9 авг. 2018 г. в 16:56, Vyacheslav Daradur :
> > >
> > > > Denis, it's not about different users services implementations.
> > > >
> > > > A real use case is user's services API versioning which is being used
> > > > widely t in SOAP/REST microservices infrastructure.
> > > >
> > > > In my opinion, it is about services with the same name and the same
> > > > full class name, but different classes versions for example in
> > > > different classloaders.
> > > >
> > > >
> > > > On Thu, Aug 9, 2018 at 4:41 PM Denis Mekhanikov <
> dmekhani...@gmail.com>
> > > > wrote:
> > > > >
> > > > > I don't think, that we really need this feature.
> > > > > It seems to me, that if you want to use a different implementation
> of a
> > > > > service, you can assign a different name to it.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Denis
> > > > >
> > > > > чт, 9 авг. 2018 г. в 16:32, Dmitriy Setrakyan <
> dsetrak...@apache.org>:
> > > > >
> > > > > > On Thu, Aug 9, 2018 at 4:41 AM, Vyacheslav Daradur <
> > > > daradu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, Igniters!
> > > > > > >
> > > > > > > I found a ticket about a service’s versioning [1].
> > > > > > >
> > > > > > > It’s out of scope IEP-17, but if we are going to implement this
> > > > > > > feature we should build a base in the first iteration of IEP-17
> > > > > > > because of change messages formats.
> > > > > > >
> > > > > > > In case of the versioning which assumes that we are able to
> host
> > > > > > > services with the same name, but with different class/version,
> we
> > > > > > > should introduce *service’s id* to manage service’s lifecycle
> > > instead
> > > > > > > of service’s name.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > >
> > > > > > My only concern would be on the usability side. Is user going to
> have
> > > > to
> > > > > > deal with IDs now, or will it be handled internally?
> > > > > >
> > > > > > D.
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Vyacheslav D.
> > > >
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
>


Re: Service grid redesign

2018-08-09 Thread Dmitriy Setrakyan
Guys,

I thought this was about automatic service redeployment, which should have
been a part of the current IEP, no? Can you please clarify?

D.

On Thu, Aug 9, 2018 at 9:26 AM, Denis Mekhanikov 
wrote:

> Vyacheslav,
>
> It looks like an overcomplication to me.
> Could you describe a case, that can be solved using versioning, but not
> naming?
>
> Denis
>
> чт, 9 авг. 2018 г. в 16:56, Vyacheslav Daradur :
>
> > Denis, it's not about different users services implementations.
> >
> > A real use case is user's services API versioning which is being used
> > widely t in SOAP/REST microservices infrastructure.
> >
> > In my opinion, it is about services with the same name and the same
> > full class name, but different classes versions for example in
> > different classloaders.
> >
> >
> > On Thu, Aug 9, 2018 at 4:41 PM Denis Mekhanikov 
> > wrote:
> > >
> > > I don't think, that we really need this feature.
> > > It seems to me, that if you want to use a different implementation of a
> > > service, you can assign a different name to it.
> > >
> > > What do you think?
> > >
> > > Denis
> > >
> > > чт, 9 авг. 2018 г. в 16:32, Dmitriy Setrakyan :
> > >
> > > > On Thu, Aug 9, 2018 at 4:41 AM, Vyacheslav Daradur <
> > daradu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, Igniters!
> > > > >
> > > > > I found a ticket about a service’s versioning [1].
> > > > >
> > > > > It’s out of scope IEP-17, but if we are going to implement this
> > > > > feature we should build a base in the first iteration of IEP-17
> > > > > because of change messages formats.
> > > > >
> > > > > In case of the versioning which assumes that we are able to host
> > > > > services with the same name, but with different class/version, we
> > > > > should introduce *service’s id* to manage service’s lifecycle
> instead
> > > > > of service’s name.
> > > > >
> > > > > Thoughts?
> > > > >
> > > >
> > > > My only concern would be on the usability side. Is user going to have
> > to
> > > > deal with IDs now, or will it be handled internally?
> > > >
> > > > D.
> > > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>


Re: Service grid redesign

2018-08-09 Thread Dmitriy Setrakyan
On Thu, Aug 9, 2018 at 4:41 AM, Vyacheslav Daradur 
wrote:

> Hi, Igniters!
>
> I found a ticket about a service’s versioning [1].
>
> It’s out of scope IEP-17, but if we are going to implement this
> feature we should build a base in the first iteration of IEP-17
> because of change messages formats.
>
> In case of the versioning which assumes that we are able to host
> services with the same name, but with different class/version, we
> should introduce *service’s id* to manage service’s lifecycle instead
> of service’s name.
>
> Thoughts?
>

My only concern would be on the usability side. Is user going to have to
deal with IDs now, or will it be handled internally?

D.


Re: DataFrame integration does not support ARRAY type

2018-08-08 Thread Dmitriy Setrakyan
Would be nice if someone in the community would pick this ticket up.

On Wed, Aug 8, 2018 at 1:19 AM, Nikolay Izhikov  wrote:

> Hello, Valentin.
>
> Yes, this is a bug.
>
> Ticket created - https://issues.apache.org/jira/browse/IGNITE-9229
>
> В Вт, 31/07/2018 в 16:01 -0700, Valentin Kulichenko пишет:
> > Hi Nikolay,
> >
> > Can you please take a look at this thread on SO?
> https://stackoverflow.com/questions/51621280/saving-a-
> spark-dataset-to-apache-ignite-with-array-column-and-savemode-overwrite
> >
> > Looks like org.apache.ignite.spark.impl.QueryUtils#dataType method
> should also support ArrayType as one of the cases. Currently it doesn't and
> throws an exception.
> >
> > Is it a bug?
> >
> > -Val
>


Re: Metrics for MVCC caches

2018-08-08 Thread Dmitriy Setrakyan
I think we should be updating the IEP with metrics specifications in
parallel.

D.

On Wed, Aug 8, 2018, 05:32 Roman Kondakov 
wrote:

> Hi Ivan!
>
> In my opinion we need to preserve the essence of the metrics: if we
> didn't consider dirty writes as updates before MVCC implementation, we
> also shouldn't count these writes in MVCC metrics implementation too.
> So, I think we need to count only committed entries. But we can count
> this dirty writes as additional metrics.
>
> Also additional metrics concerning MVCC could be:
>
> - Average count of the active transactions per snapshot
>
> - Average quantity of versions per key
>
>
> --
>
> Roman Kondakov
>
>
> On 07.08.2018 17:17, Павлухин Иван wrote:
> > Hi Igniters,
> >
> > I am working on cache metrics support for caches with enabled MVCC. As
> you
> > may know, during MVCC transaction execution entry versions are written to
> > storage right away (without deferring until commit). So, it is not
> obvious
> > for me if we should update writes count right away or defer it until
> > commit. On one hand, MVCC entry version write is not "true" write until
> > commit. On the other hand, it definitely changes the storage. How do we
> > interpret write metrics in Ignite philosophy?
> >
> > Also, it seems that bunch of MVCC specific metrics could be introduced. I
> > would like to collect some thoughts about it. Which such metrics come to
> > your mind?
> >
>
>


Re: [MTCGA]: new failures in builds [1570331] needs to be handled

2018-08-08 Thread Dmitriy Setrakyan
Dmitriy, yes, if you set it as blocker and assign it to the next release,
the likelihood of someone looking at it before the release will increase.
We should follow this practice for all new test failures.

Moreover, if someone's change caused a new test failure, then we should try
to contact that person on the dev list and bring it to his/her attention.

D.

On Wed, Aug 8, 2018 at 7:25 AM, Dmitriy Pavlov 
wrote:

> Hi Dmitriy,
>
> Would it increase likelihood that somebody will pick up this ticket if we
> will set blocker priority?
>
> If it would increase I totally agree. IMO, JIRA priorities do not have much
> effect on community members, do it?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 7 авг. 2018 г. в 14:00, Dmitriy Setrakyan :
>
> > Maxim, if it is a new failure, why is it filed as Minor and is not
> assigned
> > to any release? I would suggest that any new failure should be filed as a
> > Blocker and assigned to the next release.
> >
> > D.
> >
> > On Tue, Aug 7, 2018 at 12:55 AM, Maxim Muzafarov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Seems this is a new failures due to last changes.
> > > I've created new issue [1].
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9202
> > >
> > > On Fri, 3 Aug 2018 at 20:57  wrote:
> > >
> > > > Hi Ignite Developer,
> > > >
> > > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be
> > addressed.
> > > > I hope you can help.
> > > >
> > > >  *New test failure in master
> > > > ExamplesTest.TestRemoteNodes(BinaryModeExample)
> > > > https://ci.ignite.apache.org/project.html?projectId=
> > > IgniteTests24Java8=-3155722801840665529=%
> > > 3Cdefault%3E=testDetails
> > > >
> > > >  *New test failure in master ExamplesTest.TestRemoteNodes(
> > > LinqExample)
> > > > https://ci.ignite.apache.org/project.html?projectId=
> > > IgniteTests24Java8=8941219717117543602=%
> > > 3Cdefault%3E=testDetails
> > > >
> > > >  *New test failure in master ExamplesTest.TestRemoteNodes(
> > > SqlExample)
> > > > https://ci.ignite.apache.org/project.html?projectId=
> > > IgniteTests24Java8=-61288549283153227=%
> > > 3Cdefault%3E=testDetails
> > > >  Changes may led to failure were done by
> > > >  - nsamelchev
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827132=false
> > > >  - biryukovvitaliy92
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827130=false
> > > >  - ppozerov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827122=false
> > > >  - ppozerov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827121=false
> > > >  - dmitrievanthony
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827116=false
> > > >  - eshangareev
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827112=false
> > > >  - kaa.dev
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827106=false
> > > >  - alkuznetsov.sb
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827101=false
> > > >  - akuznetsov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827091=false
> > > >  - akuznetsov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827089=false
> > > >  - dmitriyff
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827084=false
> > > >  - ppozerov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827066=false
> > > >  - vsisko
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827036=false
> > > >  - akuznetsov
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827034=false
> > > >  - verbalab
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827032=false
> > > >  - verbalab
> > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > 827030=false
> > > >

Re: Data regions on client nodes

2018-08-08 Thread Dmitriy Setrakyan
Alexey,

I am not sure I understand. If cache creation proceeds, but memory region
was not allocated, how can the cache operate?

D.

On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk  wrote:

> I do not mind making this change, but note that the reason for non-lazy
> region allocation is the need to gracefully handle OOME errors during cache
> start. When a region is pre-allocated, no OOME can happen.
>
> If a region is allocated dynamically, then all errors that happened during
> the node start before should be properly handled (a client node must be
> stopped, but cache creation should proceed).
>
> пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > -Val
> >
> > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov 
> > wrote:
> >
> > > Maxim, thank you.
> > >
> > > If it seems it is technically possible, we can file ticket for this
> > change.
> > >
> > > I find this proposal reasonable, change makes perfectly sense to me.
> > >
> > > We can wait Alex G. feedback on this change before starting actual
> > > implementation. It can take for a while, because he is travelling now.
> > >
> > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov :
> > >
> > > > Guys,
> > > >
> > > > I can miss some details, but at the first glance we have everething
> we
> > > need
> > > > to defer
> > > > region memory allocation if it has no cache groups assignments. And
> it
> > > > doesn't matter
> > > > where it happens on client or server nodes.
> > > >
> > > > Currently region memory allocation happens at exchange future init
> > > method.
> > > > At the
> > > > node startup method initCachesOnLocalJoin executes. This method
> > > resposnible
> > > > for
> > > > memory allocation (through initiating cache managers) and it also
> > starts
> > > > caches.
> > > > So, at this point we have all existing caches descriptors and can
> find
> > > out
> > > > which
> > > > cache matches which region to defer some regions initialization to
> the
> > > > moment when
> > > > newly cache assings to this region (happend at onCacheChangeRequest).
> > > >
> > > > Please, сorrect me if I'm wrong and missing something.
> > > >
> > > >
> > > >
> > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov 
> > > wrote:
> > > >
> > > > > Hi Maxim,
> > > > >
> > > > > thank you for stepping in. How do you think, is it possible to
> check
> > > > cache
> > > > > assignment to region at stage of memory allocation?
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov :
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > I've checked memory allocation. It looks like we are allocating
> > > memory
> > > > > only
> > > > > > on the first exchange future init on local join occurs on node.
> > Also,
> > > > > seems
> > > > > > like we are allocating only the first chunk of memory (not the
> > whole
> > > > > bunch)
> > > > > > and it calculates as:
> > > > > >
> > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 *
> 1024)
> > > > > >
> > > > > > But, I'm agree with Val. It's better to allocate memory only when
> > > when
> > > > > > the first cache assigned to this region.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Also, It seems like we have some problem with user notification
> > about
> > > > > > available
> > > > > > physical resources. For client nodes method requiredOffheap()
> > returns
> > > > > > always
> > > > > > zero [1]. That's why WARN message shown here [2] would be not not
> > > quite
> > > > > > right
> > > > > > if we have a lot of client nodes in cluster.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > >

Re: Metrics for MVCC caches

2018-08-07 Thread Dmitriy Setrakyan
Ivan,

To your 2nd question, can you suggest a few MVCC metrics to get the
discussion started?

D.

On Tue, Aug 7, 2018 at 9:17 AM, Павлухин Иван  wrote:

> Hi Igniters,
>
> I am working on cache metrics support for caches with enabled MVCC. As you
> may know, during MVCC transaction execution entry versions are written to
> storage right away (without deferring until commit). So, it is not obvious
> for me if we should update writes count right away or defer it until
> commit. On one hand, MVCC entry version write is not "true" write until
> commit. On the other hand, it definitely changes the storage. How do we
> interpret write metrics in Ignite philosophy?
>
> Also, it seems that bunch of MVCC specific metrics could be introduced. I
> would like to collect some thoughts about it. Which such metrics come to
> your mind?
>
> --
> Best regards,
> Ivan Pavlukhin
>


Re: [MTCGA]: new failures in builds [1570331] needs to be handled

2018-08-07 Thread Dmitriy Setrakyan
Maxim, if it is a new failure, why is it filed as Minor and is not assigned
to any release? I would suggest that any new failure should be filed as a
Blocker and assigned to the next release.

D.

On Tue, Aug 7, 2018 at 12:55 AM, Maxim Muzafarov  wrote:

> Igniters,
>
> Seems this is a new failures due to last changes.
> I've created new issue [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9202
>
> On Fri, 3 Aug 2018 at 20:57  wrote:
>
> > Hi Ignite Developer,
> >
> > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> > I hope you can help.
> >
> >  *New test failure in master
> > ExamplesTest.TestRemoteNodes(BinaryModeExample)
> > https://ci.ignite.apache.org/project.html?projectId=
> IgniteTests24Java8=-3155722801840665529=%
> 3Cdefault%3E=testDetails
> >
> >  *New test failure in master ExamplesTest.TestRemoteNodes(
> LinqExample)
> > https://ci.ignite.apache.org/project.html?projectId=
> IgniteTests24Java8=8941219717117543602=%
> 3Cdefault%3E=testDetails
> >
> >  *New test failure in master ExamplesTest.TestRemoteNodes(
> SqlExample)
> > https://ci.ignite.apache.org/project.html?projectId=
> IgniteTests24Java8=-61288549283153227=%
> 3Cdefault%3E=testDetails
> >  Changes may led to failure were done by
> >  - nsamelchev
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827132=false
> >  - biryukovvitaliy92
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827130=false
> >  - ppozerov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827122=false
> >  - ppozerov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827121=false
> >  - dmitrievanthony
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827116=false
> >  - eshangareev
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827112=false
> >  - kaa.dev
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827106=false
> >  - alkuznetsov.sb
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827101=false
> >  - akuznetsov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827091=false
> >  - akuznetsov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827089=false
> >  - dmitriyff
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827084=false
> >  - ppozerov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827066=false
> >  - vsisko
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827036=false
> >  - akuznetsov
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827034=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827032=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827030=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827028=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827027=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827025=false
> >  - verbalab
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 827023=false
> >
> > - If your changes can led to this failure(s), please create issue
> > with label MakeTeamCityGreenAgain and assign it to you.
> > -- If you have fix, please set ticket to PA state and write to
> dev
> > list fix is ready
> > -- For case fix will require some time please mute test and set
> > label Muted_Test to issue
> > - If you know which change caused failure please contact change
> > author directly
> > - If you don't know which change caused failure please send
> > message to dev list to find out
> > Should you have any questions please contact dpav...@apache.org or write
> > to dev.list
> > Best Regards,
> > MTCGA.Bot
> > Notification generated at Fri Aug 03 20:57:43 MSK 2018
> >
> --
> --
> Maxim Muzafarov
>


Re: Page replacement policy improvements (when persistent is enabled)

2018-08-03 Thread Dmitriy Setrakyan
Vladimir,

Are we only counting timestamp of the last access? In that case, it would
create a problem. We should also count number of times a page has been
touched within a certain time frame, e.g. last hour or so. In this case,
index pages would not be evicted as they get touched the most.

I would also consider putting index pages into a separate memory region.
This way you can apply a different eviction policy to the index pages or
decide not to evict them altogether. This will also be a much simpler and
less error-prone approach than introducing new eviction policies.

D.

On Fri, Aug 3, 2018 at 12:19 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> I heard some complaints about our page replacement algorithm that index
> pages could be evicted from memory too often. I reviewed our current
> implementation and looks like we have choosen very simple approach with
> eviction of random pages, without taking in count their nature (data vs
> index) and typical usage patterns (such as scans).
>
> With our heap-based architecture typical SQL query is executed as follows:
> 1) Read non-leaf index pages, then in loop:
> 2.1) Read 1 leaf index page
> 2.2) Read several hunderds data pages
>
> This way index pages on average has smaller timestamp than data pages and
> has good probabilty of being evicted.
>
> Another major problem is scan resistance, which doesn't seem to be covered
> anyhow.
>
> My question is - what was the reason of choosing random-pseudo-LRU
> algorithm instead of commonly used variation of *real* LRU (such as LRU-K,
> 2Q, etc)? Did we perform any evaluation of it's effectiveness?
>
> I am thinking of creating new IEP to evaluate and possibly improve our page
> replacement as follows:
> 1) Implement metrics to count page cache hit/miss by page type [1]
> 2) Implement *heat map* which can optionally be enabled to track page
> hits/misses per page or per specific object (cache, index)
> 3) Run heat map on typical workloads (lookups, scans, joins, etc) to get a
> baseline
> 4) Prototype several LRU-based implementation and see if they gave any
> benefit. It makes sense to start with minor improvements to current
> algorithm (e.g. favor index pages over data pages, play with sample size,
> replace timestamps with read counters, etc).
>
> In any case the first two action items would be good addition to product
> monitoring.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8580
>


Re: Ignite as distributed file storage

2018-08-03 Thread Dmitriy Setrakyan
Pavel,

Not everything that gets put in Ignite has a class, and not everything can
be annotated. For example, what about byte[] or a binary object?

I would prefer a separate cache with specific purpose of storing blobs. It
will be easier for users to understand and configure.  It can also have a
custom logic of splitting a blob into multiple batches to be able to
transfer it over the network faster.

D.


On Fri, Aug 3, 2018 at 3:21 AM, Pavel Kovalenko  wrote:

> Dmitriy,
>
> I think we don't need a separate implementation of cache like BLOB cache.
> Instead of it, a user can mark value class or value class field with the
> special annotation "@BLOB".
> During cache put, marshaller will place a special placeholder on such
> fields, write byte[] array payload of a field to special internal blob
> storage and place the only reference to actual DataEntry in the page
> memory.
> During cache get, marshaller will place a special proxy instead of an
> actual class that can be downloaded and unmarshalled by demand from
> internal storage on the user side.
> Using such approach we will also solve eager/lazy problem, this will also
> give user possibility to adjust his own marshallers (like Jackson, Jaxb,
> etc.) to marshal/unmarshal his blob classes from/to byte[] arrays.
> No major changes in public API are required, it can be pluggable component.
>
>
> 2018-08-03 0:25 GMT+03:00 Dmitriy Setrakyan :
>
> > On Thu, Aug 2, 2018 at 1:08 AM, Pavel Kovalenko 
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I still don't understand why do you think that it will be file system?
> > > In all my previous messages I emphasized that this storage shouldn't be
> > > considered as a file system. It's just a large data storage, whose
> > entities
> > > can be easily accessed using key/link (internally, or externally using
> > > web/binary protocol interfaces).
> > >
> > > > Instead, if we must focus on large blobs, I would solve the problem
> of
> > > supporting large blobs in regular Ignite caches, as I suggested before.
> > >
> > > This is impossible. Our page memory can't handle efficiently it by
> > design.
> > >
> >
> > But our API does. What is stopping us from creating a cache as a BLOB
> cache
> > and using whatever storage we need?
> >
>


Re: Code inspection

2018-08-03 Thread Dmitriy Setrakyan
On Fri, Aug 3, 2018 at 7:49 AM, Dmitriy Pavlov 
wrote:

>
> I understand it is not so Apache-way from my side to ask volunteers to do
> some things (instead of contributing it by myself).


Dmitriy, I am not sure why you feel this is not the Apache way. No one can
do everything themselves.  You should absolutely keep recruiting more
volunteers from the community.

D.


Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Dmitriy Setrakyan
Agree with Denis. Let's do the simple refactoring first, and then more
complicated one in phase 2.

On Fri, Aug 3, 2018 at 6:47 AM, Denis Magda  wrote:

> >
> > Thus, my suggestion is:
> >  — find and update all hardcoded “fabric” usage on TC, so that they work
> > both with or without fabric in name of binaries
> >  — use current implementation — review and merge to master
> >  — plan full suffix refactoring (as Anton suggests) on the next
> iterations
> > with no rush
>
>
> I like this plan which allows us to do the things. I see the current
> implementation as one of the steps of the overall refactoring proposed by
> Anton. It sounds normal if we split refactoring into pieces.
>
> Anton, do you have time to help with the rest of refactoring tasks before
> AI 2.7? It's great if we close the whole tasks by that time. Otherwise,
> let's split it and release the current implementation first.
>
> --
> Denis
>
> On Fri, Aug 3, 2018 at 1:18 AM Petr Ivanov  wrote:
>
> > Dmitriy,
> >
> > I cannot forecast estimates for this task as it dependents on many
> factors:
> >  — I will be able to start researching the Anton’s implementation
> > suggestion not earlier than the beginning of September
> >  — I am not acquainted with assembly configuration well, it may take some
> > considerable time to understand how correctly get rid of “fabric” not
> > touching everything else
> >  — the process of review and merge can also drag on indefinitely (based
> on
> > the previous attempt to introduce this changes)
> >
> >
> > Vladimir,
> >
> > If community will approve this hack, I’ll implement it.
> > Yet it won’t resolve the problem of building from sources not on TC — the
> > fabric will stay in names of binaries and folders inside.
> > And it will add problems when the correct implementation will be
> > introduced.
> >
> >
> >
> > Thus, my suggestion is:
> >  — find and update all hardcoded “fabric” usage on TC, so that they work
> > both with or without fabric in name of binaries
> >  — use current implementation — review and merge to master
> >  — plan full suffix refactoring (as Anton suggests) on the next
> iterations
> > with no rush
> >
> >
> >
> > > On 3 Aug 2018, at 09:50, Vladimir Ozerov  wrote:
> > >
> > > Folks,
> > >
> > > Can you please explain the problem with TC and artifacts? Can we just
> > > rename final artifact at the end of a build phase just before signing,
> > and
> > > leave the rest TC infrastructure as is?
> > >
> > > On Fri, Aug 3, 2018 at 12:28 AM Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > >> Anton, Petr,
> > >>
> > >> Thanks for your readiness to assist. Can this be done for 2.7 release?
> > >>
> > >> D.
> > >>
> > >> On Thu, Aug 2, 2018 at 1:32 AM, Anton Vinogradov 
> wrote:
> > >>
> > >>> What I see is that we spent almost a year discussing how to do this.
> > >>> I'm pretty sure we had enough time to do everything properly.
> > >>>
> > >>> So, proposal is to stop this discussion and start refactoring.
> > >>>
> > >>> I do not see any pitfalls and ready to assist if necessary.
> > >>>
> > >>> чт, 2 авг. 2018 г. в 5:14, Dmitriy Setrakyan  >:
> > >>>
> > >>>> I vote to remove the fabric from the build in the easiest way
> > possible.
> > >>> Can
> > >>>> other Igniters comment?
> > >>>>
> > >>>> D.
> > >>>>
> > >>>> On Wed, Aug 1, 2018 at 12:46 PM, Petr Ivanov 
> > >>> wrote:
> > >>>>
> > >>>>> My concern here is exactly about internal build processes —
> removing
> > >>>>> fabric from the name of binary archive (with any way) will break
> lots
> > >>> of
> > >>>>> them.
> > >>>>> There will be no sacrifices, just lots of work for fixing build
> > >>> processes
> > >>>>> (where we won’t be able to introduce changes proactively).
> > >>>>>
> > >>>>> Therefore only fabric removal implementation (quick with some
> legacy
> > >>> left
> > >>>>> or full refactoring) is on the agenda.
> > >>>>> And this matter should be jugged by the commu

Re: Removing "fabric" from Ignite binary package name

2018-08-02 Thread Dmitriy Setrakyan
Anton, Petr,

Thanks for your readiness to assist. Can this be done for 2.7 release?

D.

On Thu, Aug 2, 2018 at 1:32 AM, Anton Vinogradov  wrote:

> What I see is that we spent almost a year discussing how to do this.
> I'm pretty sure we had enough time to do everything properly.
>
> So, proposal is to stop this discussion and start refactoring.
>
> I do not see any pitfalls and ready to assist if necessary.
>
> чт, 2 авг. 2018 г. в 5:14, Dmitriy Setrakyan :
>
> > I vote to remove the fabric from the build in the easiest way possible.
> Can
> > other Igniters comment?
> >
> > D.
> >
> > On Wed, Aug 1, 2018 at 12:46 PM, Petr Ivanov 
> wrote:
> >
> > > My concern here is exactly about internal build processes — removing
> > > fabric from the name of binary archive (with any way) will break lots
> of
> > > them.
> > > There will be no sacrifices, just lots of work for fixing build
> processes
> > > (where we won’t be able to introduce changes proactively).
> > >
> > > Therefore only fabric removal implementation (quick with some legacy
> left
> > > or full refactoring) is on the agenda.
> > > And this matter should be jugged by the community: currently we have
> (if
> > > our voices are equal) 1:1 with Anton about it.
> > >
> > >
> > >
> > >
> > > > On 1 Aug 2018, at 22:28, Dmitriy Setrakyan 
> > > wrote:
> > > >
> > > > Let's focus on what is important here. Our users do not care about
> our
> > > > internal build process.If we could remove the word fabric from the
> next
> > > > release without any significant sacrifices in the build process or
> > making
> > > > it less maintainable, I suggest we do it.
> > > >
> > > > D.
> > > >
> > > > On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov 
> > > wrote:
> > > >
> > > >> Simple way with some hack and legacy maintenance: accept patch as it
> > is
> > > >> implemented now.
> > > >> Hard way: full assembly refactoring and hadoop rejection.
> > > >>
> > > >> Anyway, after this is merged to master — complete automation systems
> > > >> revision (TeamCity for example) is required due to heavy hardcode of
> > > >> “fabric” in such systems.
> > > >>
> > > >>
> > > >>> On 1 Aug 2018, at 21:55, Dmitriy Setrakyan 
> > > >> wrote:
> > > >>>
> > > >>> OK, so what is the plan? How do we get rid of the fabric name?
> > > >>>
> > > >>> D.
> > > >>>
> > > >>> On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov 
> > > wrote:
> > > >>>
> > > >>>> Since you proposing patch to the community, you are the very man
> :)
> > > >>>>
> > > >>>> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov :
> > > >>>>
> > > >>>>> You are convincing the wrong person.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 1 Aug 2018, at 12:05, Anton Vinogradov 
> wrote:
> > > >>>>>>
> > > >>>>>> Peter,
> > > >>>>>>
> > > >>>>>> We had a discussion about how to do this properly.
> > > >>>>>> Proposed solution cannot be merged, since it makes code harder
> > than
> > > it
> > > >>>>> was.
> > > >>>>>>
> > > >>>>>> The only case is to perform complete refactoring and get rid of
> > all
> > > >>>>>> postfixes and other weird stuff.
> > > >>>>>>
> > > >>>>>> For example
> > > >>>>>> - 
> > > >>>>>> - 
> > > >>>>>> should be definetely removed from code.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> ср, 1 авг. 2018 г. в 9:39, Peter Ivanov :
> > > >>>>>>
> > > >>>>>>> The task was ready long ago, but community failed to review and
> > > merge
> > > >>>> it
> > > >>>>>>> ¯\_(ツ)_/¯
> > > >>>>>>> Not being a committer, my capabilities o

Re: Ignite as distributed file storage

2018-08-02 Thread Dmitriy Setrakyan
On Thu, Aug 2, 2018 at 1:08 AM, Pavel Kovalenko  wrote:

> Dmitriy,
>
> I still don't understand why do you think that it will be file system?
> In all my previous messages I emphasized that this storage shouldn't be
> considered as a file system. It's just a large data storage, whose entities
> can be easily accessed using key/link (internally, or externally using
> web/binary protocol interfaces).
>
> > Instead, if we must focus on large blobs, I would solve the problem of
> supporting large blobs in regular Ignite caches, as I suggested before.
>
> This is impossible. Our page memory can't handle efficiently it by design.
>

But our API does. What is stopping us from creating a cache as a BLOB cache
and using whatever storage we need?


Re: Upgrading Ignite to JCache 1.1

2018-08-02 Thread Dmitriy Setrakyan
Thanks to everyone for making it happen!

On Thu, Aug 2, 2018 at 6:52 AM, Dmitriy Pavlov 
wrote:

> Hi Anton, Alexander, Denis, Nikita, Pavel,
>
> It is very good contribution. It resolved license issue with JCache 1.0.
>
> Thank you guys for making this happen.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 2 авг. 2018 г. в 16:37, Anton Vinogradov :
>
> > Igniters,
> >
> > We officially support JCache 1.1 now [1].
> >
> > Huge thanks to everyone who hepled us:
> > - Alexander Menshikov
> > - Denis Garus
> > - Amelchev Nikita
> > - Pavel Pereslegin
> >
> > [1]
> >
> > https://ci.ignite.apache.org/viewLog.html?buildId=1578758;
> tab=queuedBuildOverviewTab
> >
> > чт, 19 июл. 2018 г. в 16:12, Vyacheslav Daradur :
> >
> > > Hi, Alex!
> > >
> > > Could you also help with the ticket [1]? If you have time.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9020
> > >
> > > On Tue, Jul 17, 2018 at 4:50 PM Denis Magda  wrote:
> > > >
> > > > Pavel,
> > > >
> > > > Could you chime in please?
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Tue, Jul 17, 2018 at 6:43 AM Nikita Amelchev <
> nsamelc...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, Igniters.
> > > > >
> > > > > I'm implementing new version TCK 1.1 and I found the problem [1] in
> > the
> > > > > .NET module.
> > > > >
> > > > > In brief, .NET creates cache entry event based on values exist
> > > checking.
> > > > >
> > > > > TCK 1.1 says that getValue() not null for REMOVE/EXPIRED events if
> > old
> > > > > value required and it breaks logic.
> > > > >
> > > > > I'm looking for a .net contributor. Anyone ready to help?
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/IGNITE-9020
> > > > >
> > > > > 2018-06-15 14:35 GMT+03:00 Александр Меньшиков <
> sharple...@gmail.com
> > >:
> > > > >
> > > > > > Denis, I think we can include it to a minor release. Because the
> > > network
> > > > > > protocol, API, binary compatibility will be saved. And all
> behavior
> > > > > > changes, in fact, make implementation closer to the documentation
> > of
> > > > > JCache
> > > > > > 1.0. Because TCK 1.1.0 in general fixes differents between
> > > documentation
> > > > > > and tests in 1.0.
> > > > > >
> > > > > > 2018-06-14 21:41 GMT+03:00 Denis Magda :
> > > > > >
> > > > > > > Guys, are you targeting this for the next big Ignite release?
> > > Should be
> > > > > > in
> > > > > > > 3 m from now.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > > On Thu, Jun 14, 2018 at 2:58 AM Anton Vinogradov <
> a...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Corrected IEP URL:
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > > > > 21%3A+JCache+1.1+support
> > > > > > > >
> > > > > > > > чт, 14 июн. 2018 г. в 12:48, Александр Меньшиков <
> > > > > sharple...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > I've prepared IEP-21 [1] for this JCache updating task.
> > > > > > > > > It will help us to manage the issues and see the progress
> in
> > > one
> > > > > > place.
> > > > > > > > > Also, we have finally added tests for TCK 1.1.0 [2] to our
> TC
> > > which
> > > > > > can
> > > > > > > > be
> > > > > > > > > run on any branch.
> > > > > > > > > Both tests cases (for 1.0.1 and for 1.1.0) will coexist
> until
> > > > > IEP-21
> > > > > > > > > finish.
> > > > > > > > >

Re: Ignite as distributed file storage

2018-08-01 Thread Dmitriy Setrakyan
ke crutch, to support replication
> > and
> > > > possibility to have some data in-memory I need to write again a lot
> of
> > > > stuff.
> > > > Let's finally leave IGFS alone and wait for IEP.
> > > >
> > > >
> > > > 2018-07-06 0:01 GMT+03:00 Vladimir Ozerov :
> > > >
> > > >> Pavel,
> > > >>
> > > >> IGFS doesn't enforce you to have block in heap. What you suggest can
> > be
> > > >> achieved with IGFS as follows:
> > > >> 1) Disable caching, so data cache is not used ("PROXY" mode)
> > > >> 2) Implement IgniteFileSystem interface which operates on abstract
> > > streams
> > > >>
> > > >> But the whole idea around mmap sounds like premature optimisation to
> > > me. I
> > > >> conducted a number of experiments with IGFS on large Hadoop
> workload.
> > > Even
> > > >> with old AI 1.x architecture, where everything was stored onheap, I
> > > never
> > > >> had an issue with GC. The key point is that IGFS operates on large
> > > (64Kb)
> > > >> data blocks, so even with 100Gb full of these blocks you will have
> > > >> relatively small number of objects and normal GC pauses. Additional
> > > memory
> > > >> copying is not an issue either in most workloads in distributed
> > systems,
> > > >> because most of the time is spent on IO and internal synchronization
> > > >> anyway.
> > > >>
> > > >> Do you have specific scenario when you observed long GC pauses with
> GC
> > > or
> > > >> serious performance degradation with IGFS?
> > > >>
> > > >> Even if we agree that mmap usage is a critical piece, all we need is
> > to
> > > >> implement a single IGFS interface.
> > > >>
> > > >> On Thu, Jul 5, 2018 at 10:44 PM Pavel Kovalenko  >
> > > >> wrote:
> > > >>
> > > >> > Vladimir,
> > > >> >
> > > >> > The key difference between BLOB storage and IGFS is that BLOB
> > storage
> > > >> will
> > > >> > have persistent-based architecture with possibility to cache
> blocks
> > in
> > > >> > offheap (using mmap, which is more simple, because we delegate it
> to
> > > OS
> > > >> > level)
> > > >> > , while IGFS has in-memory based architecture with possibility to
> > > >> persist
> > > >> > blocks.
> > > >> > BLOB storage will have possibility to work with small amount of
> RAM
> > > >> without
> > > >> > signficant performance drop (Using zero-copy from socket to disk)
> > and
> > > in
> > > >> > opposite case it can keep all available blocks in offheap if it's
> > > >> possible
> > > >> > (Using mmap again).
> > > >> > IGFS perform a lot of operations with blocks in on-heap which
> leads
> > to
> > > >> > unnecessary data copies, long GC pauses and performance drop. All
> > IGFS
> > > >> > architecture tightly bound with in-memory features, so it's too
> hard
> > > to
> > > >> > rewrite IGFS in persistent-based manner. But, cool IGFS features
> > such
> > > as
> > > >> > intelligent affinity routing, chunk colocation will be reused in
> > BLOB
> > > >> > storage.
> > > >> > Does it make sense?
> > > >> >
> > > >> >
> > > >> >
> > > >> > 2018-07-05 19:01 GMT+03:00 Vladimir Ozerov  >:
> > > >> >
> > > >> > > Pavel,
> > > >> > > Design you described is almost precisely what IGFS does. It has
> a
> > > >> cache
> > > >> > for
> > > >> > > metadata, split binary data in chunks with intelligent affinity
> > > >> routing.
> > > >> > In
> > > >> > > addition we have map-reduce feature on top of it and integration
> > > with
> > > >> > > underlying file system with optional caching. Data can be
> accessed
> > > in
> > > >> > > blocks or streams. IGFS is not in active development, but it is
> > not
> > > >> > > outdated either.
> > >

Re: Removing "fabric" from Ignite binary package name

2018-08-01 Thread Dmitriy Setrakyan
I vote to remove the fabric from the build in the easiest way possible. Can
other Igniters comment?

D.

On Wed, Aug 1, 2018 at 12:46 PM, Petr Ivanov  wrote:

> My concern here is exactly about internal build processes — removing
> fabric from the name of binary archive (with any way) will break lots of
> them.
> There will be no sacrifices, just lots of work for fixing build processes
> (where we won’t be able to introduce changes proactively).
>
> Therefore only fabric removal implementation (quick with some legacy left
> or full refactoring) is on the agenda.
> And this matter should be jugged by the community: currently we have (if
> our voices are equal) 1:1 with Anton about it.
>
>
>
>
> > On 1 Aug 2018, at 22:28, Dmitriy Setrakyan 
> wrote:
> >
> > Let's focus on what is important here. Our users do not care about our
> > internal build process.If we could remove the word fabric from the next
> > release without any significant sacrifices in the build process or making
> > it less maintainable, I suggest we do it.
> >
> > D.
> >
> > On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov 
> wrote:
> >
> >> Simple way with some hack and legacy maintenance: accept patch as it is
> >> implemented now.
> >> Hard way: full assembly refactoring and hadoop rejection.
> >>
> >> Anyway, after this is merged to master — complete automation systems
> >> revision (TeamCity for example) is required due to heavy hardcode of
> >> “fabric” in such systems.
> >>
> >>
> >>> On 1 Aug 2018, at 21:55, Dmitriy Setrakyan 
> >> wrote:
> >>>
> >>> OK, so what is the plan? How do we get rid of the fabric name?
> >>>
> >>> D.
> >>>
> >>> On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov 
> wrote:
> >>>
> >>>> Since you proposing patch to the community, you are the very man :)
> >>>>
> >>>> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov :
> >>>>
> >>>>> You are convincing the wrong person.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 1 Aug 2018, at 12:05, Anton Vinogradov  wrote:
> >>>>>>
> >>>>>> Peter,
> >>>>>>
> >>>>>> We had a discussion about how to do this properly.
> >>>>>> Proposed solution cannot be merged, since it makes code harder than
> it
> >>>>> was.
> >>>>>>
> >>>>>> The only case is to perform complete refactoring and get rid of all
> >>>>>> postfixes and other weird stuff.
> >>>>>>
> >>>>>> For example
> >>>>>> - 
> >>>>>> - 
> >>>>>> should be definetely removed from code.
> >>>>>>
> >>>>>>
> >>>>>> ср, 1 авг. 2018 г. в 9:39, Peter Ivanov :
> >>>>>>
> >>>>>>> The task was ready long ago, but community failed to review and
> merge
> >>>> it
> >>>>>>> ¯\_(ツ)_/¯
> >>>>>>> Not being a committer, my capabilities of introducing such changes
> >> are
> >>>>>>> limited.
> >>>>>>>
> >>>>>>> I will update code during this week and will pass for review once
> >>>> again.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan <
> >> dsetrak...@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Yes, agree, fabric has to be removed. If it is done in 2.7, would
> be
> >>>>>>> great!
> >>>>>>>>
> >>>>>>>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda 
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Peter, folks,
> >>>>>>>>>
> >>>>>>>>> It's weird, but we have been failing to introduce this minor
> change
> >>>>>>> since
> >>>>>>>>> December. Can we get it done for 2.7 that is being discussed at
> the
> >>>>>>>> moment?
> >>>>>>>>> Are there any technical issues that block you from merging the
> >>>>> changes?

Re: Removing "fabric" from Ignite binary package name

2018-08-01 Thread Dmitriy Setrakyan
Let's focus on what is important here. Our users do not care about our
internal build process.If we could remove the word fabric from the next
release without any significant sacrifices in the build process or making
it less maintainable, I suggest we do it.

D.

On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov  wrote:

> Simple way with some hack and legacy maintenance: accept patch as it is
> implemented now.
> Hard way: full assembly refactoring and hadoop rejection.
>
> Anyway, after this is merged to master — complete automation systems
> revision (TeamCity for example) is required due to heavy hardcode of
> “fabric” in such systems.
>
>
> > On 1 Aug 2018, at 21:55, Dmitriy Setrakyan 
> wrote:
> >
> > OK, so what is the plan? How do we get rid of the fabric name?
> >
> > D.
> >
> > On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov  wrote:
> >
> >> Since you proposing patch to the community, you are the very man :)
> >>
> >> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov :
> >>
> >>> You are convincing the wrong person.
> >>>
> >>>
> >>>
> >>>> On 1 Aug 2018, at 12:05, Anton Vinogradov  wrote:
> >>>>
> >>>> Peter,
> >>>>
> >>>> We had a discussion about how to do this properly.
> >>>> Proposed solution cannot be merged, since it makes code harder than it
> >>> was.
> >>>>
> >>>> The only case is to perform complete refactoring and get rid of all
> >>>> postfixes and other weird stuff.
> >>>>
> >>>> For example
> >>>> - 
> >>>> - 
> >>>> should be definetely removed from code.
> >>>>
> >>>>
> >>>> ср, 1 авг. 2018 г. в 9:39, Peter Ivanov :
> >>>>
> >>>>> The task was ready long ago, but community failed to review and merge
> >> it
> >>>>> ¯\_(ツ)_/¯
> >>>>> Not being a committer, my capabilities of introducing such changes
> are
> >>>>> limited.
> >>>>>
> >>>>> I will update code during this week and will pass for review once
> >> again.
> >>>>>
> >>>>>
> >>>>> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan <
> dsetrak...@apache.org
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be
> >>>>> great!
> >>>>>>
> >>>>>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda 
> >>> wrote:
> >>>>>>
> >>>>>>> Peter, folks,
> >>>>>>>
> >>>>>>> It's weird, but we have been failing to introduce this minor change
> >>>>> since
> >>>>>>> December. Can we get it done for 2.7 that is being discussed at the
> >>>>>> moment?
> >>>>>>> Are there any technical issues that block you from merging the
> >>> changes?
> >>>>>>>
> >>>>>>> --
> >>>>>>> Denis
> >>>>>>>
> >>>>>>> On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov 
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Ok, then I will update issue code and start preparation for build
> >>>>>>>> configuration changes.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, 7 Jun 2018 at 23:41, Denis Magda 
> >> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> With which one — current implementation in issue?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's the answer to your question:
> >>>>>>>>>
> >>>>>>>>> 1. quickly fix all of them (can be solved by preliminary
> >>>>>> preparations —
> >>>>>>>>> searching for -fabric- usages in build configuration);
> >>>>>>>>>   2. update all branches to master because otherwise old branch
> >>>>>> will
> >>>>>>>> stop
> >>>>>>>>> building.
> >>>>>>>>>
> >&g

Re: Removing "fabric" from Ignite binary package name

2018-08-01 Thread Dmitriy Setrakyan
OK, so what is the plan? How do we get rid of the fabric name?

D.

On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov  wrote:

> Since you proposing patch to the community, you are the very man :)
>
> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov :
>
> > You are convincing the wrong person.
> >
> >
> >
> > > On 1 Aug 2018, at 12:05, Anton Vinogradov  wrote:
> > >
> > > Peter,
> > >
> > > We had a discussion about how to do this properly.
> > > Proposed solution cannot be merged, since it makes code harder than it
> > was.
> > >
> > > The only case is to perform complete refactoring and get rid of all
> > > postfixes and other weird stuff.
> > >
> > > For example
> > > - 
> > > - 
> > > should be definetely removed from code.
> > >
> > >
> > > ср, 1 авг. 2018 г. в 9:39, Peter Ivanov :
> > >
> > >> The task was ready long ago, but community failed to review and merge
> it
> > >> ¯\_(ツ)_/¯
> > >> Not being a committer, my capabilities of introducing such changes are
> > >> limited.
> > >>
> > >> I will update code during this week and will pass for review once
> again.
> > >>
> > >>
> > >> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan  >
> > >> wrote:
> > >>
> > >>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be
> > >> great!
> > >>>
> > >>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda 
> > wrote:
> > >>>
> > >>>> Peter, folks,
> > >>>>
> > >>>> It's weird, but we have been failing to introduce this minor change
> > >> since
> > >>>> December. Can we get it done for 2.7 that is being discussed at the
> > >>> moment?
> > >>>> Are there any technical issues that block you from merging the
> > changes?
> > >>>>
> > >>>> --
> > >>>> Denis
> > >>>>
> > >>>> On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov 
> > >>> wrote:
> > >>>>
> > >>>>> Ok, then I will update issue code and start preparation for build
> > >>>>> configuration changes.
> > >>>>>
> > >>>>>
> > >>>>> On Thu, 7 Jun 2018 at 23:41, Denis Magda 
> wrote:
> > >>>>>
> > >>>>>>>
> > >>>>>>> With which one — current implementation in issue?
> > >>>>>>
> > >>>>>>
> > >>>>>> That's the answer to your question:
> > >>>>>>
> > >>>>>> 1. quickly fix all of them (can be solved by preliminary
> > >>> preparations —
> > >>>>>> searching for -fabric- usages in build configuration);
> > >>>>>>2. update all branches to master because otherwise old branch
> > >>> will
> > >>>>> stop
> > >>>>>> building.
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Denis
> > >>>>>>
> > >>>>>> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov 
> > >>>> wrote:
> > >>>>>>
> > >>>>>>>
> > >>>>>>>> On 7 Jun 2018, at 23:04, Denis Magda 
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>> I'm fine with the suggested approach.
> > >>>>>>>
> > >>>>>>> With which one — current implementation in issue?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> However, not sure we need to update
> > >>>>>>>> all the branches. Can't branch owners just pull the changes
> > >> back
> > >>>> from
> > >>>>>>>> master if the plan to merge back later?
> > >>>>>>>
> > >>>>>>> Of course, we as an initiative group of this issue should do
> > >>> nothing,
> > >>>>> it
> > >>>>>>> will lie on shoulders of developers.
> > >>>>>>>
> > >>>>>>>

Re: Removing "fabric" from Ignite binary package name

2018-07-31 Thread Dmitriy Setrakyan
 > > > >>>>>>>>> We have special assembly desctiptors, for example:
> > > > >>>>>>>>> dependencies-fabric.xml
> > > > >>>>>>>>> dependencies-fabric-lgpl.xml
> > > > >>>>>>>>> dependencies-hadoop.xml
> > > > >>>>>>>>> release-base.xml
> > > > >>>>>>>>> release-fabric.xml
> > > > >>>>>>>>> release-fabric-base.xml
> > > > >>>>>>>>> release-fabric-lgpl.xml
> > > > >>>>>>>>> release-hadoop.xml
> > > > >>>>>>>>>
> > > > >>>>>>>>> So, I'ts impossible for now to remove "fabric" without
> > "hadoop"
> > > > >>>>>>> removal.
> > > > >>>>>>>>> Only one case is to make some ditry hack, but that's not a
> > good
> > > > >> idea.
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Feb 8, 2018 at 11:29 AM, Sergey Kozlov <
> > > > >> skoz...@gridgain.com
> > > > >>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> +1 hadoop accelerator removing for AI 2.5
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Also probably IGFS should be either removed or refactored,
> > > e.g.
> > > > >>>> create
> > > > >>>>>>>> FS
> > > > >>>>>>>>>> directly over the data region without using "cache" entity
> > as
> > > an
> > > > >>>>>>>>>> intermidiate stage
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Feb 8, 2018 at 2:13 AM, Denis Magda <
> > > dma...@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Anton,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I don’t get how the hadoop editions are related to this
> > task.
> > > > The
> > > > >>>>>>>> project
> > > > >>>>>>>>>>> is not named as “data fabric” for a while. Check up the
> > site
> > > or
> > > > >>>> docs.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> The “fabric” word is being removed from all over the
> places
> > > and
> > > > >>>> needs
> > > > >>>>>>>> to
> > > > >>>>>>>>>>> be removed from the editions’ names.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> As for the hadoop future, my personal position is to
> retire
> > > > this
> > > > >>>>>>>>>> component
> > > > >>>>>>>>>>> and forget about it. I would restart the conversation
> again
> > > > after
> > > > >>>> we
> > > > >>>>>>>> done
> > > > >>>>>>>>>>> with 2.4.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> —
> > > > >>>>>>>>>>> Denis
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> On Feb 7, 2018, at 2:13 AM, Anton Vinogradov <
> > a...@apache.org
> > > >
> > > > >>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Denis, Petr,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I checked PR and found we have *overcomplicated* logic
> > with
> > > > >>>> "fabric"
&g

Re: Apache Ignite tickets with empty description & dev list references

2018-07-31 Thread Dmitriy Setrakyan
Completely agree. When filing a ticket, we must all understand that we are
asking the community to work on it. Without a clear description nobody will
be able to pick up the ticket.

D.

On Tue, Jul 31, 2018 at 7:02 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> I would like to ask everyone in the Community to fill ticket decsription.
> Without it ticket is almost always incomplete and it is unclear for all
> community members, why this change was done. Moreover it is unlear why
> Ignite comitters accepts such changes.
>
> Also please link ticket to dev.list dicsusions (if any). You can find your
> topic in http://apache-ignite-developers.2346864.n4.nabble.com/ and insert
> link using More->Link->Web Link and insert particular post URL there.
>
> Please make sure you've covered
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>
> I would be happy to answer any questions related to contribution, and any
> feedback, as always, is welcomed.
>
> Sincerely,
> Dmitriy Pavlov
>


Re: Assign JIRA tickets to someone else

2018-07-31 Thread Dmitriy Setrakyan
Dmitriy,

I would agree with everything you are saying. There are some tickets,
however, that cannot be done by just anyone and preferably have to be
looked at by certain domain experts. In those cases only it would be OK to
assign a ticket explicitly in my view. For all other cases we should allow
the community pick up a ticket they want to work on.

D.

On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> Ignite is complex project and sometimes we know
> 1. who can solve the issue
> 2. and we would like that particular contributor would take an issue.
>
> In the same time "in an open source community where everything is
> distributed, asynchronous and volunteer work we should leave the issues
> open to anyone until someone volunteers to work on it."
>
> It seems it is not prohibited by Apache to assign it directly, in the same
> time it is better for community grown to avoid it:
> "Assigning things to people seems the role of a project manager that has
> some sort of power over the managed team. Speaking from experience I do not
> take it nicely when someone that uses the project I am a volunteer at
> (which I might now know ) "assigns work" to me."
>
> Please find Apache mentors' comments on this:
> https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd
> 53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E
>
>
> Please share you vision.
>
> Sincerely,
> Dmitriy Pavlov
>


  1   2   3   4   5   6   7   8   9   10   >