Re: Rename "cache group" to "tablespace"?

[Dmitriy Setrakyan]

I really like the idea.

On Mon, Nov 19, 2018, 23:45 Vladimir Ozerov  Igniters,
>
> We had several discussion about cache groups [1]. Our current consensus is
> that this is not that good design decision - bad scan performance, no API.
> A kind of shortcut to solve memory consumption problems. For this reason we
> try to avoid exposing "cache group" to API when possible. Also we
> understand that current file-per-partition approach is not ideal either -
> when there are too many partitions we have to fsync a lot of files. And as
> an idea for "bright future" we consider tablespaces - one or several files
> where all database objects are stored in dedicated "segments", which are
> allocated in smaller units called "extents".
>
> I was thinking on how to "legalize" cache groups on API on the one hand
> (product needs them currently), and to let real tablespaces to appear in
> future. Idea is simple - treat cache group as special kind of tablespace.
> We may say that current storage organization is one type of tablespace, and
> proposed "file-segment-extent" storage organization is another type of
> tablespace.
>
> With this in mind we can configure tablespaces in IgniteConfiguration, then
> assign each cache a tablespace. As advanced tasks we may allow dynamic
> table space create/alter/drop, I'll show SQL examples below.
>
> // Interface
> interface Tablespace {
> String name();
> }
>
> // Current non-shared tablespace (aka "physical cache")
> class FilePerPartitionTablespace implements Tablespace {
> // ...
> }
>
> // Shared tablespace (aka "cache group") - note that now we have a legal
> place for cache group configuration:
> class FilePerPartitionSharedTablespace implements Tablespace {
> CacheMode cacheMode;
> CacheAtomicityMode cacheAtomicityMode;
> AffinityFunction affinityFunction;
> // + Other parameters from
> ClusterCachesInfo.validateCacheGroupConfiguration
> // + Some parameters from "DataRegionConfiguration" (e.g. paths)
> }
>
> // Future "real" tablespace
> class SegmentedTablespace implements Tablespace {
> // Whatever is needed.
> }
>
> // Cache configuration
> class CacheConfiguration {
> ...
> String tablespace;
>...
> }
>
> // Managing tablespaces from SQL:
> CREATE TABLESPACE my_tbs ENGINE=FILE_PER_PARTITION_SHARED
> CACHE_MODE=PARTITIONED BACKUPS=1
> ALTER TABLESPACE my_tbs ENCRYPTED=1
> CREATE TABLE my_table (...) TABLESPACE my_tbs
>
> What do you think?
>
> Vladimir.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Remove-cache-groups-in-AI-3-0-td29184.html
>

Re: [IMPORTANT] Future of Binary Objects

[Vladimir Ozerov]

Hi Alexey,

Binary Objects only.

On Tue, Nov 20, 2018 at 10:50 AM Alexey Zinoviev 
wrote:

> Do we discuss here Core features only or the roadmap for all components?
>
> вт, 20 нояб. 2018 г. в 10:05, Vladimir Ozerov :
>
> > Igniters,
> >
> > It is very likely that Apache Ignite 3.0 will be released next year. So
> we
> > need to start thinking about major product improvements. I'd like to
> start
> > with binary objects.
> >
> > Currently they are one of the main limiting factors for the product. They
> > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > comparing to other vendors. They are slow - not suitable for SQL at all.
> >
> > I would like to ask all of you who worked with binary objects to share
> your
> > feedback and ideas, so that we understand how they should look like in AI
> > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > critics. Then we will work on ideas in separate topics.
> >
> > 1) Historical background
> >
> > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> working
> > on .NET and CPP clients. During design we had several ideas in mind:
> > - ability to read object fields in O(1) without deserialization
> > - interoperabillty between Java, .NET and CPP.
> >
> > Since then a number of other concepts were mixed to the cocktail:
> > - Affinity key fields
> > - Strict typing for existing fields (aka metadata)
> > - Binary Object as storage format
> >
> > 2) My proposals
> >
> > 2.1) Introduce "Data Row Format" interface
> > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > Efficient storage typically has <10 bytes overhead per row (no metadata,
> no
> > length, no hash code, etc), allow supper-fast field access, support
> > different string formats (ASCII, UTF-8, etc), support different temporal
> > types (date, time, timestamp, timestamp with timezone, etc), and store
> > these types as efficiently as possible.
> >
> > What we need is to introduce an interface which will convert a pair of
> > key-value objects into a row. This row will be used to store data and to
> > get fields from it. Care about memory consumption, need SQL and strict
> > schema - use one format. Need flexibility and prefer key-value access -
> use
> > another format which will store binary objects unchanged (current
> > behavior).
> >
> > interface DataRowFormat {
> > DataRow create(Object key, Object value); // primitives or binary
> > objects
> > DataRowMetadata metadata();
> > }
> >
> > 2.2) Remove affinity field from metadata
> > Affinity rules are governed by cache, not type. We should remove
> > "affintiyFieldName" from metadata.
> >
> > 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
> >
> > 2.4) Use bitmaps for "null" and default values and for fixed-length
> fields,
> > put fixed-length fields before variable-length.
> > Motivation: to save space.
> >
> > What else? Please share your ideas.
> >
> > Vladimir.
> >
>

Re: [IMPORTANT] Future of Binary Objects

[Alexey Zinoviev]

Do we discuss here Core features only or the roadmap for all components?

вт, 20 нояб. 2018 г. в 10:05, Vladimir Ozerov :

> Igniters,
>
> It is very likely that Apache Ignite 3.0 will be released next year. So we
> need to start thinking about major product improvements. I'd like to start
> with binary objects.
>
> Currently they are one of the main limiting factors for the product. They
> are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> comparing to other vendors. They are slow - not suitable for SQL at all.
>
> I would like to ask all of you who worked with binary objects to share your
> feedback and ideas, so that we understand how they should look like in AI
> 3.0. This is a brain storm - let's accumulate ideas first and minimize
> critics. Then we will work on ideas in separate topics.
>
> 1) Historical background
>
> BO were implemented around 2014 (Apache Ignite 1.5) when we started working
> on .NET and CPP clients. During design we had several ideas in mind:
> - ability to read object fields in O(1) without deserialization
> - interoperabillty between Java, .NET and CPP.
>
> Since then a number of other concepts were mixed to the cocktail:
> - Affinity key fields
> - Strict typing for existing fields (aka metadata)
> - Binary Object as storage format
>
> 2) My proposals
>
> 2.1) Introduce "Data Row Format" interface
> Binary Objects are terrible candidates for storage. Too fat, too slow.
> Efficient storage typically has <10 bytes overhead per row (no metadata, no
> length, no hash code, etc), allow supper-fast field access, support
> different string formats (ASCII, UTF-8, etc), support different temporal
> types (date, time, timestamp, timestamp with timezone, etc), and store
> these types as efficiently as possible.
>
> What we need is to introduce an interface which will convert a pair of
> key-value objects into a row. This row will be used to store data and to
> get fields from it. Care about memory consumption, need SQL and strict
> schema - use one format. Need flexibility and prefer key-value access - use
> another format which will store binary objects unchanged (current
> behavior).
>
> interface DataRowFormat {
> DataRow create(Object key, Object value); // primitives or binary
> objects
> DataRowMetadata metadata();
> }
>
> 2.2) Remove affinity field from metadata
> Affinity rules are governed by cache, not type. We should remove
> "affintiyFieldName" from metadata.
>
> 2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.
>
> 2.4) Use bitmaps for "null" and default values and for fixed-length fields,
> put fixed-length fields before variable-length.
> Motivation: to save space.
>
> What else? Please share your ideas.
>
> Vladimir.
>

Rename "cache group" to "tablespace"?

[Vladimir Ozerov]

Igniters,

We had several discussion about cache groups [1]. Our current consensus is
that this is not that good design decision - bad scan performance, no API.
A kind of shortcut to solve memory consumption problems. For this reason we
try to avoid exposing "cache group" to API when possible. Also we
understand that current file-per-partition approach is not ideal either -
when there are too many partitions we have to fsync a lot of files. And as
an idea for "bright future" we consider tablespaces - one or several files
where all database objects are stored in dedicated "segments", which are
allocated in smaller units called "extents".

I was thinking on how to "legalize" cache groups on API on the one hand
(product needs them currently), and to let real tablespaces to appear in
future. Idea is simple - treat cache group as special kind of tablespace.
We may say that current storage organization is one type of tablespace, and
proposed "file-segment-extent" storage organization is another type of
tablespace.

With this in mind we can configure tablespaces in IgniteConfiguration, then
assign each cache a tablespace. As advanced tasks we may allow dynamic
table space create/alter/drop, I'll show SQL examples below.

// Interface
interface Tablespace {
String name();
}

// Current non-shared tablespace (aka "physical cache")
class FilePerPartitionTablespace implements Tablespace {
// ...
}

// Shared tablespace (aka "cache group") - note that now we have a legal
place for cache group configuration:
class FilePerPartitionSharedTablespace implements Tablespace {
CacheMode cacheMode;
CacheAtomicityMode cacheAtomicityMode;
AffinityFunction affinityFunction;
// + Other parameters from
ClusterCachesInfo.validateCacheGroupConfiguration
// + Some parameters from "DataRegionConfiguration" (e.g. paths)
}

// Future "real" tablespace
class SegmentedTablespace implements Tablespace {
// Whatever is needed.
}

// Cache configuration
class CacheConfiguration {
...
String tablespace;
   ...
}

// Managing tablespaces from SQL:
CREATE TABLESPACE my_tbs ENGINE=FILE_PER_PARTITION_SHARED
CACHE_MODE=PARTITIONED BACKUPS=1
ALTER TABLESPACE my_tbs ENCRYPTED=1
CREATE TABLE my_table (...) TABLESPACE my_tbs

What do you think?

Vladimir.

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Remove-cache-groups-in-AI-3-0-td29184.html

Re: [MTCGA]: new failures in builds [2356663] needs to be handled

[Alexey Goncharuk]

Hi, this is an intentionally failed test with a ticket, I've now muted it
on TC.

вт, 20 нояб. 2018 г. в 07:38, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New test failure in master
> CacheExchangeMergeTest.testMergeServerAndClientJoin1
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=4021629064265895900=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - dmitriy.govorukhin
> https://ci.ignite.apache.org/viewModification.html?modId=839470
>  - bessonov.ip
> https://ci.ignite.apache.org/viewModification.html?modId=839471
>  - bessonov.ip
> https://ci.ignite.apache.org/viewModification.html?modId=839416
>  - ilantukh
> https://ci.ignite.apache.org/viewModification.html?modId=839419
>  - bessonov.ip
> https://ci.ignite.apache.org/viewModification.html?modId=839413
>  - verbalab
> https://ci.ignite.apache.org/viewModification.html?modId=839414
>  - bessonov.ip
> https://ci.ignite.apache.org/viewModification.html?modId=839415
>  - pudov.max
> https://ci.ignite.apache.org/viewModification.html?modId=839472
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 07:38:03 20-11-2018
>

[IMPORTANT] Future of Binary Objects

[Vladimir Ozerov]

Igniters,

It is very likely that Apache Ignite 3.0 will be released next year. So we
need to start thinking about major product improvements. I'd like to start
with binary objects.

Currently they are one of the main limiting factors for the product. They
are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
comparing to other vendors. They are slow - not suitable for SQL at all.

I would like to ask all of you who worked with binary objects to share your
feedback and ideas, so that we understand how they should look like in AI
3.0. This is a brain storm - let's accumulate ideas first and minimize
critics. Then we will work on ideas in separate topics.

1) Historical background

BO were implemented around 2014 (Apache Ignite 1.5) when we started working
on .NET and CPP clients. During design we had several ideas in mind:
- ability to read object fields in O(1) without deserialization
- interoperabillty between Java, .NET and CPP.

Since then a number of other concepts were mixed to the cocktail:
- Affinity key fields
- Strict typing for existing fields (aka metadata)
- Binary Object as storage format

2) My proposals

2.1) Introduce "Data Row Format" interface
Binary Objects are terrible candidates for storage. Too fat, too slow.
Efficient storage typically has <10 bytes overhead per row (no metadata, no
length, no hash code, etc), allow supper-fast field access, support
different string formats (ASCII, UTF-8, etc), support different temporal
types (date, time, timestamp, timestamp with timezone, etc), and store
these types as efficiently as possible.

What we need is to introduce an interface which will convert a pair of
key-value objects into a row. This row will be used to store data and to
get fields from it. Care about memory consumption, need SQL and strict
schema - use one format. Need flexibility and prefer key-value access - use
another format which will store binary objects unchanged (current behavior).

interface DataRowFormat {
DataRow create(Object key, Object value); // primitives or binary
objects
DataRowMetadata metadata();
}

2.2) Remove affinity field from metadata
Affinity rules are governed by cache, not type. We should remove
"affintiyFieldName" from metadata.

2.3) Remove restrictions on changing field type
I do not know why we did that in the first place. This restriction prevents
type evolution and confuses users.

2.4) Use bitmaps for "null" and default values and for fixed-length fields,
put fixed-length fields before variable-length.
Motivation: to save space.

What else? Please share your ideas.

Vladimir.

Re: Apache Ignite 2.7. Last Mile

[Nikolay Izhikov]

Hello, Dmitrii.

I see 2 tickets for this improvement:

IGNITE-602 - [Test] GridToStringBuilder is vulnerable for StackOverflowError 
caused by infinite recursion [1]
IGNITE-9209 - GridDistributedTxMapping.toString() returns broken string [2]

Should we revert both commits?

[1] https://github.com/apache/ignite/commit/d67c5bf
[2] https://github.com/apache/ignite/commit/9bb9c04


В Пн, 19/11/2018 в 13:36 +0300, Dmitrii Ryabov пишет:
> I agree to revert and make fix for 2.8. So, we will have more time to test
> it.
> 
> пн, 19 нояб. 2018 г., 10:53 Vladimir Ozerov voze...@gridgain.com:
> 
> > +1 for revert.
> > 
> > On Sun, Nov 18, 2018 at 11:31 PM Dmitriy Pavlov 
> > wrote:
> > 
> > > I personally don't mind.
> > > 
> > > But I would like Dmitry Ryabov and Alexey Goncharuck share their
> > 
> > opinions.
> > > 
> > > вс, 18 нояб. 2018 г., 20:43 Nikolay Izhikov :
> > > 
> > > > Yes, I think so.
> > > > 
> > > > вс, 18 нояб. 2018 г., 20:34 Denis Magda dma...@apache.org:
> > > > 
> > > > > Sounds good to me. Are we starting the vote then?
> > > > > 
> > > > > Denis
> > > > > 
> > > > > 
> > > > > 
> > > > > On Sun, Nov 18, 2018 at 8:25 AM Nikolay Izhikov  > > > > wrote:
> > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > This issue is the only ticket that blocks 2.7 release.
> > > > > > 
> > > > > > I looked at IGNITE-602 PR and GridToStringBuilder.
> > > > > > The code looks complicated for me.
> > > > > > And it's not obvious for me how to fix this issue in a short period
> > > 
> > > of
> > > > > > time.
> > > > > > Especially, code deals with recursion and other things that can
> > 
> > lead
> > > to
> > > > > > very dangerous errors.
> > > > > > 
> > > > > > Let's revert this patch and fix it in calmly.
> > > > > > Also, we need additional tests for it.
> > > > > > 
> > > > > > В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет:
> > > > > > > Ok, I'll check the issue.
> > > > > > > пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk <
> > > > > > 
> > > > > > alexey.goncha...@gmail.com>:
> > > > > > > > 
> > > > > > > > Igniters,
> > > > > > > > 
> > > > > > > > I've just found that S.toString() implementation is broken in
> > > > > > 
> > > > > > ignite-2.7 and master [1]. It leads to a message
> > > > > > > > Wrapper [p=Parent [a=0]Child [b=0, super=]]
> > > > > > > > being formed instead of
> > > > > > > > Wrapper [p=Child [b=0, super=Parent [a=0]]]
> > > > > > > > for classes with inheritance that use
> > 
> > S.toString(SomeClass.class,
> > > > > > this, super.toString()) embedded to some other object.
> > > > > > > > 
> > > > > > > > Dmitrii Ryabov, I've reverted two commits related to IGNITE-602
> > > 
> > > and
> > > > > > IGNITE-9209 tickets locally and it fixes the issue. Can you take a
> > > 
> > > look
> > > > > at
> > > > > > the issue?
> > > > > > > > 
> > > > > > > > I think this regression essentially makes our logs unreadable
> > 
> > in
> > > > some
> > > > > > cases and I would like to get it fixed in ignite-2.7 or revert both
> > > > > 
> > > > > commits
> > > > > > from the release.
> > > > > > > > 
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10301
> > > > > > > > 
> > > > > > > > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org
> > > > > :
> > > > > > > > > 
> > > > > > > > > Hello, Igniters.
> > > > > > > > > 
> > > > > > > > > We still have 5 tickets for 2.7:
> > > > > > > > > 
> > > > > > > > > IGNITE-10052Andrew Mashenkov Restart node during TX
> > 
> > causes
> > > > > > vacuum error.
> > > > > > > > > IGNITE-10170Unassigned   .NET:
> > > 
> > > Services.ServicesTestAsync
> > > > > > fails
> > > > > > > > > IGNITE-10196Maxim Pudov  Remove kafka-clients-*-test
> > > > > > 
> > > > > > dependency
> > > > > > > > > IGNITE-10154Andrey Gura  Critical worker liveness
> > 
> > check
> > > > > > configuration is non-trivial and inconsistent
> > > > > > > > > IGNITE-9996 Nikolay Izhikov  Investigate possible
> > > 
> > > performance
> > > > > > drop in FSYNC mode for ignite-2.7 compared to ignite-2.6
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > В Чт, 08/11/2018 в 14:25 +0300, Nikolay Izhikov пишет:
> > > > > > > > > > I'm OK with this.
> > > > > > > > > > 
> > > > > > > > > > чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:
> > > > > > > > > > > Long, long way to release :)
> > > > > > > > > > > 
> > > > > > > > > > > Guys, we have a breaking change in Ignite 2.7 so we must
> > > 
> > > add
> > > > > > > > > > > IGNITE-10154 [1] fix to the release.
> > > > > > > > > > > 
> > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10154
> > > > > > > > > > > On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego <
> > > > 
> > > > isap...@apache.org
> > > > > > 
> > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > Guys,
> > > > > > > > > > > > 
> > > > > > > > > > > > I've found the following issue: [1]. It is quite local
> > > > 
> > > > (only
> > > > > > affects

[jira] [Created] (IGNITE-10338) Add Disk page compression test suites to TC

[Sergi Vladykin (JIRA)]

Sergi Vladykin created IGNITE-10338:
---

 Summary: Add Disk page compression test suites to TC
 Key: IGNITE-10338
 URL: https://issues.apache.org/jira/browse/IGNITE-10338
 Project: Ignite
  Issue Type: Test
Reporter: Sergi Vladykin
Assignee: Peter Ivanov


There are 2 suites in the module ignite-compress:

org.apache.ignite.testsuites.IgnitePdsCompressionTestSuite

org.apache.ignite.testsuites.IgnitePdsCompressionTestSuite2

 

They should be executed the same way as Direct-IO suites only on Linux agents.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Disk page compression for Ignite persistent store

[Sergi Vladykin]

Denis,

See inline.


пн, 19 нояб. 2018 г. в 20:17, Denis Magda :

> Hi Sergi,
>
> Didn't know you were cooking this dish in the background ) Excellent.  Just
> to be sure, that's part of this IEP, right?
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-Withoutin-memorycompression


Correct.


>
>
> Since we can release only full file system blocks which are typically 4k
> > size, user must configure page size to be at least multiple FS blocks,
> e.g.
> > 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> > pageSize = 4k / 16k = 0.25
>
>
> How to we handle the case if the page size is not a multiple of 4K? What is
> the most optimal page size if the user wants to get the best compression?
> Probably, we can adjust the default page size automatically if it's a clean
> deployment.
>
>
We already force page size to be between 1k and 16k and to be power of 2.
Thus there are only 2 options greater than 4k: either 8k or 16k. So page
must be just large enough.

Obviously the greater page size, the better compression you have, but
having very large pages may affect performance badly. Thus 8k with ratio
0.5 or 16k with ratio 0.25 must be OK for the most of cases.



> There will be 2 new properties on CacheConfiguration
> > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk
> page
> > compression.
>
>
> How about setting it at DataRegionConfiguration level as well so that it's
> applied for all the caches/tables from there?
>
>
Does not seem to make much sense until we can tweak page size for different
data regions independently (now we can't). I would start with that one
first.

Sergi


--
> Denis
>
> On Mon, Nov 19, 2018 at 2:01 AM Sergi Vladykin 
> wrote:
>
> > Folks,
> >
> > I've implemented page compression for persistent store and going to merge
> > it to master.
> >
> > https://github.com/apache/ignite/pull/5200
> >
> > Some design notes:
> >
> > It employs "hole punching" approach, it means that the pages are kept
> > uncompressed in memory,
> > but when they get written to disk, they will be compressed and all the
> > extra file system blocks for the page will be released. Thus the storage
> > files become sparse.
> >
> > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
> > basically just takes only meaningful data from half-filled pages but does
> > not apply any compression. It is easy to add more if needed.
> >
> > Since we can release only full file system blocks which are typically 4k
> > size, user must configure page size to be at least multiple FS blocks,
> e.g.
> > 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> > pageSize = 4k / 16k = 0.25
> >
> > It is possible to enable compression for existing databases if they were
> > configured for large enough page size. In this case pages will be written
> > to disk in compressed form when updated, and the database will become
> > compressed gradually.
> >
> > There will be 2 new properties on CacheConfiguration
> > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk
> page
> > compression.
> >
> > Compression dictionaries are not supported at the time, but may in the
> > future. IMO it should be added as a separate feature if needed.
> >
> > The only supported platform for now is Linux. Since all popular file
> > systems support sparse files, it must be  relatively easy to support more
> > platforms.
> >
> > Please take a look and provide your thoughts and suggestions.
> >
> > Thanks!
> >
> > Sergi
> >
>

[MTCGA]: new failures in builds [2356663] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New test failure in master 
CacheExchangeMergeTest.testMergeServerAndClientJoin1 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=4021629064265895900=%3Cdefault%3E=testDetails
 Changes may lead to failure were done by 
 - dmitriy.govorukhin 
https://ci.ignite.apache.org/viewModification.html?modId=839470
 - bessonov.ip 
https://ci.ignite.apache.org/viewModification.html?modId=839471
 - bessonov.ip 
https://ci.ignite.apache.org/viewModification.html?modId=839416
 - ilantukh 
https://ci.ignite.apache.org/viewModification.html?modId=839419
 - bessonov.ip 
https://ci.ignite.apache.org/viewModification.html?modId=839413
 - verbalab 
https://ci.ignite.apache.org/viewModification.html?modId=839414
 - bessonov.ip 
https://ci.ignite.apache.org/viewModification.html?modId=839415
 - pudov.max 
https://ci.ignite.apache.org/viewModification.html?modId=839472

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:38:03 20-11-2018 

[GitHub] ignite pull request #5440: IGNITE-9339 Form-field-size improvements

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5440


---

[GitHub] ignite pull request #5440: IGNITE-9339 Form-field-size improvements

[Klaster1]

GitHub user Klaster1 opened a pull request:

https://github.com/apache/ignite/pull/5440

IGNITE-9339 Form-field-size improvements

@nva please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5440


commit c13ae1cbfa808b082c64239e744f6d93d0e66c25
Author: Ilya Borisov 
Date:   2018-10-18T10:18:17Z

IGNITE-9339 Add autofocus support to form-field-size.

commit 78561c8fdb20c36b1055846f8004f1c48e5cff0f
Author: Ilya Borisov 
Date:   2018-10-26T05:09:37Z

IGNITE-9339 Fix form-field-size validation message position.

commit db863c81036cc3f24b6223d2f2385282a5745592
Author: Ilya Borisov 
Date:   2018-10-26T07:30:18Z

IGNITE-9339 Add gigabytes scale to form-field-size.

commit 4494982b3aa8339076560b81da949a4ce4aabbe0
Author: Ilya Borisov 
Date:   2018-11-02T06:39:23Z

Fix form field by label selection.




---

Re: Network partitioning

[Denis Magda]

Hi,

Ignite comes with partition loss policies that might be useful:
https://apacheignite.readme.io/docs/partition-loss-policies

As for the network segmentation, check out GridGain that provides this for
Ignite users.

Denis


On Sat, Nov 17, 2018 at 7:39 AM Vj Anand 
wrote:

> Hi Ignite Developers,
>
> Here is the scenario that I am trying to  mitigate – I have a two node
> ignite cluster as cache; I am using Partitioning as the cache policy. I
> need to address network partitioning and avoid the split-brain scenario.
> What are the options?
> Are there hooks into the Ignite cache that I can use to be notified when,
> there is a partition, or when nodes rejoin? Any pointers? Code Snippets? I
> am trying to avoid external quorum mechanism.
>
> Thanks
> VJ
>

[GitHub] ignite pull request #5253: IGNITE-10142 Cache 8 tests optimization

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5253


---

[GitHub] ignite pull request #4933: IGNITE-9558

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4933


---

[GitHub] ignite pull request #5427: IGNITE-10321 Bug in CacheContinuousWithTransforme...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5427


---

[GitHub] ignite pull request #5379: IGNITE-10098 TcpCommunicationSpi for .NET misses ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5379


---

[GitHub] ignite pull request #5400: IGNITE-10192 OptimizedMarshallerTest#testAllocati...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5400


---

[GitHub] ignite pull request #5404: IGNITE-10285 Implement job stealing for U.doInPar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5404


---

[GitHub] ignite pull request #5410: IGNITE-10197 unexpected IllegalArgumentException ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/5410


---

[GitHub] ignite pull request #5439: IGNITE-10238

[NSAmelchev]

GitHub user NSAmelchev opened a pull request:

https://github.com/apache/ignite/pull/5439

IGNITE-10238

Fix possible deadlock of system threads.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NSAmelchev/ignite ignite-10238

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5439


commit 630156cf16168702dabab4fd7b2c41bf0eaf83e7
Author: NSAmelchev 
Date:   2018-11-19T23:01:09Z

Fix system threads deadlock




---

[GitHub] ignite pull request #5438: Tde with fixes

[nizhikov]

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite/pull/5438

Tde with fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite TDE-with-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5438.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5438


commit d56de82031de9607b533208e5005eb93db741660
Author: Nikolay Izhikov 
Date:   2018-10-31T18:20:24Z

IGNITE-9996: Performance drop fix. - Fixes #5215.

Signed-off-by: Nikolay Izhikov 

commit e11c199caa949000cddf059215b81afb2cb72730
Author: Nikolay Izhikov 
Date:   2018-11-17T07:24:19Z

IGNITE-9996: Performance drop fix base on instanceof NoopEncryptionSpi

commit eb7ddf5e3884c5ae468c5a1e822b8a17496398fd
Author: Nikolay Izhikov 
Date:   2018-11-19T12:05:38Z

IGNITE-9996: Final fix

commit 9cc555457dade2b781925d557963aac71879528c
Author: Nikolay Izhikov 
Date:   2018-11-19T20:14:10Z

IGNITE-9996: Performance drop fix base on instanceof NoopEncryptionSpi




---

[jira] [Created] (IGNITE-10337) [TC Bot] Test ticket

[PetrovMikhail (JIRA)]

PetrovMikhail created IGNITE-10337:
--

 Summary: [TC Bot] Test ticket
 Key: IGNITE-10337
 URL: https://issues.apache.org/jira/browse/IGNITE-10337
 Project: Ignite
  Issue Type: Task
Reporter: PetrovMikhail






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5437: Ignite 2.4.8 p7 test

[aealeksandrov]

GitHub user aealeksandrov opened a pull request:

https://github.com/apache/ignite/pull/5437

Ignite 2.4.8 p7 test

For team city run only.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.8-p7-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5437


commit 130adcf29ddb61f8e9baa784b81454d3ed7c3b75
Author: Pavel Tupitsyn 
Date:   2018-01-26T08:48:14Z

IGNITE-7530 .NET: Fix GetAll memory usage and performance in binary mode

This closes #3436

commit 824004909b23a9a7d599500967af34103acb8c47
Author: Igor Sapego 
Date:   2018-01-30T12:56:17Z

IGNITE-6810: Implemented SSL support for ODBC.

This closes #3361

commit 82da0b5e9dc2ee2339c3fb1023e35d415bf1b647
Author: Pavel Kuznetsov 
Date:   2018-02-07T12:37:52Z

IGNITE-6217: Added benchmark to compare JDBC vs native SQL

This closes #2558

commit 701c6f141f6812ad7bc050a86266e696cf5863ed
Author: tledkov-gridgain 
Date:   2018-02-08T12:29:42Z

IGNITE-6625: JDBC thin: support SSL connection to Ignite node

This closes #3233

commit 2d729bf5c6f6fca9d07be2d57850642ba4b55004
Author: tledkov-gridgain 
Date:   2018-02-09T11:08:15Z

IGNITE-6625: SSL support for thin JDBC: additional fix test; change error 
message

commit 8994f847d7f5f15db73706d9210cdccb1cf3fb26
Author: devozerov 
Date:   2018-02-12T13:34:24Z

IGNITE-7003: Fixed faulty test 
WalModeChangeAdvancedSelfTest.testJoinCoordinator.

commit b142712264007d7397d1594541681a4a7e3d4b93
Author: Igor Sapego 
Date:   2018-02-26T12:02:07Z

IGNITE-7362: Fixed PDO ODBC integration issue

commit a2b2aee52cc65d01f2ecaf9462adc4bd368438ea
Author: Igor Sapego 
Date:   2018-02-28T10:23:12Z

IGNITE-7763: Fixed 32bit tests configurations to prevent OOM

This closes #3557

commit 652f3c4cdbaad40f5de25b06f0c13710aa7f2fd9
Author: Pavel Kuznetsov 
Date:   2018-03-13T12:46:36Z

IGNITE-7531: Data load benchmarks. This closes #3571.

commit 9337a53d9fcd62af87f6760080d350b43e275105
Author: tledkov-gridgain 
Date:   2018-03-16T11:38:38Z

IGNITE-7879: Fixed splitter logic for DISTINCT with subqueries. This closes 
#3634.

commit 7bec8b13cb373002d2a6b1b268d410338259bac2
Author: Igor Sapego 
Date:   2018-03-19T11:17:33Z

IGNITE-7811: Implemented connection failover for ODBC

This closes #3643

commit e512e5e0a2602df0ecfb69b2b5efabce836b04db
Author: Igor Sapego 
Date:   2018-03-20T10:37:02Z

IGNITE-7888: Implemented login timeout for ODBC.

This closes #3657

commit 211fca3a55e84b78ff0c1af04d91e25d6fc846c4
Author: devozerov 
Date:   2018-03-20T11:13:46Z

IGNITE-7984: SQL: improved deadlock handling in DML. This closes #3655.

commit bcd2888d27afe65f1a060e35b99a05ea420979c7
Author: Roman Guseinov 
Date:   2018-02-16T09:57:26Z

IGNITE-7192: Implemented JDBC support FQDN to multiple IPs

This closes #3439

commit d2659d0ec9f6e1a0b905fc7bf23b65fd5522c80a
Author: Alexander Paschenko 
Date:   2018-03-14T09:23:37Z

IGNITE-7253: JDBC thin driver: implemented streaming. This closes #3499. 
This closes #3591.

commit bc9018ef8b116f81b8e06d2ff7651ba2b6c7beae
Author: tledkov-gridgain 
Date:   2018-03-19T08:01:26Z

IGNITE-7029: JDBC thin driver now supports multiple IP addresses with 
transparent failover. This closes #3585.

commit 587022862fd5bdbb076ab6207ae6fd9bc7583c13
Author: Sergey Chugunov 
Date:   2018-03-16T16:24:17Z

IGNITE-7964 rmvId is stored to MetaStorage metapage during operations - 
Fixes #3645.

commit 006ef4d15e4faedb6dfce6ce9637602055b97293
Author: tledkov-gridgain 
Date:   2018-03-22T11:47:06Z

IGNITE-7436: Simple username/password authentication. This closes #3483.

commit 1c7f59c90514670e802d9d07544b00b7562fe6d2
Author: Pavel Tupitsyn 
Date:   2018-01-30T09:48:16Z

.NET: Fix build status icon in README

commit 162df61b305fccfc55e186d07351727f35b55179
Author: Pavel Tupitsyn 
Date:   2018-02-01T11:53:16Z

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

This closes #3457

commit 9a0328ebbc9d52f8e96898a384fa45743d2efa5b
Author: Pavel Tupitsyn 
Date:   2018-02-02T12:01:27Z

.NET: Update README regarding C++ interop and thin client

commit b804cfea51c87724b45b40de4fd58d300c049be1
Author: Pavel Tupitsyn 
Date:   2018-01-31T09:39:19Z

.NET: Suppress API parity check for SSL in ClientConnectorConfiguration

commit 6f8014de7250c4c0d87cbc8764afae4a225f654b
Author: apopov 
Date:   2018-02-13T10:13:15Z

IGNITE-3111 .NET can be now configured SSL without Spring

This closes #3498

commit 5131bcd71ce787cf2c61bf98446f5ec0a616ab1c
Author: Pavel Tupitsin 
Date:   2018-02-16T20:36:01Z

IGNITE-3111 .NET: Configure SSL without Spring - cleanup

[jira] [Created] (IGNITE-10336) [TC Bot] Running build can cause bot to think there are no failures

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-10336:
---

 Summary: [TC Bot] Running build can cause bot to think there are 
no failures
 Key: IGNITE-10336
 URL: https://issues.apache.org/jira/browse/IGNITE-10336
 Project: Ignite
  Issue Type: Task
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov


Partially canceled builds support introduced a problem.

During RunAll run there is possibility bot will count tests and suites and show 
no blockers found, but actually, we need to wait queued/running completion 
before any visa can be shown.



https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull/5431/head=Latest
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5436: IGNITE-10043 Do not reset LOST partitions when on...

[agoncharuk]

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/5436

IGNITE-10043 Do not reset LOST partitions when only one node is left in a 
cluster



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10043

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5436


commit 5f61419270ca4813e8d0edbb4c23341591fa1816
Author: sboikov 
Date:   2018-11-15T14:14:21Z

ignite-10043

commit 51933a8e1f3840a4e6a18e8d32de33b775a35321
Author: sboikov 
Date:   2018-11-15T19:04:46Z

ignite-10043

commit 9a6d642f840897f8b70dc8b7bb7d7cbfedcbd0bd
Author: sboikov 
Date:   2018-11-16T09:58:21Z

ignite-10043

commit fb8fd071b1d69f2cb054d3798320380eb3401362
Author: sboikov 
Date:   2018-11-16T13:37:04Z

 merge

commit bc1ac15c36fb459f221c9b723ad82a6e43a5bfca
Author: sboikov 
Date:   2018-11-16T13:38:07Z

 merge

commit 96e2fddc9b7c18ec11ab181ad45633f4041a9aa3
Author: ibessonov 
Date:   2018-11-19T09:41:05Z

IGNITE-10192 Fixed OptimizedMarshallerTest#testAllocationOverflow throws 
OOME instead of expected IOE - Fixes #5400.

Signed-off-by: Dmitriy Govorukhin 

commit 166e87ed19929b9e064732752d5a5ce1069b5076
Author: Alexander Kalinin 
Date:   2018-11-19T09:47:12Z

IGNITE-10320 Web Console: Workaround for memory leak in chart component.

commit c9906aab8efa2c70a164fb593d9e22ef1edaa9ec
Author: ibessonov 
Date:   2018-11-19T09:50:56Z

IGNITE-10142 Scale factor added to several tests from Cache 8 suite. - 
Fixes #5253.

Signed-off-by: Dmitriy Govorukhin 

commit 9e3bd7dd686d3392e6bbe2c7defebe260eb029f7
Author: ibessonov 
Date:   2018-11-19T10:06:47Z

IGNITE-10197 Fixed unexpected IllegalArgumentException in 
IgniteDbPutGetAbstractTest#testRandomPutGetRemove - Fixes #5410.

Signed-off-by: Dmitriy Govorukhin 

commit a63a81a51418a8ae1dadff019d9e5eace38e1631
Author: Ilya Lantukh 
Date:   2018-11-19T12:07:49Z

IGNITE-9558 Avoid blocking transactions on client connect when possible - 
Fixes #4933.

commit acfdcdaa0f667b1ebebc26cea90df030804056a7
Author: Dmitriy Govorukhin 
Date:   2018-11-19T13:41:28Z

IGNITE-10285 Fixed U.doInParallel may lead to deadlock - Fixes #5404.

Signed-off-by: Dmitriy Govorukhin 

commit c711d4f6f7ecc6ad59643a2a82d7d487d8d45a90
Author: ibessonov 
Date:   2018-11-19T14:02:48Z

IGNITE-10321 Fix flaky test 
CacheContinuousWithTransformerReplicatedSelfTest.LocalEventListener - Fixes 
#5427.

Signed-off-by: Dmitriy Govorukhin 

commit a198b3ff5799899690419e3c0b6306a1eaebd6e7
Author: Max-Pudov 
Date:   2018-11-19T14:31:47Z

IGNITE-10098 .NET: Add missing TcpCommunicationSpi properties

This closes #5379

commit 673da458d94881874806a428bef13d829ac25cbb
Author: Alexey Goncharuk 
Date:   2018-11-19T17:54:02Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/ignite 
into ignite-10043




---

[GitHub] ignite pull request #5435: temporary PR to verify renamed tests on Teamcity ...

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/5435

temporary PR to verify renamed tests on Teamcity for review of 10174



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-renamed-tests-10174

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5435


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fd6b659bb8be1992618ce4ce91f568a0988b3afa
Author: Oleg Ignatenko 
Date:   2018-09-02T06:11:42Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944
Author: Oleg Ignatenko 
Date:   2018-09-03T10:27:35Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf
Author: Oleg Ignatenko 
Date:   2018-09-04T09:49:44Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5
Author: Oleg Ignatenko 
Date:   2018-09-05T10:50:28Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2
Author: Oleg Ignatenko 
Date:   2018-09-05T11:45:58Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 480ed78869277d7e32f725ab71bec9621f1ac03a
Author: Oleg Ignatenko 
Date:   2018-09-06T07:52:55Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit c99762498f617c0e98ea3062a43c0b30092166ef
Author: Oleg Ignatenko 
Date:   2018-09-06T14:45:04Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 2e17175225c72f747d370b5fee96f8be69d6d2cb
Author: Oleg Ignatenko 
Date:   2018-09-06T17:33:54Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f
Author: Oleg Ignatenko 
Date:   2018-09-07T13:38:51Z

Merge branch 

Re: [MTCGA]: new failures in builds [2339409] needs to be handled

[Vyacheslav Daradur]

I can't reproduce this tests locally, also the test is successful on
TC in last two days.

Looks like the failure relates to an engine of .NET Core platform
because the test passes without failures at .NET framework [1] in
comparison to .NET Core [2].

[1] 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2423313125230633239=testDetails_IgniteTests24Java8=%3Cdefault%3E
[2] 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8068196411489261503=%3Cdefault%3E=testDetails_IgniteTests24Java8=%3Cdefault%3E
On Sat, Nov 17, 2018 at 9:51 PM Vyacheslav Daradur  wrote:
>
> Found the issue https://issues.apache.org/jira/browse/IGNITE-9563
>
> I think we should mute the test if we won't be able to resolve this in
> the nearest time.
> On Sat, Nov 17, 2018 at 9:42 PM Vyacheslav Daradur  
> wrote:
> >
> > Looks like the test has a high failure rate. I'll take a look.
> > Is there a Jira issue?
> > On Sat, Nov 17, 2018 at 4:18 PM  wrote:
> > >
> > > Hi Igniters,
> > >
> > >  I've detected some new issue on TeamCity to be handled. You are more 
> > > than welcomed to help.
> > >
> > >  If your changes can lead to this failure(s): We're grateful that you 
> > > were a volunteer to make the contribution to this project, but things 
> > > change and you may no longer be able to finalize your contribution.
> > >  Could you respond to this email and indicate if you wish to continue and 
> > > fix test failures or step down and some committer may revert you commit.
> > >
> > >  *New stable failure of a flaky test in master 
> > > IgniteConfigurationTest.TestSpringXml 
> > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8068196411489261503=%3Cdefault%3E=testDetails
> > >  No changes in the build
> > >
> > >  - Here's a reminder of what contributors were agreed to do 
> > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> > >  - Should you have any questions please contact 
> > > dev@ignite.apache.org
> > >
> > > Best Regards,
> > > Apache Ignite TeamCity Bot
> > > https://github.com/apache/ignite-teamcity-bot
> > > Notification generated at 16:18:21 17-11-2018
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.

Re: Disk page compression for Ignite persistent store

[Denis Magda]

Hi Sergi,

Didn't know you were cooking this dish in the background ) Excellent.  Just
to be sure, that's part of this IEP, right?
https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-Withoutin-memorycompression

Since we can release only full file system blocks which are typically 4k
> size, user must configure page size to be at least multiple FS blocks, e.g.
> 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> pageSize = 4k / 16k = 0.25


How to we handle the case if the page size is not a multiple of 4K? What is
the most optimal page size if the user wants to get the best compression?
Probably, we can adjust the default page size automatically if it's a clean
deployment.

There will be 2 new properties on CacheConfiguration
> (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
> compression.


How about setting it at DataRegionConfiguration level as well so that it's
applied for all the caches/tables from there?

--
Denis

On Mon, Nov 19, 2018 at 2:01 AM Sergi Vladykin 
wrote:

> Folks,
>
> I've implemented page compression for persistent store and going to merge
> it to master.
>
> https://github.com/apache/ignite/pull/5200
>
> Some design notes:
>
> It employs "hole punching" approach, it means that the pages are kept
> uncompressed in memory,
> but when they get written to disk, they will be compressed and all the
> extra file system blocks for the page will be released. Thus the storage
> files become sparse.
>
> Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
> basically just takes only meaningful data from half-filled pages but does
> not apply any compression. It is easy to add more if needed.
>
> Since we can release only full file system blocks which are typically 4k
> size, user must configure page size to be at least multiple FS blocks, e.g.
> 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> pageSize = 4k / 16k = 0.25
>
> It is possible to enable compression for existing databases if they were
> configured for large enough page size. In this case pages will be written
> to disk in compressed form when updated, and the database will become
> compressed gradually.
>
> There will be 2 new properties on CacheConfiguration
> (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
> compression.
>
> Compression dictionaries are not supported at the time, but may in the
> future. IMO it should be added as a separate feature if needed.
>
> The only supported platform for now is Linux. Since all popular file
> systems support sparse files, it must be  relatively easy to support more
> platforms.
>
> Please take a look and provide your thoughts and suggestions.
>
> Thanks!
>
> Sergi
>

[GitHub] ignite pull request #5420: for tests

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/5420


---

[GitHub] ignite pull request #5434: IGNITE-10335 move ML examples datasets files to r...

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/5434

IGNITE-10335 move ML examples datasets files to resources



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10335

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5434


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fd6b659bb8be1992618ce4ce91f568a0988b3afa
Author: Oleg Ignatenko 
Date:   2018-09-02T06:11:42Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944
Author: Oleg Ignatenko 
Date:   2018-09-03T10:27:35Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf
Author: Oleg Ignatenko 
Date:   2018-09-04T09:49:44Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5
Author: Oleg Ignatenko 
Date:   2018-09-05T10:50:28Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2
Author: Oleg Ignatenko 
Date:   2018-09-05T11:45:58Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 480ed78869277d7e32f725ab71bec9621f1ac03a
Author: Oleg Ignatenko 
Date:   2018-09-06T07:52:55Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit c99762498f617c0e98ea3062a43c0b30092166ef
Author: Oleg Ignatenko 
Date:   2018-09-06T14:45:04Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 2e17175225c72f747d370b5fee96f8be69d6d2cb
Author: Oleg Ignatenko 
Date:   2018-09-06T17:33:54Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f
Author: Oleg Ignatenko 
Date:   2018-09-07T13:38:51Z

Merge branch 'master' of 

[GitHub] asfgit closed pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

asfgit closed pull request #74: IGNITE-10275 Refactor of visa caching. Jira 
spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
index f3ae04aa..7df50747 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
@@ -178,8 +178,12 @@ private BranchesTracked getTrackedBranches() {
 
 String comment = generateJiraComment(suitesStatuses, build.webUrl);
 
-blockers = suitesStatuses.stream().mapToInt(suite ->
-suite.testFailures.size()).sum();
+blockers = suitesStatuses.stream().mapToInt(suite -> {
+if (suite.testFailures.isEmpty())
+return 1;
+
+return suite.testFailures.size();})
+.sum();
 
 res = objectMapper.readValue(teamcity.sendJiraComment(ticket, 
comment), JiraCommentResponse.class);
 }
diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/BuildObserver.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/BuildObserver.java
index cf294684..b4686f74 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/BuildObserver.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/BuildObserver.java
@@ -17,19 +17,24 @@
 
 package org.apache.ignite.ci.observer;
 
-import java.util.Collection;
 import java.util.Objects;
 import java.util.Timer;
 import javax.inject.Inject;
+import org.apache.ignite.ci.IAnalyticsEnabledTeamcity;
+import org.apache.ignite.ci.ITcHelper;
 import org.apache.ignite.ci.tcmodel.result.Build;
 import org.apache.ignite.ci.user.ICredentialsProv;
-import org.apache.ignite.ci.web.model.VisaRequest;
-import org.apache.ignite.ci.web.model.hist.VisasHistoryStorage;
+import org.apache.ignite.ci.web.model.ContributionKey;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  *
  */
 public class BuildObserver {
+/** Logger. */
+private static final Logger logger = 
LoggerFactory.getLogger(BuildObserver.class);
+
 /** Time between observing actions in milliseconds. */
 private static final long PERIOD = 10 * 60 * 1_000;
 
@@ -39,8 +44,8 @@
 /** Task, which should be done periodically. */
 private ObserverTask observerTask;
 
-/** Visas History Storage. */
-@Inject private VisasHistoryStorage visasStorage;
+/** Helper. */
+@Inject private ITcHelper tcHelper;
 
 /**
  */
@@ -60,6 +65,25 @@ public void stop() {
 timer.cancel();
 }
 
+/** */
+public ObserverTask getObserverTask() {
+return observerTask;
+}
+
+/** */
+public boolean stopObservation(ContributionKey key) {
+try {
+observerTask.removeBuildInfo(key);
+
+return true;
+}
+catch (Exception e) {
+logger.error("Observation stop: " + e.getMessage(), e);
+
+return false;
+}
+}
+
 /**
  * @param srvId Server id.
  * @param prov Credentials.
@@ -69,28 +93,26 @@ public void stop() {
 public void observe(String srvId, ICredentialsProv prov, String ticket, 
String branchForTc, Build... builds) {
 BuildsInfo buildsInfo = new BuildsInfo(srvId, prov, ticket, 
branchForTc, builds);
 
-visasStorage.put(new VisaRequest(buildsInfo));
-
 observerTask.addInfo(buildsInfo);
 }
 
 /**
- * @param srvId Server id.
- * @param branch Branch.
+ * @param key {@code Contribution Key}.
  */
-public String getObservationStatus(String srvId, String branch) {
+public String getObservationStatus(ContributionKey key) {
 StringBuilder sb = new StringBuilder();
 
-Collection builds = observerTask.getInfos();
+BuildsInfo buildsInfo = observerTask.getInfo(key);
+
+ICredentialsProv creds = tcHelper.getServerAuthorizerCreds();
+
+IAnalyticsEnabledTeamcity teamcity = tcHelper.server(key.srvId, creds);
 
-for (BuildsInfo bi : builds) {
-if (Objects.equals(bi.branchForTc, branch)
-&& Objects.equals(bi.srvId, srvId)) {
-sb.append(bi.ticket).append(" to be commented, waiting for 
builds. ");
-sb.append(bi.finishedBuildsCount());
-sb.append(" builds done from ");
-sb.append(bi.buildsCount());
-}
+if (Objects.nonNull(buildsInfo)) {
+sb.append(buildsInfo.ticket).append(" to be commented, waiting for 

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

[Vladimir Ozerov]

Denis,

I partially agree with you. But there are several problem with syntax
proposed by you:
1) This is harder to implement technically - more parsing logic to
implement. Ok, this is our internal problem, users do not care about it
2) User will have to consult to docs in any case
3) "nodeId" is not really node ID. For Ignite users node ID is UUID. In our
case this is node order, and we intentionally avoided any naming here.
Query is just identified by a string, no more than that
4) Proposed syntax is more verbose and open ways for misuse. E.g. what is
"KILL QUERY WHERE queryId=1234"?

I am not 100% satisfied with both variants, but the first one looks simpler
to me. Remember, that user will not guess query ID. Instead, he will get
the list of running queries with some other syntax. What we need to
understand for now is how this syntax will look like. I think that we
should implement getting list of running queries, and only then start
working on cancellation.

Vladimir.


On Mon, Nov 19, 2018 at 7:02 PM Denis Mekhanikov 
wrote:

> Guys,
>
> Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me.
> I'm going to look up in documentation, which parameter goes first in this
> query every time I use it.
> I like the syntax, that Igor suggested more.
> Will it be better if we make *nodeId* and *queryId *named properties?
>
> Something like this:
> KILL QUERY WHERE nodeId=25 and queryId=1234
>
> Denis
>
> пт, 16 нояб. 2018 г. в 14:12, Юрий :
>
> > I fully agree with last sentences and can start to implement this part.
> >
> > Guys, thanks for your productive participate at discussion.
> >
> > пт, 16 нояб. 2018 г. в 2:53, Denis Magda :
> >
> > > Vladimir,
> > >
> > > Thanks, make perfect sense to me.
> > >
> > >
> > > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > The idea is that QueryDetailMetrics will be exposed through separate
> > > > "historical" SQL view in addition to current API. So we are on the
> same
> > > > page here.
> > > >
> > > > As far as query ID I do not see any easy way to operate on a single
> > > integer
> > > > value (even 64bit). This is distributed system - we do not want to
> have
> > > > coordination between nodes to get query ID. And coordination is the
> > only
> > > > possible way to get sexy "long". Instead, I would propose to form ID
> > from
> > > > node order and query counter within node. This will be (int, long)
> > pair.
> > > > For use convenience we may convert it to a single string, e.g.
> > > > "[node_order].[query_counter]". Then the syntax would be:
> > > >
> > > > KILL QUERY '25.1234'; // Kill query 1234 on node 25
> > > > KILL QUERY '25.*; // Kill all queries on the node 25
> > > >
> > > > Makes sense?
> > > >
> > > > Vladimir.
> > > >
> > > > On Wed, Nov 14, 2018 at 1:25 PM Denis Magda 
> wrote:
> > > >
> > > > > Yury,
> > > > >
> > > > > As I understand you mean that the view should contains both running
> > and
> > > > > > finished queries. If be honest for the view I was going to use
> just
> > > > > queries
> > > > > > running right now. For finished queries I thought about another
> > view
> > > > with
> > > > > > another set of fields which should include I/O related ones. Is
> it
> > > > works?
> > > > >
> > > > >
> > > > > Got you, so if only running queries are there then your initial
> > > proposal
> > > > > makes total sense. Not sure we need a view of the finished queries.
> > It
> > > > will
> > > > > be possible to analyze them through the updated DetailedMetrics
> > > approach,
> > > > > won't it?
> > > > >
> > > > > For "KILL QUERY node_id query_id"  node_id required as part of
> unique
> > > key
> > > > > > of query and help understand Ignite which node start the
> > distributed
> > > > > query.
> > > > > > Use both parameters will allow cheap generate unique key across
> all
> > > > > nodes.
> > > > > > Node which started a query can cancel it on all nodes participate
> > > > nodes.
> > > > > > So, to stop any queries initially we need just send the cancel
> > > request
> > > > to
> > > > > > node who started the query. This mechanism is already in Ignite.
> > > > >
> > > > >
> > > > > Can we locate node_id behind the scenes if the user supplies
> query_id
> > > > only?
> > > > > A query record in the view already contains query_id and node_id
> and
> > it
> > > > > sounds like an extra work for the user to fill in all the details
> for
> > > us.
> > > > > Embed node_id into query_id if you'd like to avoid extra network
> hops
> > > for
> > > > > query_id to node_id mapping.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Wed, Nov 14, 2018 at 1:04 AM Юрий 
> > > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > Under the hood 'time' will be as startTime, but for system view I
> > > > planned
> > > > > > use duration which will be simple calculated as now - startTime.
> > So,
> > > > > there
> > > > > > is't a performance issue.
> > > > > > As I understand you 

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

[Юрий]

Hi Denis,

It's not a problem, the full query id could be get from additional column
from *running_queries* view. So you may not known real meaning of each of
part of the string to use it.
Is it works?

пн, 19 нояб. 2018 г. в 19:02, Denis Mekhanikov :

> Guys,
>
> Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me.
> I'm going to look up in documentation, which parameter goes first in this
> query every time I use it.
> I like the syntax, that Igor suggested more.
> Will it be better if we make *nodeId* and *queryId *named properties?
>
> Something like this:
> KILL QUERY WHERE nodeId=25 and queryId=1234
>
> Denis
>
> пт, 16 нояб. 2018 г. в 14:12, Юрий :
>
> > I fully agree with last sentences and can start to implement this part.
> >
> > Guys, thanks for your productive participate at discussion.
> >
> > пт, 16 нояб. 2018 г. в 2:53, Denis Magda :
> >
> > > Vladimir,
> > >
> > > Thanks, make perfect sense to me.
> > >
> > >
> > > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > The idea is that QueryDetailMetrics will be exposed through separate
> > > > "historical" SQL view in addition to current API. So we are on the
> same
> > > > page here.
> > > >
> > > > As far as query ID I do not see any easy way to operate on a single
> > > integer
> > > > value (even 64bit). This is distributed system - we do not want to
> have
> > > > coordination between nodes to get query ID. And coordination is the
> > only
> > > > possible way to get sexy "long". Instead, I would propose to form ID
> > from
> > > > node order and query counter within node. This will be (int, long)
> > pair.
> > > > For use convenience we may convert it to a single string, e.g.
> > > > "[node_order].[query_counter]". Then the syntax would be:
> > > >
> > > > KILL QUERY '25.1234'; // Kill query 1234 on node 25
> > > > KILL QUERY '25.*; // Kill all queries on the node 25
> > > >
> > > > Makes sense?
> > > >
> > > > Vladimir.
> > > >
> > > > On Wed, Nov 14, 2018 at 1:25 PM Denis Magda 
> wrote:
> > > >
> > > > > Yury,
> > > > >
> > > > > As I understand you mean that the view should contains both running
> > and
> > > > > > finished queries. If be honest for the view I was going to use
> just
> > > > > queries
> > > > > > running right now. For finished queries I thought about another
> > view
> > > > with
> > > > > > another set of fields which should include I/O related ones. Is
> it
> > > > works?
> > > > >
> > > > >
> > > > > Got you, so if only running queries are there then your initial
> > > proposal
> > > > > makes total sense. Not sure we need a view of the finished queries.
> > It
> > > > will
> > > > > be possible to analyze them through the updated DetailedMetrics
> > > approach,
> > > > > won't it?
> > > > >
> > > > > For "KILL QUERY node_id query_id"  node_id required as part of
> unique
> > > key
> > > > > > of query and help understand Ignite which node start the
> > distributed
> > > > > query.
> > > > > > Use both parameters will allow cheap generate unique key across
> all
> > > > > nodes.
> > > > > > Node which started a query can cancel it on all nodes participate
> > > > nodes.
> > > > > > So, to stop any queries initially we need just send the cancel
> > > request
> > > > to
> > > > > > node who started the query. This mechanism is already in Ignite.
> > > > >
> > > > >
> > > > > Can we locate node_id behind the scenes if the user supplies
> query_id
> > > > only?
> > > > > A query record in the view already contains query_id and node_id
> and
> > it
> > > > > sounds like an extra work for the user to fill in all the details
> for
> > > us.
> > > > > Embed node_id into query_id if you'd like to avoid extra network
> hops
> > > for
> > > > > query_id to node_id mapping.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Wed, Nov 14, 2018 at 1:04 AM Юрий 
> > > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > Under the hood 'time' will be as startTime, but for system view I
> > > > planned
> > > > > > use duration which will be simple calculated as now - startTime.
> > So,
> > > > > there
> > > > > > is't a performance issue.
> > > > > > As I understand you mean that the view should contains both
> running
> > > and
> > > > > > finished queries. If be honest for the view I was going to use
> just
> > > > > queries
> > > > > > running right now. For finished queries I thought about another
> > view
> > > > with
> > > > > > another set of fields which should include I/O related ones. Is
> it
> > > > works?
> > > > > >
> > > > > > For "KILL QUERY node_id query_id"  node_id required as part of
> > unique
> > > > key
> > > > > > of query and help understand Ignite which node start the
> > distributed
> > > > > query.
> > > > > > Use both parameters will allow cheap generate unique key across
> all
> > > > > nodes.
> > > > > > Node which started a query can cancel it on all nodes participate
> > > > nodes.
> > > > > > So, to stop any queries 

Re: proposed realization KILL QUERY command

[Vladimir Ozerov]

Hi Yuriy,

I think we can use MANAGEMENT_POOL for this. It is already used for some
internal Ignite tasks, and it appears to be a good candidate to process
cancel requests.

But there are several things which are not clear enough for me at the
moment:
1) How user is going to get the list of running queries in the first place?
Do we already have any SQL commands/views to get this information?
2) We need to ensure that KILL command will be processed properly by all
kinds of SQL queries - SELECT/DML/DDL, non-transactional or transactional,
local queries and distributed queries. Will we be able to support all these
modes?

Vladimir.

On Mon, Nov 19, 2018 at 6:37 PM Юрий  wrote:

> Hi Igniters,
>
> Earlier we agreed about syntax KILL QUERY '[node_order].[query_counter]',
> e.g. KILL QUERY '25.123' for single query  or KILL QUERY '25.*' for all
> queries on the node. Which is part of IEP-29
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> >
> .
>
> Now I want to discuss internal realization of KILL query feature.
>
> My current vision is following:
> After parsing, Ignite create KILL query command with two parameters:
> nodeOrderId, nodeQryId. To determine that need to kill all queries on a
> node we can use negative value of query id, due to qry id always have
> positive values.
> The command process at IgniteH2Indexing as native command.
> By nodeOrderId we find node which initial for the query and send to the
> node new GridQueryKillRequest with nodeQryId to TOPIC_QUERY with not QUERY
> POOL executor.
> At GridReduceQueryExecutor we add support of processing new
> GridQueryKillRequest
> which just run already exists cancelQueries method with given qryId or with
> all qryIds which currently running at the node in case at initial KILL
> QUERY parameters used star symbol.
>
> I have a doubt which of thread pool we should use to process
> GridQueryKillRequest.
> My opinion it shouldn't be QUERY pool, due to the pool can be fully used by
> executing queries, it such case we can't cancel query immediately. May we
> use one of already existed pool or create new one? Or may be I'm mistaken
> and it should use QUERY pool.
>
> What do you think about proposed plan of implementation?
>
> And please give comments about which of thread pool will be better to use
> for kill query requests. It's small, but really important part of the
> realization.
>
>
> Thanks.
>
>
> --
> Живи с улыбкой! :D
>

[jira] [Created] (IGNITE-10335) move ML examples datasets files to resources

[Oleg Ignatenko (JIRA)]

Oleg Ignatenko created IGNITE-10335:
---

 Summary: move ML examples datasets files to resources
 Key: IGNITE-10335
 URL: https://issues.apache.org/jira/browse/IGNITE-10335
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Affects Versions: 2.7
Reporter: Oleg Ignatenko


ML examples datasets files were moved from resources to some folder under 
src/main/java in the course of IGNITE-10209.

This was done as a quick workaround to unblock another change. However, 
resources is a more appropriate location for this kind of content and in the 
course of IGNITE-10234 we discovered that there is a way to make things work 
with dataset files in this folder.

Solution appears to be to rework {{SandboxMLCache}} to use 
{{IgniteUtils.resolveIgnitePath}} API like it is done in 
[SqlJdbcCopyExample|https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlJdbcCopyExample.java].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10334) XAException not causing a rollback exception to propagate to application server

[Dionysis Stavropoulos (JIRA)]

Dionysis Stavropoulos created IGNITE-10334:
--

 Summary: XAException not causing a rollback exception to propagate 
to application server
 Key: IGNITE-10334
 URL: https://issues.apache.org/jira/browse/IGNITE-10334
 Project: Ignite
  Issue Type: Bug
  Components: cache, cassandra
Affects Versions: 2.6, 2.5, 2.4, 2.3
 Environment: Thorntail: 2.2.0.Final

Apache Ignite 2.6

Cassandra driver 3.0.0
Reporter: Dionysis Stavropoulos
 Attachments: ignite-config.xml, log.txt

When trying to put a null value in the POJO key of a cached object, although 
the error that is in the attached log occurs the exception is not propagated to 
the application server (thorntail) and transaction's state is marked as 
committed.



We can see that in 
org.apache.ignite.internal.processors.cache.jta.CacheJtaResource.java - line 
164, if an error occur in the commit() method an XAException is 
thrown(correctly) but with errorCode=0 (which as we understand means that the 
transaction was committed and that's why no rollback is propagated to the 
transaction manager).

See commit method below:

@Override public void commit(Xid xid, boolean onePhase) throws XAException {
 assert this.xid.equals(xid);

if (log.isDebugEnabled())
 log.debug("XA resource commit(...) [xid=" + xid + ", onePhase=" + onePhase + 
"]");

try {
 ctx.cache().context().commitTxAsync(cacheTx).get();<<< if error occur
 }
 catch (IgniteCheckedException e) {
 throwException("Failed to commit cache transaction: " + e.getMessage(), e);
 }
 }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

[Denis Mekhanikov]

Guys,

Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me.
I'm going to look up in documentation, which parameter goes first in this
query every time I use it.
I like the syntax, that Igor suggested more.
Will it be better if we make *nodeId* and *queryId *named properties?

Something like this:
KILL QUERY WHERE nodeId=25 and queryId=1234

Denis

пт, 16 нояб. 2018 г. в 14:12, Юрий :

> I fully agree with last sentences and can start to implement this part.
>
> Guys, thanks for your productive participate at discussion.
>
> пт, 16 нояб. 2018 г. в 2:53, Denis Magda :
>
> > Vladimir,
> >
> > Thanks, make perfect sense to me.
> >
> >
> > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov 
> > wrote:
> >
> > > Denis,
> > >
> > > The idea is that QueryDetailMetrics will be exposed through separate
> > > "historical" SQL view in addition to current API. So we are on the same
> > > page here.
> > >
> > > As far as query ID I do not see any easy way to operate on a single
> > integer
> > > value (even 64bit). This is distributed system - we do not want to have
> > > coordination between nodes to get query ID. And coordination is the
> only
> > > possible way to get sexy "long". Instead, I would propose to form ID
> from
> > > node order and query counter within node. This will be (int, long)
> pair.
> > > For use convenience we may convert it to a single string, e.g.
> > > "[node_order].[query_counter]". Then the syntax would be:
> > >
> > > KILL QUERY '25.1234'; // Kill query 1234 on node 25
> > > KILL QUERY '25.*; // Kill all queries on the node 25
> > >
> > > Makes sense?
> > >
> > > Vladimir.
> > >
> > > On Wed, Nov 14, 2018 at 1:25 PM Denis Magda  wrote:
> > >
> > > > Yury,
> > > >
> > > > As I understand you mean that the view should contains both running
> and
> > > > > finished queries. If be honest for the view I was going to use just
> > > > queries
> > > > > running right now. For finished queries I thought about another
> view
> > > with
> > > > > another set of fields which should include I/O related ones. Is it
> > > works?
> > > >
> > > >
> > > > Got you, so if only running queries are there then your initial
> > proposal
> > > > makes total sense. Not sure we need a view of the finished queries.
> It
> > > will
> > > > be possible to analyze them through the updated DetailedMetrics
> > approach,
> > > > won't it?
> > > >
> > > > For "KILL QUERY node_id query_id"  node_id required as part of unique
> > key
> > > > > of query and help understand Ignite which node start the
> distributed
> > > > query.
> > > > > Use both parameters will allow cheap generate unique key across all
> > > > nodes.
> > > > > Node which started a query can cancel it on all nodes participate
> > > nodes.
> > > > > So, to stop any queries initially we need just send the cancel
> > request
> > > to
> > > > > node who started the query. This mechanism is already in Ignite.
> > > >
> > > >
> > > > Can we locate node_id behind the scenes if the user supplies query_id
> > > only?
> > > > A query record in the view already contains query_id and node_id and
> it
> > > > sounds like an extra work for the user to fill in all the details for
> > us.
> > > > Embed node_id into query_id if you'd like to avoid extra network hops
> > for
> > > > query_id to node_id mapping.
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Wed, Nov 14, 2018 at 1:04 AM Юрий 
> > > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > Under the hood 'time' will be as startTime, but for system view I
> > > planned
> > > > > use duration which will be simple calculated as now - startTime.
> So,
> > > > there
> > > > > is't a performance issue.
> > > > > As I understand you mean that the view should contains both running
> > and
> > > > > finished queries. If be honest for the view I was going to use just
> > > > queries
> > > > > running right now. For finished queries I thought about another
> view
> > > with
> > > > > another set of fields which should include I/O related ones. Is it
> > > works?
> > > > >
> > > > > For "KILL QUERY node_id query_id"  node_id required as part of
> unique
> > > key
> > > > > of query and help understand Ignite which node start the
> distributed
> > > > query.
> > > > > Use both parameters will allow cheap generate unique key across all
> > > > nodes.
> > > > > Node which started a query can cancel it on all nodes participate
> > > nodes.
> > > > > So, to stop any queries initially we need just send the cancel
> > request
> > > to
> > > > > node who started the query. This mechanism is already in Ignite.
> > > > >
> > > > > Native SQL APIs will automatically support the futures after
> > > implementing
> > > > > for thin clients. So we are good here.
> > > > >
> > > > >
> > > > >
> > > > > вт, 13 нояб. 2018 г. в 18:52, Denis Magda :
> > > > >
> > > > > > Yury,
> > > > > >
> > > > > > Please consider the following:
> > > > > >
> > > > > >- If we record the duration instead of startTime, then the
> > former
> > > > 

Re: Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Vladimir Ozerov]

There is no public API currently. This information can be extracted using
JDBC metadata requests.

On Mon, Nov 19, 2018 at 6:50 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Vladimir,
>
> What is the best way to get current schema information (list of tables,
> columns, etc.)?
>
> -Val
>
> On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov 
> wrote:
>
> > Hi,
> >
> > In this case Spark integration should be fixed. as we never stated that
> DDL
> > updates will be reflected in IgniteCache.getConfiguration().
> >
> >
> > On Mon, Nov 19, 2018 at 11:58 AM Ray  wrote:
> >
> > > When user performs column and index modification operation in SQL(ex
> > create
> > > index, drop index, add column, drop column),  QueryEntity in
> > > CacheConfiguration for the modified cache is not updated.
> > >
> > > Here's my analysis
> > >
> > > QueryEntity in QuerySchema is a local copy of the original QueryEntity,
> > so
> > > the original QueryEntity is not updated when modification happens.
> > >
> > > I have created a ticket for this issue
> > > https://issues.apache.org/jira/browse/IGNITE-10314
> > >
> > > But as Vlad said in the comments "public configuration is immutable, it
> > > represents initial cache parameters. So it is expected that
> configuration
> > > will not be updated after DDL commands. Real changes are accumulated in
> > > separate query entity which is hidden from user and used internally"
> > >
> > > But I think it's only reasonable to return the newest QueryEntity to
> > user.
> > >
> > > For example, a user adds a column to a table then he reads data using
> > Spark
> > > data frame API which currently relies on QueryEntity to construct data
> > > frame
> > > schema, so user will get wrong schema.
> > >
> > > What do you guys think?
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>

Re: Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Valentin Kulichenko]

Vladimir,

What is the best way to get current schema information (list of tables,
columns, etc.)?

-Val

On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov 
wrote:

> Hi,
>
> In this case Spark integration should be fixed. as we never stated that DDL
> updates will be reflected in IgniteCache.getConfiguration().
>
>
> On Mon, Nov 19, 2018 at 11:58 AM Ray  wrote:
>
> > When user performs column and index modification operation in SQL(ex
> create
> > index, drop index, add column, drop column),  QueryEntity in
> > CacheConfiguration for the modified cache is not updated.
> >
> > Here's my analysis
> >
> > QueryEntity in QuerySchema is a local copy of the original QueryEntity,
> so
> > the original QueryEntity is not updated when modification happens.
> >
> > I have created a ticket for this issue
> > https://issues.apache.org/jira/browse/IGNITE-10314
> >
> > But as Vlad said in the comments "public configuration is immutable, it
> > represents initial cache parameters. So it is expected that configuration
> > will not be updated after DDL commands. Real changes are accumulated in
> > separate query entity which is hidden from user and used internally"
> >
> > But I think it's only reasonable to return the newest QueryEntity to
> user.
> >
> > For example, a user adds a column to a table then he reads data using
> Spark
> > data frame API which currently relies on QueryEntity to construct data
> > frame
> > schema, so user will get wrong schema.
> >
> > What do you guys think?
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[GitHub] ignite pull request #5433: IGNITE-10044 Proper LOST partitions handling

[agoncharuk]

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/5433

IGNITE-10044 Proper LOST partitions handling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10044

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5433


commit 5f61419270ca4813e8d0edbb4c23341591fa1816
Author: sboikov 
Date:   2018-11-15T14:14:21Z

ignite-10043

commit 51933a8e1f3840a4e6a18e8d32de33b775a35321
Author: sboikov 
Date:   2018-11-15T19:04:46Z

ignite-10043

commit a24688616b95ed2889c6e802b6fa55df0c0fcb0c
Author: sboikov 
Date:   2018-11-16T09:57:35Z

ignite-10044

commit 9a0351994a93d5baa5d53035a3c1b04963a58797
Author: sboikov 
Date:   2018-11-16T12:41:43Z

ignite-10044




---

proposed realization KILL QUERY command

[Юрий]

Hi Igniters,

Earlier we agreed about syntax KILL QUERY '[node_order].[query_counter]',
e.g. KILL QUERY '25.123' for single query  or KILL QUERY '25.*' for all
queries on the node. Which is part of IEP-29

.

Now I want to discuss internal realization of KILL query feature.

My current vision is following:
After parsing, Ignite create KILL query command with two parameters:
nodeOrderId, nodeQryId. To determine that need to kill all queries on a
node we can use negative value of query id, due to qry id always have
positive values.
The command process at IgniteH2Indexing as native command.
By nodeOrderId we find node which initial for the query and send to the
node new GridQueryKillRequest with nodeQryId to TOPIC_QUERY with not QUERY
POOL executor.
At GridReduceQueryExecutor we add support of processing new
GridQueryKillRequest
which just run already exists cancelQueries method with given qryId or with
all qryIds which currently running at the node in case at initial KILL
QUERY parameters used star symbol.

I have a doubt which of thread pool we should use to process
GridQueryKillRequest.
My opinion it shouldn't be QUERY pool, due to the pool can be fully used by
executing queries, it such case we can't cancel query immediately. May we
use one of already existed pool or create new one? Or may be I'm mistaken
and it should use QUERY pool.

What do you think about proposed plan of implementation?

And please give comments about which of thread pool will be better to use
for kill query requests. It's small, but really important part of the
realization.


Thanks.


-- 
Живи с улыбкой! :D

[GitHub] ignite pull request #5432: Ignite 2.5.1 p160

[antonovsergey93]

GitHub user antonovsergey93 opened a pull request:

https://github.com/apache/ignite/pull/5432

Ignite 2.5.1 p160



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.5.1-p160

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5432


commit afeaeea37a687c7329ae98a1a1925abf7727a812
Author: Evgeny Stanilovskiy 
Date:   2018-07-23T08:56:21Z

IGNITE-8892 Fixed OOME when scan query is used for a big partition - Fixes 
#4391.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit a164296)

commit 47d299c4961143f27d1f8aaa0d8c8e26b8ade7ed
Author: Evgeny Stanilovskiy 
Date:   2018-07-23T08:56:21Z

IGNITE-8892 Fixed OOME when scan query is used for a big partition - Fixes 
#4391.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit a164296)

commit a6a09865f776453ed88c093f4a58852ecb7efb0e
Author: Andrey V. Mashenkov 
Date:   2018-07-23T09:39:17Z

Merge remote-tracking branch 'origin/ignite-2.5.1-master' into 
ignite-2.5.1-master

commit e2695cd960b7f0cbbba545ef60645ea964f3d7fa
Author: Dmitriy Govorukhin 
Date:   2018-07-23T08:25:49Z

IGNITE-9042 Fixed partial tranasaction state wheh transaction is timed out 
- Fixes #4397.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 33f485a)

commit 702c31859251738b187e8db2a8155264f39b8b52
Author: Ivan Daschinskiy 
Date:   2018-07-23T12:29:08Z

IGNITE-8820 Fix rollback logic when tx is initiated from client. - Fixes 
#4399.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 5794eb0)

commit 32777e77797f4cf3260d42d4ca408336247d4e55
Author: Dmitriy Govorukhin 
Date:   2018-07-23T15:01:37Z

IGNITE-9049 Fixed write of SWITCH_SEGMENT_RECORD at the end of a segment 
file - Fixes #4401.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 713a428)

commit bbf8f83afe3b07e807912d9bab78873edf698d4d
Author: Andrey V. Mashenkov 
Date:   2018-07-24T14:38:34Z

IGNITE-8892 Fixed CacheQueryFuture usage in DataStructures processor - 
Fixes #4415.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 281a400)

commit e0404913088b5cd97cd79de1b62e6aaa4c00b5d2
Author: Andrey V. Mashenkov 
Date:   2018-07-24T15:24:57Z

Merge remote-tracking branch 'origin/ignite-2.5.1-master' into 
ignite-2.5.1-master

commit 86c52700a007368a351ac3595a8e38bb02923080
Author: Sergey Chugunov 
Date:   2018-07-25T13:26:12Z

IGNITE-9040 StopNodeFailureHandler is not able to stop node correctly on 
node segmentation

Signed-off-by: Andrey Gura 

(cherry-picked from commit#469aaba59c0539507972f4725642b2f2f81c08a0)

commit e1c3cc9d30baeaff4e653751775ab39a347b753b
Author: Pavel Kovalenko 
Date:   2018-07-24T14:48:57Z

IGNITE-8791 Fixed missed update counter in WAL data record for backup 
transaction - Fixes #4264.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 13e2a31)

commit 2ea5b376d895fe41cc882c4383b5d3d2793ba684
Author: devozerov 
Date:   2018-07-31T09:53:51Z

IGNITE-9114: SQL: fail query after some timeout if it cannot be mapped to 
topology. This closes #4453.

commit 5b230b7015148e483136c229c9e6081bb9c68023
Author: Denis Mekhanikov 
Date:   2018-04-20T15:41:06Z

IGNITE-8134 Subscribe to system cache events on nodes outside BLT

Signed-off-by: Andrey Gura 
(cherry picked from commit c82277e)

commit 2528435ec946c479f6bf4eb7eaeabf092a0536a3
Author: devozerov 
Date:   2018-07-31T14:15:49Z

IGNITE-9114: SQL: use query time in retry timeout calculation. This closes 
#4460.

commit 98d0ccc729e647a4ad8876e2d29423904e6fa6ca
Author: devozerov 
Date:   2018-07-31T14:18:49Z

Merge remote-tracking branch 'upstream/ignite-2.5.1-master' into 
ignite-2.5.1-master

commit c78a7f215699f65983edfcc0c9014d9e3108f381
Author: Anton Kalashnikov 
Date:   2018-08-01T09:23:03Z

IGNITE-8757 idle_verify utility doesn't show both update counter and hash 
conflicts

(cherry picked from commit 9131e4d)

commit 66369583096b04d71c3f82e1ebfd1922ae6b0b6a
Author: Anton Kalashnikov 
Date:   2018-07-31T13:13:27Z

IGNITE-8973 Calculate partition hash and print into standard output

Signed-off-by: Andrey Gura 

(cherry picked from commit 8309cef)

commit e92788d392d4c60f76c4fd0f38c52669f4e3e772
Author: Evgeny Stanilovskiy 
Date:   2018-08-01T11:05:58Z

IGNITE-8939 Catch and proper propagate transaction string reprsentation - 
Fixes #4454.

Signed-off-by: Dmitriy Pavlov 
(cherry picked from commit 458480c5b0520aa8e28935361a37ab49e1e65ff6)

commit 60155cb55ca35790da7f80ae167dd7fb3f3793fd
Author: Andrey V. Mashenkov 
Date:   

Re: Time to remove automated messages from the devlist?

[Dmitriy Pavlov]

Denis, we need because contributors do not announce their
intent/designs/etc manually. It is the best way ever? No, of course.

We have consensus on PR removal, so let's do it and see results.

пн, 19 нояб. 2018 г. в 18:11, Denis Mekhanikov :

> Dmitriy,
>
> If a person wants to track all new tickets, then he may go to JIRA, create
> a filter for Ignite tickets
> and subscribe to it. JIRA has a pretty flexible configuration of filters
> and subscriptions, so you can
> specify exactly what issues you are interested in, and how often you want
> to receive these emails.
> This is much more convenient and more flexible than filtering emails from a
> bot.
>
> So, most people ignore JIRA messages, and the ones who want to track new
> tickets,
> may go to JIRA and configure their own filters. I don't see, why we need to
> keep the forwarding to dev list.
>
> Denis
>
> пт, 16 нояб. 2018 г. в 22:30, Павлухин Иван :
>
> > Hi,
> >
> > Can we collect opinions about keeping messages of mentioned types on
> > dev list? From my side (+ means keeping on dev list):
> > TC bot +
> > Jira -
> > GitHub -
> > пт, 16 нояб. 2018 г. в 22:25, Dmitriy Pavlov :
> > >
> > > Importance is hardly definable and it is not possible that importance
> is
> > > equal for everyone. You can say about other human emails it is not
> > > important if some product area is not interesting for you. So I can
> only
> > > understand the terms: email needs action/does not need action.
> > >
> > > If some contributor never reacted to JIRA notification he or she may
> > think
> > > it is not important. But even we have a majority of contributors who
> > > ignores JIRA, it does not mean it is a right decision to switch it off.
> > We
> > > don't play in a democracy, hopefully.
> > >
> > > My suggestion now: keep showing an excellent example of human-human
> > > interaction, announces, etc from all Ignite veterans (especially,
> PMCs),
> > so
> > > newcomers can use the same approach.
> > >
> > > If PRs removal to notifications@ will show a positive tendency in
> > > human-human interaction, I can easily agree with the second step. Only
> > > practice is truth criteria.
> > >
> > > пт, 16 нояб. 2018 г. в 22:08, Vladimir Ozerov :
> > >
> > > > We want important emails to be easily observable. This is the only
> > goal.
> > > >
> > > > пт, 16 нояб. 2018 г. в 21:51, Dmitriy Pavlov :
> > > >
> > > > > I suggest to think in another paradigm, let's not classify emails
> to
> > be
> > > > > automatically issued or not, lets separate emails to other
> classes: a
> > > > > needed action from humans or not needed.
> > > > >
> > > > > If you don't have any interest in a change announced by JIRA issue
> > > > created
> > > > > email, you can just skip. If you can help with comments, review,
> > etc, you
> > > > > can become watcher or comment ticket, you can also point to
> > duplicate.
> > > > >
> > > > > In that paradigm,
> > > > > A) PR is perfectly ok to be redirected to notifications@ .- PR
> > creation
> > > > > does not require any action from anyone.
> > > > > B) JIRA - I'm not sure (maybe as a second step, if we will see
> > > > contributors
> > > > > will write about important tickets). And instead we can discuss
> Open
> > ->
> > > > > Patch available transition, as a reviewer needed.
> > > > > C) TC Bot - I'm sure - should never be redirected. Hopefully, it
> > will not
> > > > > generate any alerts.
> > > > >
> > > > > I hardly understand goal: is our target metric - message count to
> be
> > as
> > > > > less as possible? (extreme - 0 emails, let's not write here at all,
> > we
> > > > can
> > > > > get 0). Who can explain what do we want from redirection?
> > > > >
> > > > >
> > > > > пт, 16 нояб. 2018 г. в 16:28, Sergi Vladykin <
> > sergi.vlady...@gmail.com>:
> > > > >
> > > > > > I also would like to separate all the automated stuff.
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > пт, 16 нояб. 2018 г. в 13:58, Павлухин Иван  >:
> > > > > >
> > > > > > > Oleg,
> > > > > > >
> > > > > > > I join to Dmitriy. I found your summary quite interesting.
> > > > > > > пт, 16 нояб. 2018 г. в 13:12, Dmitriy Pavlov <
> dpav...@apache.org
> > >:
> > > > > > > >
> > > > > > > > Oleg,
> > > > > > > >
> > > > > > > > excellent research! It allows me to avoid bothering community
> > > > > > developers
> > > > > > > > once again.
> > > > > > > >
> > > > > > > > Thank you for your efforts and for contributing to this
> > discussion.
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > чт, 15 нояб. 2018 г. в 23:14, Denis Magda  >:
> > > > > > > >
> > > > > > > > > Let's move git notifications to a separate list. As for
> > JIRA, not
> > > > > > sure
> > > > > > > it
> > > > > > > > > bothers me, it took me several minutes to set up all the
> > filters
> > > > to
> > > > > > > spread
> > > > > > > > > the messages out across specific folders. Otherwise, some
> of
> > us
> > > > > might
> > > > > > > > > ignore 

Re: Time to remove automated messages from the devlist?

[Denis Mekhanikov]

Dmitriy,

If a person wants to track all new tickets, then he may go to JIRA, create
a filter for Ignite tickets
and subscribe to it. JIRA has a pretty flexible configuration of filters
and subscriptions, so you can
specify exactly what issues you are interested in, and how often you want
to receive these emails.
This is much more convenient and more flexible than filtering emails from a
bot.

So, most people ignore JIRA messages, and the ones who want to track new
tickets,
may go to JIRA and configure their own filters. I don't see, why we need to
keep the forwarding to dev list.

Denis

пт, 16 нояб. 2018 г. в 22:30, Павлухин Иван :

> Hi,
>
> Can we collect opinions about keeping messages of mentioned types on
> dev list? From my side (+ means keeping on dev list):
> TC bot +
> Jira -
> GitHub -
> пт, 16 нояб. 2018 г. в 22:25, Dmitriy Pavlov :
> >
> > Importance is hardly definable and it is not possible that importance is
> > equal for everyone. You can say about other human emails it is not
> > important if some product area is not interesting for you. So I can only
> > understand the terms: email needs action/does not need action.
> >
> > If some contributor never reacted to JIRA notification he or she may
> think
> > it is not important. But even we have a majority of contributors who
> > ignores JIRA, it does not mean it is a right decision to switch it off.
> We
> > don't play in a democracy, hopefully.
> >
> > My suggestion now: keep showing an excellent example of human-human
> > interaction, announces, etc from all Ignite veterans (especially, PMCs),
> so
> > newcomers can use the same approach.
> >
> > If PRs removal to notifications@ will show a positive tendency in
> > human-human interaction, I can easily agree with the second step. Only
> > practice is truth criteria.
> >
> > пт, 16 нояб. 2018 г. в 22:08, Vladimir Ozerov :
> >
> > > We want important emails to be easily observable. This is the only
> goal.
> > >
> > > пт, 16 нояб. 2018 г. в 21:51, Dmitriy Pavlov :
> > >
> > > > I suggest to think in another paradigm, let's not classify emails to
> be
> > > > automatically issued or not, lets separate emails to other classes: a
> > > > needed action from humans or not needed.
> > > >
> > > > If you don't have any interest in a change announced by JIRA issue
> > > created
> > > > email, you can just skip. If you can help with comments, review,
> etc, you
> > > > can become watcher or comment ticket, you can also point to
> duplicate.
> > > >
> > > > In that paradigm,
> > > > A) PR is perfectly ok to be redirected to notifications@ .- PR
> creation
> > > > does not require any action from anyone.
> > > > B) JIRA - I'm not sure (maybe as a second step, if we will see
> > > contributors
> > > > will write about important tickets). And instead we can discuss Open
> ->
> > > > Patch available transition, as a reviewer needed.
> > > > C) TC Bot - I'm sure - should never be redirected. Hopefully, it
> will not
> > > > generate any alerts.
> > > >
> > > > I hardly understand goal: is our target metric - message count to be
> as
> > > > less as possible? (extreme - 0 emails, let's not write here at all,
> we
> > > can
> > > > get 0). Who can explain what do we want from redirection?
> > > >
> > > >
> > > > пт, 16 нояб. 2018 г. в 16:28, Sergi Vladykin <
> sergi.vlady...@gmail.com>:
> > > >
> > > > > I also would like to separate all the automated stuff.
> > > > >
> > > > > Sergi
> > > > >
> > > > > пт, 16 нояб. 2018 г. в 13:58, Павлухин Иван :
> > > > >
> > > > > > Oleg,
> > > > > >
> > > > > > I join to Dmitriy. I found your summary quite interesting.
> > > > > > пт, 16 нояб. 2018 г. в 13:12, Dmitriy Pavlov  >:
> > > > > > >
> > > > > > > Oleg,
> > > > > > >
> > > > > > > excellent research! It allows me to avoid bothering community
> > > > > developers
> > > > > > > once again.
> > > > > > >
> > > > > > > Thank you for your efforts and for contributing to this
> discussion.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > чт, 15 нояб. 2018 г. в 23:14, Denis Magda :
> > > > > > >
> > > > > > > > Let's move git notifications to a separate list. As for
> JIRA, not
> > > > > sure
> > > > > > it
> > > > > > > > bothers me, it took me several minutes to set up all the
> filters
> > > to
> > > > > > spread
> > > > > > > > the messages out across specific folders. Otherwise, some of
> us
> > > > might
> > > > > > > > ignore subscribing to jira-list and would miss notifications
> when
> > > > > their
> > > > > > > > input is needed.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > On Thu, Nov 15, 2018 at 12:03 PM Vladimir Ozerov <
> > > > > voze...@gridgain.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dmitry,
> > > > > > > > >
> > > > > > > > > I am not referring to some "authoritative ASF member" as a
> > > guide
> > > > > for
> > > > > > us.
> > > > > > > > We
> > > > > > > > > are on our 

[GitHub] ignite pull request #5431: Ignite 10328

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/5431

Ignite 10328



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10328

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5431.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5431


commit 2208d097da681191452555d7f23bdfec0d17c57a
Author: EdShangGG 
Date:   2018-11-12T19:04:26Z

GG-14437 Can't restore sql cache in snasphot

commit 96e2fddc9b7c18ec11ab181ad45633f4041a9aa3
Author: ibessonov 
Date:   2018-11-19T09:41:05Z

IGNITE-10192 Fixed OptimizedMarshallerTest#testAllocationOverflow throws 
OOME instead of expected IOE - Fixes #5400.

Signed-off-by: Dmitriy Govorukhin 

commit 166e87ed19929b9e064732752d5a5ce1069b5076
Author: Alexander Kalinin 
Date:   2018-11-19T09:47:12Z

IGNITE-10320 Web Console: Workaround for memory leak in chart component.

commit c9906aab8efa2c70a164fb593d9e22ef1edaa9ec
Author: ibessonov 
Date:   2018-11-19T09:50:56Z

IGNITE-10142 Scale factor added to several tests from Cache 8 suite. - 
Fixes #5253.

Signed-off-by: Dmitriy Govorukhin 

commit 9e3bd7dd686d3392e6bbe2c7defebe260eb029f7
Author: ibessonov 
Date:   2018-11-19T10:06:47Z

IGNITE-10197 Fixed unexpected IllegalArgumentException in 
IgniteDbPutGetAbstractTest#testRandomPutGetRemove - Fixes #5410.

Signed-off-by: Dmitriy Govorukhin 

commit a63a81a51418a8ae1dadff019d9e5eace38e1631
Author: Ilya Lantukh 
Date:   2018-11-19T12:07:49Z

IGNITE-9558 Avoid blocking transactions on client connect when possible - 
Fixes #4933.

commit 055b28daebce85a38175937fe91ab80bd99ca566
Author: EdShangGG 
Date:   2018-11-19T14:37:32Z

Merge branch 'master1' into ignite-gg-14437

commit 7a844c489acd3732940eb24420a7c566ea85e180
Author: EdShangGG 
Date:   2018-11-19T15:01:23Z

IGNITE-10328 Allow to destroy cache in code which were created via SQL




---

Re: Update our C++ library to C++11 standard

[Igor Sapego]

I've filed a ticket for this task [1].

[1] - https://issues.apache.org/jira/browse/IGNITE-10333

Best Regards,
Igor


On Mon, Nov 12, 2018 at 1:39 PM Igor Sapego  wrote:

> Vladimir,
>
> I'd wish we could do that, but unfortunately, even VS 2017 does not
> support all C++11 (and even C++03) features [1]. So, maybe we should
> also speak about dropping support of old VS and moving to at least VS 2012.
>
> You can find the benefits of this at [2]. For example, this will give us
> native support of atomics, fences, threads and also better language
> features,
> that can help a lot when developing API and help finding bugs during
> compilation.
>
> If we will adopt the "limited C++11" approach, then I believe we will need
> to
> update our API to make it more modern and convenient for C++11 users.
>
> But, unfortunately, this won't give us the complete support of C++11.
>
> Something else we should also consider here, is that C++ community can be
> very conservative, so some part of community could still be on C++03.
>
> [1] -
> https://docs.microsoft.com/en-us/cpp/visual-cpp-language-conformance?view=vs-2017
> [2] - https://msdn.microsoft.com/en-gb/library/hh567368.aspx
>
> Best Regards,
> Igor
>
> On Mon, Nov 12, 2018 at 10:28 AM Vladimir Ozerov 
> wrote:
>
>> Igniters,
>>
>> Over time we were very conservative about language levels for our
>> libraries
>> to be able to target wider platforms. But it looks like some of them are
>> way too old, what doing more harm than good.
>>
>> For C++ we still use C++03 standard, which 15 years old. C++11, C++15 and
>> C++17 were released since then. I propose to plan upgrade to C++11 version
>> at least. May be event C++15. Major improvements in C++11:
>>
>> 1) Standard threading model - we will be able to remove a lot
>> platform-dependent code (atomics, shared pointers, etc)
>> 2) Rvalues - most probably we will be able to benefit from move semantics
>> in terms of both performance and cleaner API
>> 3) Lambdas - this needs to be investigated, but may be we will be able to
>> integrate them into our compute API.
>>
>> If agreed we should plan it to AI 3.0 release, since this is a breaking
>> change.
>>
>> What do you think?
>>
>> Vladimir.
>>
>

[jira] [Created] (IGNITE-10333) CPP: Move to newer C++ standard and drop VS 2010 support

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-10333:


 Summary: CPP: Move to newer C++ standard and drop VS 2010 support
 Key: IGNITE-10333
 URL: https://issues.apache.org/jira/browse/IGNITE-10333
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Igor Sapego
 Fix For: 3.0


Over time we were very conservative about language levels for our libraries to 
be able to target wider platforms. But it looks like some of them are way too 
old, what doing more harm than good.

For C++ we still use C++03 standard, which 15 years old. C++11, C++14 and C++17 
were released since then. I propose to plan upgrade to C++11 version at least. 
May be even C++14. Major improvements in C++11:

1) Standard threading model - we will be able to remove a lot 
platform-dependent code (atomics, shared pointers, etc)
2) Rvalues - most probably we will be able to benefit from move semantics in 
terms of both performance and cleaner API
3) Lambdas - this needs to be investigated, but may be we will be able to 
integrate them into our compute API.

Also, we should take into account that even VS 2017 does not support all C++11 
(and even C++03) features [1]. So, we should also drop support of VS 2010 and 
move to at least VS 2012. Features list for different VSs [2].

[1] - 
https://docs.microsoft.com/en-us/cpp/visual-cpp-language-conformance?view=vs-2017
[2] - 
https://msdn.microsoft.com/en-gb/library/hh567368.aspx?f=255=-2147217396




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10332) Add Ignite.NET configuration for disk page compression

[Sergi Vladykin (JIRA)]

Sergi Vladykin created IGNITE-10332:
---

 Summary: Add Ignite.NET configuration for disk page compression 
 Key: IGNITE-10332
 URL: https://issues.apache.org/jira/browse/IGNITE-10332
 Project: Ignite
  Issue Type: New Feature
  Components: cache
Reporter: Sergi Vladykin
Assignee: Vladimir Ozerov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10331) Document Disk page compression

[Sergi Vladykin (JIRA)]

Sergi Vladykin created IGNITE-10331:
---

 Summary: Document Disk page compression
 Key: IGNITE-10331
 URL: https://issues.apache.org/jira/browse/IGNITE-10331
 Project: Ignite
  Issue Type: New Feature
  Components: documentation
Reporter: Sergi Vladykin
Assignee: Sergey Kozlov


There is an email thread titled "Disk page compression for Ignite persistent 
store"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10330) Implement disk page compression

[Sergi Vladykin (JIRA)]

Sergi Vladykin created IGNITE-10330:
---

 Summary: Implement disk page compression
 Key: IGNITE-10330
 URL: https://issues.apache.org/jira/browse/IGNITE-10330
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Sergi Vladykin
Assignee: Sergi Vladykin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10329) Create JDBC "query" and "query join" benchmarks and compare them with Postgres and MySQL

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-10329:


 Summary: Create JDBC "query" and "query join" benchmarks and 
compare them with Postgres and MySQL
 Key: IGNITE-10329
 URL: https://issues.apache.org/jira/browse/IGNITE-10329
 Project: Ignite
  Issue Type: Task
  Components: sql, yardstick
Reporter: Vladimir Ozerov
Assignee: Pavel Kuznetsov
 Fix For: 2.8


Currently we have {{IgniteSqlQueryBenchmark}} and 
{{IgniteSqlQueryJoinBenchmark}} benchmarks which query data over salary range 
and optionally joins it with second table. Let's create a set of similar 
benchmarks which will use JDBC to load and query data, and execute them against 
one-node Ignite cluster, MySQL and Postgres.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10328) Allow to destroy cache in code which were created via SQL

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-10328:
--

 Summary: Allow to destroy cache in code which were created via SQL
 Key: IGNITE-10328
 URL: https://issues.apache.org/jira/browse/IGNITE-10328
 Project: Ignite
  Issue Type: Improvement
Reporter: Eduard Shangareev
Assignee: Eduard Shangareev


We couldn't destroy cache without using SQL if it was created via SQL.

{code}
Caused by: class org.apache.ignite.IgniteCheckedException: Only cache created 
with cache API may be removed with direct call to destroyCache 
[cacheName=SQL_PUBLIC_T1]
at 
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.processCacheChangeRequests(ClusterCachesInfo.java:677)
at 
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.onCacheChangeRequested(ClusterCachesInfo.java:430)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onCustomEvent(GridCacheProcessor.java:3827)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:697)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
... 1 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10327) Claster take NPE due cache stop and stop by handler

[ARomantsov (JIRA)]

ARomantsov created IGNITE-10327:
---

 Summary: Claster take NPE due cache stop and stop by handler
 Key: IGNITE-10327
 URL: https://issues.apache.org/jira/browse/IGNITE-10327
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 2.8
Reporter: ARomantsov
 Fix For: 2.7


Stop all caches
Take NPE : null on preloadEntry(GridDhtPartitionDemander.java:893)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5430: IGNITE-9872 [Test falied] IgniteSinkConnectorTest...

[Max-Pudov]

GitHub user Max-Pudov opened a pull request:

https://github.com/apache/ignite/pull/5430

IGNITE-9872 [Test falied] IgniteSinkConnectorTest.testSinkPutsWithout…

…Transformation (updating Kafka to 2.0.1)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9872

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5430


commit af0996f03ed67bd22f4144882449f3d62082d844
Author: Max-Pudov 
Date:   2018-11-19T14:07:47Z

IGNITE-9872 [Test falied] 
IgniteSinkConnectorTest.testSinkPutsWithoutTransformation (updating Kafka to 
2.0.1)




---

Re: Disk page compression for Ignite persistent store

[Sergi Vladykin]

Ilya,

Zstd itself has default compression level 3. I just used that number to be
consistent with the library defaults.

I will check if there is a significant difference in performance.

Sergi

пн, 19 нояб. 2018 г. в 14:59, Ilya Kasnacheev :

> Hello!
>
> You have zstd default level of 3. In my tests, zstd usually performed much
> better with compression level 2. Please consider.
>
> I admire your effort!
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin :
>
> > Right now the functionality has nothing to do with WAL, but your idea
> > definitely makes sense and worth being implemented as a next step.
> >
> > Sergi
> >
> > пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov <
> andrey.mashen...@gmail.com
> > >:
> >
> > > Hi Sergi,
> > >
> > > It is not clear for me will your changes affect PageSnapshot WAL
> record.
> > > Is it possible to add compression support for PageSnapshot WAL record
> as
> > > well, to reduce WAL size?
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <
> sergi.vlady...@gmail.com
> > >
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > I've implemented page compression for persistent store and going to
> > merge
> > > > it to master.
> > > >
> > > > https://github.com/apache/ignite/pull/5200
> > > >
> > > > Some design notes:
> > > >
> > > > It employs "hole punching" approach, it means that the pages are kept
> > > > uncompressed in memory,
> > > > but when they get written to disk, they will be compressed and all
> the
> > > > extra file system blocks for the page will be released. Thus the
> > storage
> > > > files become sparse.
> > > >
> > > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY
> and
> > > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE,
> > which
> > > > basically just takes only meaningful data from half-filled pages but
> > does
> > > > not apply any compression. It is easy to add more if needed.
> > > >
> > > > Since we can release only full file system blocks which are typically
> > 4k
> > > > size, user must configure page size to be at least multiple FS
> blocks,
> > > e.g.
> > > > 8k or 16k. It also means that max compression ratio here is
> > fsBlockSize /
> > > > pageSize = 4k / 16k = 0.25
> > > >
> > > > It is possible to enable compression for existing databases if they
> > were
> > > > configured for large enough page size. In this case pages will be
> > written
> > > > to disk in compressed form when updated, and the database will become
> > > > compressed gradually.
> > > >
> > > > There will be 2 new properties on CacheConfiguration
> > > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup
> disk
> > > page
> > > > compression.
> > > >
> > > > Compression dictionaries are not supported at the time, but may in
> the
> > > > future. IMO it should be added as a separate feature if needed.
> > > >
> > > > The only supported platform for now is Linux. Since all popular file
> > > > systems support sparse files, it must be  relatively easy to support
> > more
> > > > platforms.
> > > >
> > > > Please take a look and provide your thoughts and suggestions.
> > > >
> > > > Thanks!
> > > >
> > > > Sergi
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > >
> >
>

[jira] [Created] (IGNITE-10326) Add trusted suites

[PetrovMikhail (JIRA)]

PetrovMikhail created IGNITE-10326:
--

 Summary: Add trusted suites
 Key: IGNITE-10326
 URL: https://issues.apache.org/jira/browse/IGNITE-10326
 Project: Ignite
  Issue Type: Task
Reporter: PetrovMikhail
Assignee: PetrovMikhail


In trusted suites every new failure will be notified as critical.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4837: IGNITE-8717

[antonovsergey93]

Github user antonovsergey93 closed the pull request at:

https://github.com/apache/ignite/pull/4837


---

[GitHub] ignite pull request #5429: ignite-8432: Test for column name case in PK and ...

[pavel-kuznetsov]

GitHub user pavel-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/5429

ignite-8432: Test for column name case in PK and CREATE TABLE.

Although, bug have been fixed during IGNITE-8052, added test for this
special case: primary keys list is case sensitive; correct exception
should be thrown if we forgot to quote lower case names in the one place
(columns list/PK list) but quoted in the other.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8432

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5429.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5429


commit 368303a16553982c3f900dd6f0ca322dd005be68
Author: Pavel Kuznetsov 
Date:   2018-11-19T13:42:44Z

ignite-8432: Test for column name case in PK and CREATE TABLE.

Although, bug have been fixed during IGNITE-8052, added test for this
special case: primary keys list is case sensitive; correct exception
should be thrown if we forgot to quote lower case names in the one place
(columns list/PK list) but quoted in the other.




---

[GitHub] ignite pull request #5428: IGNITE-10298 Fixed deadlock during checkpointer a...

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/5428

IGNITE-10298 Fixed deadlock during checkpointer and caches start for 2.4



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10298-2.4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5428.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5428


commit d99a50ccd6923aa48da78955207fe747b287ea81
Author: mcherkasov 
Date:   2018-04-11T00:23:29Z

IGNITE-8153 Nodes fail to connect each other when SSL is enabled - Fixes 
#3773.

Signed-off-by: Valentin Kulichenko 

commit 03285e953d005a18fb88a35d21701100f58f7a29
Author: devozerov 
Date:   2018-04-12T07:37:36Z

IGNITE-8042: .NET thin client: authentication support. This closes #3790.

commit 8e740164e8a0102a94f408d72b41f73df675cfb1
Author: Ilya Kasnacheev 
Date:   2018-04-05T09:55:36Z

IGNITE-7712: SQL: system property to force lazy query execution.

Cherry-picked from 064c816c177d31f18af2954175ca3ad0f3eee957

commit df405d439d461d3576cd3b22d53716cd324b3ef7
Author: Alexander Paschenko 
Date:   2018-04-11T14:03:16Z

IGNITE-8204: SQL: fixed hangs when lazy flag is enabled.

Cherry-picked from 747e6c5.

commit 8af36584a81f6411a08b66935b843a5c1ba3169c
Author: devozerov 
Date:   2018-04-12T12:02:57Z

IGNITE-8135: SQL: authentication for CREATE TABLE and DROP TABLE commands. 
This closes #3801.

commit 1c31e07bb7510b908b540169a6ded7a909e23fda
Author: devozerov 
Date:   2018-04-12T12:13:51Z

IGNITE-8230: SQL: Fixed backup number propagation in CREATE TABLE command. 
This closes #3803.

commit 131b9b974c01d1b5c05f98e2fe3947f0672c94ec
Author: tledkov-gridgain 
Date:   2018-04-16T08:28:39Z

IGNITE-8129: MTCGA: setup default SSL context in JdbcthinConnectionSSLTest 
(because sometimes default SSL context may be setup by build system). This 
closes #3795.

commit 2bf4c8a76dc1a02a82b167d69109ed591a882f0c
Author: Alexey Kukushkin 
Date:   2018-04-16T08:47:19Z

IGNITE-8097: Java thin client: throw handshake exception eagerly on connect 
phase in case of failure. This closes #3822.

commit b208bb255916382ff06464170a56891d2bd5107e
Author: Ilya Kasnacheev 
Date:   2018-04-16T15:55:03Z

IGNITE-2766 - Opportunistically reopen cache after client reconnect.

Cherry-picked from 0991437a3f4d38e68483a8bcadd3daf614b7b2dc

commit 47b1f4007348e8c0697638b7824227eaa36dca32
Author: Alexey Kuznetsov 
Date:   2018-04-17T04:46:45Z

IGNITE-8201 REST: Added AUTHENTICATE command. Fixed session tokens. Added 
new tests.

(cherry picked from commit 1cfc989)

commit d1d813eb783f9adae8bf10c5d6a0c53613eadf2b
Author: Anton Kalashnikov 
Date:   2018-04-17T09:03:08Z

GG-13715 Store dynamic indexes to cache data on node join - Fixes #3719.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit bbc439b)

commit 7cfc7377e1a0d08d56ac4b36995b8bf0d91609b9
Author: Stanislav Lukyanov 
Date:   2018-04-17T14:15:08Z

IGNITE-8210 Fixed custom event handling for baseline topology change - 
Fixes #3814.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit d79c640)

commit 83b9ffd1307c17435c1aae81f42480d90404f8a2
Author: ezhuravl 
Date:   2018-04-17T15:41:56Z

IGNITE-6113 Changed mistake in version for partition demand version

commit afc5fc1789d75573f71b40c5e241484c7a578197
Author: ezhuravl 
Date:   2018-04-17T15:41:56Z

IGNITE-6113 Changed mistake in version for partition demand version

(cherry picked from commit 83b9ffd)

commit 25f6a2013aa584559623410b7a96951f79fb00ff
Author: Ivan Daschinskiy 
Date:   2018-04-17T16:55:42Z

IGNITE-8021 Delete cache config files when cache is destroyed - Fixes #3697.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit 2edcb22fbb566981097733af6470ed6dde8e786b)

commit 5461dd64ee15f02be7934b33d0ca92130aa70512
Author: Ilya Kasnacheev 
Date:   2018-04-17T17:04:28Z

IGNITE-2766 Fix .net test.

commit 9f5b27fae9ac57ae5b256cb8593dfe587b4accb8
Author: oleg-ostanin 
Date:   2018-04-17T17:58:53Z

IGNITE-8274 sqlline.sh script uses JAVA_HOME now

Signed-off-by: Andrey Gura 

(cherry picked from commit c3ff274)

commit 640167f2c9384fddd69e6244b615e4974bfe2b50
Author: Maxim Muzafarov 
Date:   2018-04-18T09:20:13Z

IGNITE-8301 testReconnectCacheDestroyedAndCreated should excpect recreated 
client cache.

Cherry-picked from 56be24b9dfc14023bacaab63f40e0504b317eda3

commit 89b8426a2a113b6893a2295044d6dc0e94015a94
Author: Alexey Kuznetsov 
Date:   2018-04-18T11:49:12Z

ignite-2.4.4 Fixed default node version.

commit 048c21a3cc7d00a1c5951137f3747904e00405ea
Author: Alexey Kuznetsov 
Date:   2018-04-19T07:14:51Z

IGNITE-8298 Web Console: 

Re: [DISCUSSION] Spark Data Frame through Thin Client

[Nikolay Izhikov]

IGNITE-10325 created.

ср, 7 нояб. 2018 г., 11:42 Ray ray...@cisco.com:

> From my past experience with Spark Data Frame API, the thick client
> approach
> leads to many usability problems.
>
> Ex.
>
> http://apache-ignite-users.70518.x6.nabble.com/Local-node-SEGMENTED-error-causing-node-goes-down-for-no-obvious-reason-td25061.html
>
> I think it makes a lot of sense to change to thin client.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[jira] [Created] (IGNITE-10325) Spark Data Frame - Thin Client

[Nikolay Izhikov (JIRA)]

Nikolay Izhikov created IGNITE-10325:


 Summary: Spark Data Frame - Thin Client
 Key: IGNITE-10325
 URL: https://issues.apache.org/jira/browse/IGNITE-10325
 Project: Ignite
  Issue Type: Improvement
  Components: spark
Affects Versions: 2.6
Reporter: Nikolay Izhikov


For now, client node required to connect to Ignite cluster from Spark.

We need to add ability to use Thin Client protocol for Spark integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10324) Disallow fallback to Scanner in control.sh when asking password

[Pavel Voronkin (JIRA)]

Pavel Voronkin created IGNITE-10324:
---

 Summary: Disallow fallback to Scanner in control.sh when asking 
password
 Key: IGNITE-10324
 URL: https://issues.apache.org/jira/browse/IGNITE-10324
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Voronkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ololo3000 opened a new pull request #75: IGNITE-10319 Suite compilation error failure handling added

[GitBox]

ololo3000 opened a new pull request #75: IGNITE-10319 Suite compilation error 
failure handling added
URL: https://github.com/apache/ignite-teamcity-bot/pull/75
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of 
visa caching. Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74#discussion_r234598100
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
 ##
 @@ -63,33 +70,78 @@
 @Inject private Ignite ignite;
 
 /** */
-@Inject private VisasHistoryStorage visasHistoryStorage;
+@Inject private VisasHistoryStorage visasHistStorage;
 
 /** */
 @Inject private IStringCompactor strCompactor;
 
+/** */
+private ReentrantLock observationLock = new ReentrantLock();
+
 /**
  */
 ObserverTask() {
 }
 
 /** */
-private IgniteCache compactInfos() {
+private IgniteCache 
compactInfos() {
 return 
ignite.getOrCreateCache(TcHelperDb.getCacheV2TxConfig(BUILDS_CACHE_NAME));
 }
 
+/** */
+@Nullable public BuildsInfo getInfo(ContributionKey key) {
+CompactBuildsInfo compactBuildsInfo = compactInfos().get(new 
CompactContributionKey(key, strCompactor));
+
+return Objects.isNull(compactBuildsInfo) ? null : 
compactBuildsInfo.toBuildInfo(strCompactor);
+}
+
+
 /** */
 public Collection getInfos() {
 List buildsInfos = new ArrayList<>();
 
-compactInfos().forEach(entry -> 
buildsInfos.add(entry.getKey().toBuildInfo(strCompactor)));
+compactInfos().forEach(entry -> 
buildsInfos.add(entry.getValue().toBuildInfo(strCompactor)));
 
 return buildsInfos;
 }
 
 /** */
 public void addInfo(BuildsInfo info) {
-compactInfos().put(new CompactBuildsInfo(info, strCompactor), new 
Object());
+visasHistStorage.put(new VisaRequest(info));
+
+compactInfos().put(new 
CompactContributionKey(info.getContributionKey(), strCompactor),
+new CompactBuildsInfo(info, strCompactor));
+}
+
+/** */
+private void removeBuildInfo(CompactContributionKey key) {
+try {
+BuildsInfo buildsInfo = 
compactInfos().get(key).toBuildInfo(strCompactor);
+
+boolean rmv = compactInfos().remove(key);
+
+Preconditions.checkState(rmv, "Key not found: " + 
key.toContributionKey(strCompactor).toString());
+}
+catch (Exception e) {
+logger.error("Cache remove: " + e.getMessage(), e);
+
+throw new RuntimeException("Observer queue: " +
+getInfos().stream().map(bi -> 
bi.getContributionKey().toString())
+.collect(Collectors.joining(", ")) +
+" Error: " + X.getFullStackTrace(e));
+}
+}
+
+/** */
+public void removeBuildInfo(ContributionKey key) {
 
 Review comment:
   Move public method higher than private.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of 
visa caching. Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74#discussion_r234602866
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/ContributionKey.java
 ##
 @@ -17,23 +17,32 @@
 
 package org.apache.ignite.ci.web.model;
 
+import org.apache.ignite.ci.teamcity.ignited.IStringCompactor;
+
 /**
  *
  */
 public class ContributionKey {
 /** */
 public final String srvId;
 
-/** */
-public final String ticket;
-
 /** */
 public final String branchForTc;
 
 /** */
-public ContributionKey(String srvId, String ticket, String branchForTc) {
+public ContributionKey(String srvId, String branchForTc) {
 this.branchForTc = branchForTc;
 this.srvId = srvId;
-this.ticket = ticket;
+}
+
+/** */
+public ContributionKey(CompactContributionKey key, IStringCompactor 
strCompactor) {
+this.branchForTc = strCompactor.getStringFromId(key.branchForTc);
+this.srvId = strCompactor.getStringFromId(key.srvId);
+}
+
+/** */
 
 Review comment:
   ```suggestion
   /** {@inheritDoc} */
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of 
visa caching. Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74#discussion_r234601602
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/visa/TcBotTriggerAndSignOffService.java
 ##
 @@ -56,10 +59,20 @@
 import org.jetbrains.annotations.NotNull;
 import org.jetbrains.annotations.Nullable;
 
+import static org.apache.ignite.ci.observer.BuildsInfo.CANCELLED_STATUS;
+import static org.apache.ignite.ci.observer.BuildsInfo.FINISHED_STATUS;
+import static org.apache.ignite.ci.observer.BuildsInfo.RUNNING_STATUS;
+
 /**
  * Provides method for TC Bot Visa obtaining
  */
 public class TcBotTriggerAndSignOffService {
+/** */
+private static final ThreadLocal THREAD_FORMATTER = new 
ThreadLocal() {
 
 Review comment:
   Why thread local? Looks like we need a simple object.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of 
visa caching. Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74#discussion_r234597233
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
 ##
 @@ -63,33 +70,78 @@
 @Inject private Ignite ignite;
 
 /** */
-@Inject private VisasHistoryStorage visasHistoryStorage;
+@Inject private VisasHistoryStorage visasHistStorage;
 
 /** */
 @Inject private IStringCompactor strCompactor;
 
+/** */
+private ReentrantLock observationLock = new ReentrantLock();
+
 /**
  */
 ObserverTask() {
 }
 
 /** */
-private IgniteCache compactInfos() {
+private IgniteCache 
compactInfos() {
 return 
ignite.getOrCreateCache(TcHelperDb.getCacheV2TxConfig(BUILDS_CACHE_NAME));
 }
 
+/** */
+@Nullable public BuildsInfo getInfo(ContributionKey key) {
+CompactBuildsInfo compactBuildsInfo = compactInfos().get(new 
CompactContributionKey(key, strCompactor));
+
+return Objects.isNull(compactBuildsInfo) ? null : 
compactBuildsInfo.toBuildInfo(strCompactor);
+}
+
+
 /** */
 public Collection getInfos() {
 List buildsInfos = new ArrayList<>();
 
-compactInfos().forEach(entry -> 
buildsInfos.add(entry.getKey().toBuildInfo(strCompactor)));
+compactInfos().forEach(entry -> 
buildsInfos.add(entry.getValue().toBuildInfo(strCompactor)));
 
 return buildsInfos;
 }
 
 /** */
 public void addInfo(BuildsInfo info) {
-compactInfos().put(new CompactBuildsInfo(info, strCompactor), new 
Object());
+visasHistStorage.put(new VisaRequest(info));
+
+compactInfos().put(new 
CompactContributionKey(info.getContributionKey(), strCompactor),
+new CompactBuildsInfo(info, strCompactor));
+}
+
+/** */
+private void removeBuildInfo(CompactContributionKey key) {
+try {
+BuildsInfo buildsInfo = 
compactInfos().get(key).toBuildInfo(strCompactor);
 
 Review comment:
   Not used variable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

SomeFire commented on a change in pull request #74: IGNITE-10275 Refactor of 
visa caching. Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74#discussion_r234600605
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/BuildObserver.java
 ##
 @@ -69,28 +93,27 @@ public void stop() {
 public void observe(String srvId, ICredentialsProv prov, String ticket, 
String branchForTc, Build... builds) {
 BuildsInfo buildsInfo = new BuildsInfo(srvId, prov, ticket, 
branchForTc, builds);
 
-visasStorage.put(new VisaRequest(buildsInfo));
-
 observerTask.addInfo(buildsInfo);
 }
 
 /**
  * @param srvId Server id.
  * @param branch Branch.
  */
-public String getObservationStatus(String srvId, String branch) {
+public String getObservationStatus(ContributionKey key) {
 
 Review comment:
   Update javadocs too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-10323) Contol utility --deactivate on non-activate cluster produse NPE and handler stop nodes

[ARomantsov (JIRA)]

ARomantsov created IGNITE-10323:
---

 Summary: Contol utility --deactivate on non-activate cluster 
produse NPE and handler stop nodes
 Key: IGNITE-10323
 URL: https://issues.apache.org/jira/browse/IGNITE-10323
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.7
Reporter: ARomantsov
 Fix For: 2.8


Scenario:
1) Start cluster
2) Call control.sh --deactivate




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10322) MVCC TX: Clean out messages.

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-10322:
-

 Summary: MVCC TX: Clean out messages.
 Key: IGNITE-10322
 URL: https://issues.apache.org/jira/browse/IGNITE-10322
 Project: Ignite
  Issue Type: Improvement
  Components: mvcc
Reporter: Igor Seliverstov
 Fix For: 2.8


Currently MvccSnapshot is a part of many message types and some of usages look 
unnecessary. Since MvccSnapshot potentially can be really long (in bytes) we 
need to review and possibly optimize all places where MvccSnapshot is a message 
part.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5427: IGNITE-10321 Bug in CacheContinuousWithTransforme...

[ibessonov]

GitHub user ibessonov opened a pull request:

https://github.com/apache/ignite/pull/5427

IGNITE-10321 Bug in 
CacheContinuousWithTransformerReplicatedSelfTest.LocalEventListener causes 
certain tests to flak.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10321

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5427


commit 96e2fddc9b7c18ec11ab181ad45633f4041a9aa3
Author: ibessonov 
Date:   2018-11-19T09:41:05Z

IGNITE-10192 Fixed OptimizedMarshallerTest#testAllocationOverflow throws 
OOME instead of expected IOE - Fixes #5400.

Signed-off-by: Dmitriy Govorukhin 

commit 166e87ed19929b9e064732752d5a5ce1069b5076
Author: Alexander Kalinin 
Date:   2018-11-19T09:47:12Z

IGNITE-10320 Web Console: Workaround for memory leak in chart component.

commit c9906aab8efa2c70a164fb593d9e22ef1edaa9ec
Author: ibessonov 
Date:   2018-11-19T09:50:56Z

IGNITE-10142 Scale factor added to several tests from Cache 8 suite. - 
Fixes #5253.

Signed-off-by: Dmitriy Govorukhin 

commit 9e3bd7dd686d3392e6bbe2c7defebe260eb029f7
Author: ibessonov 
Date:   2018-11-19T10:06:47Z

IGNITE-10197 Fixed unexpected IllegalArgumentException in 
IgniteDbPutGetAbstractTest#testRandomPutGetRemove - Fixes #5410.

Signed-off-by: Dmitriy Govorukhin 

commit 0c59a83b570113b2ae71d67e365cdd4ddca1bf1f
Author: ibessonov 
Date:   2018-11-19T12:17:02Z

IGNITE-10321 Correct operations order that fixes some flaky tests.




---

[jira] [Created] (IGNITE-10321) Bug CacheContinuousWithTransformerReplicatedSelfTest.LocalEventListener causes certain tests to flack.

[Ivan Bessonov (JIRA)]

Ivan Bessonov created IGNITE-10321:
--

 Summary: Bug 
CacheContinuousWithTransformerReplicatedSelfTest.LocalEventListener causes 
certain tests to flack.
 Key: IGNITE-10321
 URL: https://issues.apache.org/jira/browse/IGNITE-10321
 Project: Ignite
  Issue Type: Improvement
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov
 Fix For: 2.8


Known problems: 

https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5285455933531531639=testDetails

https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=579266511269744969=testDetails



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5426: IGNITE-9996: Final fix

[nizhikov]

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite/pull/5426

IGNITE-9996: Final fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite IGNITE-9996-2.7-final-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5426


commit b1708c37e55f213bbab42e7e8322d8d598ed2dd7
Author: Nikolay Izhikov 
Date:   2018-11-19T12:05:38Z

IGNITE-9996: Final fix




---

Re: Disk page compression for Ignite persistent store

[Ilya Kasnacheev]

Hello!

You have zstd default level of 3. In my tests, zstd usually performed much
better with compression level 2. Please consider.

I admire your effort!

Regards,
-- 
Ilya Kasnacheev


пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin :

> Right now the functionality has nothing to do with WAL, but your idea
> definitely makes sense and worth being implemented as a next step.
>
> Sergi
>
> пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov  >:
>
> > Hi Sergi,
> >
> > It is not clear for me will your changes affect PageSnapshot WAL record.
> > Is it possible to add compression support for PageSnapshot WAL record as
> > well, to reduce WAL size?
> >
> > Thanks.
> >
> > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin  >
> > wrote:
> >
> > > Folks,
> > >
> > > I've implemented page compression for persistent store and going to
> merge
> > > it to master.
> > >
> > > https://github.com/apache/ignite/pull/5200
> > >
> > > Some design notes:
> > >
> > > It employs "hole punching" approach, it means that the pages are kept
> > > uncompressed in memory,
> > > but when they get written to disk, they will be compressed and all the
> > > extra file system blocks for the page will be released. Thus the
> storage
> > > files become sparse.
> > >
> > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE,
> which
> > > basically just takes only meaningful data from half-filled pages but
> does
> > > not apply any compression. It is easy to add more if needed.
> > >
> > > Since we can release only full file system blocks which are typically
> 4k
> > > size, user must configure page size to be at least multiple FS blocks,
> > e.g.
> > > 8k or 16k. It also means that max compression ratio here is
> fsBlockSize /
> > > pageSize = 4k / 16k = 0.25
> > >
> > > It is possible to enable compression for existing databases if they
> were
> > > configured for large enough page size. In this case pages will be
> written
> > > to disk in compressed form when updated, and the database will become
> > > compressed gradually.
> > >
> > > There will be 2 new properties on CacheConfiguration
> > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk
> > page
> > > compression.
> > >
> > > Compression dictionaries are not supported at the time, but may in the
> > > future. IMO it should be added as a separate feature if needed.
> > >
> > > The only supported platform for now is Linux. Since all popular file
> > > systems support sparse files, it must be  relatively easy to support
> more
> > > platforms.
> > >
> > > Please take a look and provide your thoughts and suggestions.
> > >
> > > Thanks!
> > >
> > > Sergi
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>

Re: Brainstorm: Make TC Run All faster

[Павлухин Иван]

Hi,

I would like to understand following. We are going to make TC green.
We are going to make TC fast. Are we going to do it in parallel?
пн, 19 нояб. 2018 г. в 08:39, Petr Ivanov :
>
> Concerning current TeamCity compute capacity — I think we should invest into 
> it’s stability at first: there are lots of problems associated with
>  — tests hangs (which now can render agent useless up to 3 days)
>  — checkout issues (speedup proxy is installed but sometimes even it is not 
> enough for checkout on 100 agents simultaneously, server side checkout 
> research is required)
>  — tests architecture (current one relies heavily on large file movement over 
> network which also can be a bottleneck in case of 100 agents simultaneous 
> download start)
> And so on.
>
> After it additional donation can be discussed.
>
> > On 16 Nov 2018, at 17:05, Vyacheslav Daradur  wrote:
> >
> > At one of the meetups, Vladimir Ozerov has said that we may specify
> > and move less risky to be broken tests in separate build plan which
> > will be executed daily or weekly
> >
> > For example tests of compatibility with different JDK versions or
> > compatibility between Ignite's releases.
> >
> > Also, I agree with Denis we should find and remove tests with the same 
> > checks.
> >
> > By the way, if someone of the community donates TC agents, can this
> > help to reduce the time?
> >
> > On Thu, Nov 15, 2018 at 5:38 PM Yakov Zhdanov  wrote:
> >>
> >> Denis, you can go even further. E.g. you can start topology once for the
> >> full set of single threaded full api cache tests. Each test should start
> >> cache dynamically and run it logic.
> >>
> >> As for me, I would think of splitting RunAll to 2 steps - one containing
> >> basic tests and another with more complex tests. 2nd step should not start
> >> (except manually) if 1st step results in any build failure.
> >>
> >> --Yakov
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>


-- 
Best regards,
Ivan Pavlukhin

Re: Disk page compression for Ignite persistent store

[Sergi Vladykin]

Right now the functionality has nothing to do with WAL, but your idea
definitely makes sense and worth being implemented as a next step.

Sergi

пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov :

> Hi Sergi,
>
> It is not clear for me will your changes affect PageSnapshot WAL record.
> Is it possible to add compression support for PageSnapshot WAL record as
> well, to reduce WAL size?
>
> Thanks.
>
> On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin 
> wrote:
>
> > Folks,
> >
> > I've implemented page compression for persistent store and going to merge
> > it to master.
> >
> > https://github.com/apache/ignite/pull/5200
> >
> > Some design notes:
> >
> > It employs "hole punching" approach, it means that the pages are kept
> > uncompressed in memory,
> > but when they get written to disk, they will be compressed and all the
> > extra file system blocks for the page will be released. Thus the storage
> > files become sparse.
> >
> > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
> > basically just takes only meaningful data from half-filled pages but does
> > not apply any compression. It is easy to add more if needed.
> >
> > Since we can release only full file system blocks which are typically 4k
> > size, user must configure page size to be at least multiple FS blocks,
> e.g.
> > 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> > pageSize = 4k / 16k = 0.25
> >
> > It is possible to enable compression for existing databases if they were
> > configured for large enough page size. In this case pages will be written
> > to disk in compressed form when updated, and the database will become
> > compressed gradually.
> >
> > There will be 2 new properties on CacheConfiguration
> > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk
> page
> > compression.
> >
> > Compression dictionaries are not supported at the time, but may in the
> > future. IMO it should be added as a separate feature if needed.
> >
> > The only supported platform for now is Linux. Since all popular file
> > systems support sparse files, it must be  relatively easy to support more
> > platforms.
> >
> > Please take a look and provide your thoughts and suggestions.
> >
> > Thanks!
> >
> > Sergi
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Re: Disk page compression for Ignite persistent store

[Andrey Mashenkov]

Hi Sergi,

It is not clear for me will your changes affect PageSnapshot WAL record.
Is it possible to add compression support for PageSnapshot WAL record as
well, to reduce WAL size?

Thanks.

On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin 
wrote:

> Folks,
>
> I've implemented page compression for persistent store and going to merge
> it to master.
>
> https://github.com/apache/ignite/pull/5200
>
> Some design notes:
>
> It employs "hole punching" approach, it means that the pages are kept
> uncompressed in memory,
> but when they get written to disk, they will be compressed and all the
> extra file system blocks for the page will be released. Thus the storage
> files become sparse.
>
> Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
> basically just takes only meaningful data from half-filled pages but does
> not apply any compression. It is easy to add more if needed.
>
> Since we can release only full file system blocks which are typically 4k
> size, user must configure page size to be at least multiple FS blocks, e.g.
> 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> pageSize = 4k / 16k = 0.25
>
> It is possible to enable compression for existing databases if they were
> configured for large enough page size. In this case pages will be written
> to disk in compressed form when updated, and the database will become
> compressed gradually.
>
> There will be 2 new properties on CacheConfiguration
> (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
> compression.
>
> Compression dictionaries are not supported at the time, but may in the
> future. IMO it should be added as a separate feature if needed.
>
> The only supported platform for now is Linux. Since all popular file
> systems support sparse files, it must be  relatively easy to support more
> platforms.
>
> Please take a look and provide your thoughts and suggestions.
>
> Thanks!
>
> Sergi
>


-- 
Best regards,
Andrey V. Mashenkov

Re: Apache Ignite 2.7. Last Mile

[Dmitrii Ryabov]

I agree to revert and make fix for 2.8. So, we will have more time to test
it.

пн, 19 нояб. 2018 г., 10:53 Vladimir Ozerov voze...@gridgain.com:

> +1 for revert.
>
> On Sun, Nov 18, 2018 at 11:31 PM Dmitriy Pavlov 
> wrote:
>
> > I personally don't mind.
> >
> > But I would like Dmitry Ryabov and Alexey Goncharuck share their
> opinions.
> >
> > вс, 18 нояб. 2018 г., 20:43 Nikolay Izhikov :
> >
> > > Yes, I think so.
> > >
> > > вс, 18 нояб. 2018 г., 20:34 Denis Magda dma...@apache.org:
> > >
> > > > Sounds good to me. Are we starting the vote then?
> > > >
> > > > Denis
> > > >
> > > >
> > > >
> > > > On Sun, Nov 18, 2018 at 8:25 AM Nikolay Izhikov  >
> > > > wrote:
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > This issue is the only ticket that blocks 2.7 release.
> > > > >
> > > > > I looked at IGNITE-602 PR and GridToStringBuilder.
> > > > > The code looks complicated for me.
> > > > > And it's not obvious for me how to fix this issue in a short period
> > of
> > > > > time.
> > > > > Especially, code deals with recursion and other things that can
> lead
> > to
> > > > > very dangerous errors.
> > > > >
> > > > > Let's revert this patch and fix it in calmly.
> > > > > Also, we need additional tests for it.
> > > > >
> > > > > В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет:
> > > > > > Ok, I'll check the issue.
> > > > > > пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com>:
> > > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > I've just found that S.toString() implementation is broken in
> > > > > ignite-2.7 and master [1]. It leads to a message
> > > > > > > Wrapper [p=Parent [a=0]Child [b=0, super=]]
> > > > > > > being formed instead of
> > > > > > > Wrapper [p=Child [b=0, super=Parent [a=0]]]
> > > > > > > for classes with inheritance that use
> S.toString(SomeClass.class,
> > > > > this, super.toString()) embedded to some other object.
> > > > > > >
> > > > > > > Dmitrii Ryabov, I've reverted two commits related to IGNITE-602
> > and
> > > > > IGNITE-9209 tickets locally and it fixes the issue. Can you take a
> > look
> > > > at
> > > > > the issue?
> > > > > > >
> > > > > > > I think this regression essentially makes our logs unreadable
> in
> > > some
> > > > > cases and I would like to get it fixed in ignite-2.7 or revert both
> > > > commits
> > > > > from the release.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10301
> > > > > > >
> > > > > > > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov <
> > nizhi...@apache.org
> > > >:
> > > > > > > >
> > > > > > > > Hello, Igniters.
> > > > > > > >
> > > > > > > > We still have 5 tickets for 2.7:
> > > > > > > >
> > > > > > > > IGNITE-10052Andrew Mashenkov Restart node during TX
> causes
> > > > > vacuum error.
> > > > > > > > IGNITE-10170Unassigned   .NET:
> > Services.ServicesTestAsync
> > > > > fails
> > > > > > > > IGNITE-10196Maxim Pudov  Remove kafka-clients-*-test
> > > > > dependency
> > > > > > > > IGNITE-10154Andrey Gura  Critical worker liveness
> check
> > > > > configuration is non-trivial and inconsistent
> > > > > > > > IGNITE-9996 Nikolay Izhikov  Investigate possible
> > performance
> > > > > drop in FSYNC mode for ignite-2.7 compared to ignite-2.6
> > > > > > > >
> > > > > > > >
> > > > > > > > В Чт, 08/11/2018 в 14:25 +0300, Nikolay Izhikov пишет:
> > > > > > > > > I'm OK with this.
> > > > > > > > >
> > > > > > > > > чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:
> > > > > > > > > > Long, long way to release :)
> > > > > > > > > >
> > > > > > > > > > Guys, we have a breaking change in Ignite 2.7 so we must
> > add
> > > > > > > > > > IGNITE-10154 [1] fix to the release.
> > > > > > > > > >
> > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10154
> > > > > > > > > > On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego <
> > > isap...@apache.org
> > > > >
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Guys,
> > > > > > > > > > >
> > > > > > > > > > > I've found the following issue: [1]. It is quite local
> > > (only
> > > > > affects
> > > > > > > > > > > Ignite C++ Linux build system) but quite critical too.
> I
> > > > think
> > > > > it
> > > > > > > > > > > should be included in 2.7.
> > > > > > > > > > >
> > > > > > > > > > > What do you think?
> > > > > > > > > > >
> > > > > > > > > > > [1] -
> https://issues.apache.org/jira/browse/IGNITE-10147
> > > > > > > > > > >
> > > > > > > > > > > Best Regards,
> > > > > > > > > > > Igor
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Oct 30, 2018 at 4:35 PM Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello Nikolay, Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > It seems that we lost the following commit that
> should
> > be
> > > > > included in
> > > > > > > > > > > > 'ignite-2.7' branch
> > > > > > > > > > 

Disk page compression for Ignite persistent store

[Sergi Vladykin]

Folks,

I've implemented page compression for persistent store and going to merge
it to master.

https://github.com/apache/ignite/pull/5200

Some design notes:

It employs "hole punching" approach, it means that the pages are kept
uncompressed in memory,
but when they get written to disk, they will be compressed and all the
extra file system blocks for the page will be released. Thus the storage
files become sparse.

Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
basically just takes only meaningful data from half-filled pages but does
not apply any compression. It is easy to add more if needed.

Since we can release only full file system blocks which are typically 4k
size, user must configure page size to be at least multiple FS blocks, e.g.
8k or 16k. It also means that max compression ratio here is fsBlockSize /
pageSize = 4k / 16k = 0.25

It is possible to enable compression for existing databases if they were
configured for large enough page size. In this case pages will be written
to disk in compressed form when updated, and the database will become
compressed gradually.

There will be 2 new properties on CacheConfiguration
(setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
compression.

Compression dictionaries are not supported at the time, but may in the
future. IMO it should be added as a separate feature if needed.

The only supported platform for now is Linux. Since all popular file
systems support sparse files, it must be  relatively easy to support more
platforms.

Please take a look and provide your thoughts and suggestions.

Thanks!

Sergi

Spark dataframe API will get wrong schema if user executes add/drop column DDL

[Ray]

Currently, when user performs add/remove column DDL, the QueryEntity will not
change.

This result in Spark getting wrong schema because Spark relies on
QueryEntity to construct data frame schema.

In Vladimir Ozerov's reply in dev list,
http://apache-ignite-developers.2346864.n4.nabble.com/Schema-in-CacheConfig-is-not-updated-after-DDL-commands-Add-drop-column-Create-drop-index-td38002.html.

This behavior is by design, so I decide to fix this issue from the Spark
side.

 

So I propose this solution, instead of getting schema by QueryEntity I want
to get schema by a SQL select command.

Nikolay Izhikov, what do you think about this solution?

I already created a ticket in JIRA,
https://issues.apache.org/jira/browse/IGNITE-10314

If you think this solution OK then I'll start implementing.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #5422: IGNITE-10314 QueryEntity is not updated when colu...

[ldzhjn]

Github user ldzhjn closed the pull request at:

https://github.com/apache/ignite/pull/5422


---

[jira] [Created] (IGNITE-10319) Handle suite compilation error failure

[PetrovMikhail (JIRA)]

PetrovMikhail created IGNITE-10319:
--

 Summary: Handle suite compilation error failure
 Key: IGNITE-10319
 URL: https://issues.apache.org/jira/browse/IGNITE-10319
 Project: Ignite
  Issue Type: Task
Reporter: PetrovMikhail


We need to notify community if some suite fails N times in a row with 
compillation errror status.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10320) Web Console: Memory leak in char component

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-10320:
-

 Summary: Web Console: Memory leak in char component
 Key: IGNITE-10320
 URL: https://issues.apache.org/jira/browse/IGNITE-10320
 Project: Ignite
  Issue Type: Task
  Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.8


Found issue with chartjs-plugin-streaming: 
[https://github.com/nagix/chartjs-plugin-streaming/issues/53,] that hidden 
series are not removed by TTL.

We can implement a workaround while this issue is not fixed in 
chartjs-plugin-streaming.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5425: IGNITE-10300 Fixed correct message in case of use...

[voropava]

GitHub user voropava opened a pull request:

https://github.com/apache/ignite/pull/5425

IGNITE-10300 Fixed correct message in case of user/password retries e…

…xceeded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-10300

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5425


commit 82c57cbf0a42057c9c4a7aea26dbdf7da8542a3e
Author: Pavel Voronkin 
Date:   2018-11-09T12:00:03Z

IGNITE-10300 Fixed correct message in case of user/password retries 
exceeded.




---

Re: Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Vladimir Ozerov]

Hi,

In this case Spark integration should be fixed. as we never stated that DDL
updates will be reflected in IgniteCache.getConfiguration().


On Mon, Nov 19, 2018 at 11:58 AM Ray  wrote:

> When user performs column and index modification operation in SQL(ex create
> index, drop index, add column, drop column),  QueryEntity in
> CacheConfiguration for the modified cache is not updated.
>
> Here's my analysis
>
> QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
> the original QueryEntity is not updated when modification happens.
>
> I have created a ticket for this issue
> https://issues.apache.org/jira/browse/IGNITE-10314
>
> But as Vlad said in the comments "public configuration is immutable, it
> represents initial cache parameters. So it is expected that configuration
> will not be updated after DDL commands. Real changes are accumulated in
> separate query entity which is hidden from user and used internally"
>
> But I think it's only reasonable to return the newest QueryEntity to user.
>
> For example, a user adds a column to a table then he reads data using Spark
> data frame API which currently relies on QueryEntity to construct data
> frame
> schema, so user will get wrong schema.
>
> What do you guys think?
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Ray]

When user performs column and index modification operation in SQL(ex create
index, drop index, add column, drop column),  QueryEntity in
CacheConfiguration for the modified cache is not updated.

Here's my analysis 

QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
the original QueryEntity is not updated when modification happens.

I have created a ticket for this issue 
https://issues.apache.org/jira/browse/IGNITE-10314

But as Vlad said in the comments "public configuration is immutable, it
represents initial cache parameters. So it is expected that configuration
will not be updated after DDL commands. Real changes are accumulated in
separate query entity which is hidden from user and used internally"

But I think it's only reasonable to return the newest QueryEntity to user.

For example, a user adds a column to a table then he reads data using Spark
data frame API which currently relies on QueryEntity to construct data frame
schema, so user will get wrong schema.

What do you guys think?






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ololo3000 opened a new pull request #74: IGNITE-10275 Refactor of visa caching. Jira spam fix.

[GitBox]

ololo3000 opened a new pull request #74: IGNITE-10275 Refactor of visa caching. 
Jira spam fix.
URL: https://github.com/apache/ignite-teamcity-bot/pull/74
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-10318) Web console: use newer panels in query notebook screen

[Ilya Borisov (JIRA)]

Ilya Borisov created IGNITE-10318:
-

 Summary: Web console: use newer panels in query notebook screen
 Key: IGNITE-10318
 URL: https://issues.apache.org/jira/browse/IGNITE-10318
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Ilya Borisov
Assignee: Ilya Borisov


"Queries notebook" screen is the last place where older style of collapsible 
panels has remain in use.

What to do:
1. Replace panels on "queries notebook" screen with panel-collapsible component.
2. Update query rename feature to use "Rename" button and new name prompt 
instead on inline form.
3. Move query remove button to context menu in panel actions.
4. Move "Execute", "Execute on selected node", "Explain" and "Scan" button to 
panel actions. All actions, except for "Execute" and "Scan", should be in 
context menu.
5. Remove legacy code used in older style collapsible panels.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #2476: Ignite-5983: Memory policy metrics should be avai...

[shroman]

Github user shroman closed the pull request at:

https://github.com/apache/ignite/pull/2476


---

[jira] [Created] (IGNITE-10317) “Failed to do service reassignment” on server stop with singleton service

[Grigory Domozhirov (JIRA)]

Grigory Domozhirov created IGNITE-10317:
---

 Summary: “Failed to do service reassignment” on server stop with 
singleton service
 Key: IGNITE-10317
 URL: https://issues.apache.org/jira/browse/IGNITE-10317
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Grigory Domozhirov


I have Ignite cluster with 2 server nodes and singleton service deployed on it.

If I first stop node which currently runs service, and then (with particular 
delay, about ~ 100 ms) second one, I'm getting
{code:java}
ERROR o.a.i.i.p.s.GridServiceProcessor - Failed to do service reassignment 
(will retry): statService
org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted 
while waiting for future to complete.
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:185)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at 
org.apache.ignite.internal.processors.service.GridServiceProcessor$TopologyListener$1.run0(GridServiceProcessor.java:1810)
at 
org.apache.ignite.internal.processors.service.GridServiceProcessor$DepRunnable.run(GridServiceProcessor.java:2027)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
 I guess, second server sees another server is down, so it wants to start 
service. Then at unlucky moment this server stops, and {{ExecutorService 
depExe}} at {{GridServiceProcessor}} get interrupted, so exception occurs. But 
Ignite knows if it stopping, so I'm sure Ignite could expect interrupts, or 
take care of it in some way.

Ignite version: 2.5.2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #5424: GG PE 2.7.1

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/5424

GG PE 2.7.1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.7.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5424


commit f89f8e34ff81c89a798a3bef6d52daceede55945
Author: Anton Kalashnikov 
Date:   2018-09-28T15:20:00Z

IGNITE-9731 Fixed NPE on concurrent WAL flush - Fixes #4863.

Signed-off-by: Alexey Goncharuk 

commit 090c890d2a47b6ee8499f978e9be74184919ac6c
Author: Aleksei Scherbakov 
Date:   2018-09-28T15:50:05Z

IGNITE-9612 Fixed checkpoint listener logic - Fixes #4864.

Signed-off-by: Alexey Goncharuk 

commit c328ea6c6d48c8b5ac2ada5247a7f7188d085b63
Author: Ilya Borisov 
Date:   2018-10-01T03:45:40Z

IGNITE-9569 Web Console: Fixed broken unit test.

(cherry picked from commit 9ab8ebd7adaa70548ba013f4d8c7de05ba48312f)

commit 8a05a9627320cfa0989848bab86da421f1f3fc56
Author: Pavel Kovalenko 
Date:   2018-10-01T08:52:42Z

IGNITE-9736 Fixed usages of Discovery SPI listener. - Fixes #4868.

Signed-off-by: Alexey Goncharuk 

commit 4de78fa59fddb3332beb5dd4a9dbc55e7047a350
Author: Anton Vinogradov 
Date:   2018-10-01T11:16:15Z

IGNITE-7251 Remove term "fabric" from Ignite deliverables

(cherry picked from commit 7989173b4d64a681fd08b7d7bcda2d26559941ae)
Signed-off-by: Anton Vinogradov 

commit e86b239037983a6681b3880ebf0dceb3b2e90682
Author: ascherbakoff 
Date:   2018-10-01T11:54:41Z

IGNITE-9658 Add an option to reuse memory after deactivation - Fixes #4874.

Signed-off-by: Alexey Goncharuk 

commit 7fe52af08adf0d8a125e0aa52ff41c13fe751c80
Author: Maxim Muzafarov 
Date:   2018-10-01T13:03:24Z

IGNITE-9723 Use blockingSectionBegin method to wrap exchange future

Signed-off-by: Andrey Gura 

commit 7cb0784cd3cf9b759c1c28eda04210d783cb7598
Author: Anton Kalashnikov 
Date:   2018-10-01T14:39:24Z

IGNITE-9741 Fix SegmentArchivedStorage and SegmentCompressStorage remain 
`iterrupted` after de-activation occurs before activation - Fixes #4878.

Signed-off-by: Dmitriy Govorukhin 

(cherry picked from commit caa4b0aa9d117d08d78186c82627b23b1d21fb5e)

commit 78fe3eaeff702d741f99e73d74f3ace5d29bad46
Author: Dmitriy Govorukhin 
Date:   2018-10-01T14:57:25Z

IGNITE-9741 Fix compile test

commit 861e5f708357e412bddc14b402b72ceed79d0df0
Author: uday 
Date:   2018-10-01T16:31:18Z

IGNITE-8251 Reduce testPageEviction run time - Fixes #4577.

Signed-off-by: Dmitriy Pavlov 

(cherry picked from commit 0e29b87)

commit da3d9205a1d28e27a23569f25f053874c38b4232
Author: devozerov 
Date:   2018-10-02T11:27:39Z

IGNITE-9390: MVCC: added new properties to .NET IgniteConfiguration and 
CacheAtomicityMode. This closes #4887.

commit 9d6e6ff394c05ddf7ef31a9d9ed1b492d9eeba69
Author: Anton Kalashnikov 
Date:   2018-10-02T15:26:20Z

IGNITE-9761 Fixed deadlock in WAL manager - Fixes #4890.

Signed-off-by: Alexey Goncharuk 

commit 3355201f3e8cafd23b2250aaf3b91b8b8ed1
Author: Anton Kalashnikov 
Date:   2018-10-02T16:07:36Z

IGNITE-9760 Fixed NPE in WAL manager for FSYNC mode - Fixes #4888.

Signed-off-by: Alexey Goncharuk 

commit a9dc14e2e638eddaf2adaa700c81dc783062dca5
Author: Peter Ivanov 
Date:   2018-10-03T07:44:59Z

IGNITE-9772: Removed word "fabric" from various places. This closes #4889.

commit 678bc9187dcdeb2fd7b3ef8669405bdaee0e9b90
Author: tledkov-gridgain 
Date:   2018-10-03T10:09:17Z

IGNITE-9727: Fixed ignite.sh and ignite.bat scripts to support Java 9+. 
This closes #4871.

commit bf0c1225ac42257a50fb8673c54f278579839feb
Author: Oleg Ignatenko 
Date:   2018-10-03T10:28:40Z

IGNITE-9691 testConcurrentAuthorize uses outdated assumption about 
exception message
- updated test assumptions to accommodate changes in the implementation
-- verified with diffs overview, clean rebuild and trial execution of 
modified tests on my machine - Fixes #4828.

Signed-off-by: Alexey Goncharuk 

commit 85a6b477c7dc9482f024e1416e5f36ca36f550ff
Author: AMRepo 
Date:   2018-10-03T12:50:07Z

IGNITE-9540: MVCC: support IgniteCache.invoke method family. This closes 
#4832. This closes #4881.

commit e71df460ad82a310477f97f528704971dba1cf12
Author: devozerov 
Date:   2018-10-03T13:09:52Z

IGNITE-9750: MVCC: do not allow near cache for TRANSACTIONAL_SNAPSHOT 
atomicity mode. This closes #4891.

commit cfe3654bea40e8ec896403cba17fd7f18e8ab381
Author: devozerov 
Date:   2018-10-03T18:38:06Z

IGNITE-9722: MVCC: disallow reads from backups for GET operations. This 
closes #4903.

commit beb7eb0b26b647c221157e0f734202981eb47ff5
Author: