Re: Apache Ignite 2.7. Last Mile

2018-11-19 Thread Nikolay Izhikov
Hello, Dmitrii.

I see 2 tickets for this improvement:

IGNITE-602 - [Test] GridToStringBuilder is vulnerable for StackOverflowError 
caused by infinite recursion [1]
IGNITE-9209 - GridDistributedTxMapping.toString() returns broken string [2]

Should we revert both commits?

[1] https://github.com/apache/ignite/commit/d67c5bf
[2] https://github.com/apache/ignite/commit/9bb9c04


В Пн, 19/11/2018 в 13:36 +0300, Dmitrii Ryabov пишет:
> I agree to revert and make fix for 2.8. So, we will have more time to test
> it.
> 
> пн, 19 нояб. 2018 г., 10:53 Vladimir Ozerov voze...@gridgain.com:
> 
> > +1 for revert.
> > 
> > On Sun, Nov 18, 2018 at 11:31 PM Dmitriy Pavlov 
> > wrote:
> > 
> > > I personally don't mind.
> > > 
> > > But I would like Dmitry Ryabov and Alexey Goncharuck share their
> > 
> > opinions.
> > > 
> > > вс, 18 нояб. 2018 г., 20:43 Nikolay Izhikov :
> > > 
> > > > Yes, I think so.
> > > > 
> > > > вс, 18 нояб. 2018 г., 20:34 Denis Magda dma...@apache.org:
> > > > 
> > > > > Sounds good to me. Are we starting the vote then?
> > > > > 
> > > > > Denis
> > > > > 
> > > > > 
> > > > > 
> > > > > On Sun, Nov 18, 2018 at 8:25 AM Nikolay Izhikov  > > > > wrote:
> > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > This issue is the only ticket that blocks 2.7 release.
> > > > > > 
> > > > > > I looked at IGNITE-602 PR and GridToStringBuilder.
> > > > > > The code looks complicated for me.
> > > > > > And it's not obvious for me how to fix this issue in a short period
> > > 
> > > of
> > > > > > time.
> > > > > > Especially, code deals with recursion and other things that can
> > 
> > lead
> > > to
> > > > > > very dangerous errors.
> > > > > > 
> > > > > > Let's revert this patch and fix it in calmly.
> > > > > > Also, we need additional tests for it.
> > > > > > 
> > > > > > В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет:
> > > > > > > Ok, I'll check the issue.
> > > > > > > пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk <
> > > > > > 
> > > > > > alexey.goncha...@gmail.com>:
> > > > > > > > 
> > > > > > > > Igniters,
> > > > > > > > 
> > > > > > > > I've just found that S.toString() implementation is broken in
> > > > > > 
> > > > > > ignite-2.7 and master [1]. It leads to a message
> > > > > > > > Wrapper [p=Parent [a=0]Child [b=0, super=]]
> > > > > > > > being formed instead of
> > > > > > > > Wrapper [p=Child [b=0, super=Parent [a=0]]]
> > > > > > > > for classes with inheritance that use
> > 
> > S.toString(SomeClass.class,
> > > > > > this, super.toString()) embedded to some other object.
> > > > > > > > 
> > > > > > > > Dmitrii Ryabov, I've reverted two commits related to IGNITE-602
> > > 
> > > and
> > > > > > IGNITE-9209 tickets locally and it fixes the issue. Can you take a
> > > 
> > > look
> > > > > at
> > > > > > the issue?
> > > > > > > > 
> > > > > > > > I think this regression essentially makes our logs unreadable
> > 
> > in
> > > > some
> > > > > > cases and I would like to get it fixed in ignite-2.7 or revert both
> > > > > 
> > > > > commits
> > > > > > from the release.
> > > > > > > > 
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10301
> > > > > > > > 
> > > > > > > > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org
> > > > > :
> > > > > > > > > 
> > > > > > > > > Hello, Igniters.
> > > > > > > > > 
> > > > > > > > > We still have 5 tickets for 2.7:
> > > > > > > > > 
> > > > > > > > > IGNITE-10052Andrew Mashenkov Restart node 

Re: [DISCUSSION] Spark Data Frame through Thin Client

2018-11-19 Thread Nikolay Izhikov
IGNITE-10325 created.

ср, 7 нояб. 2018 г., 11:42 Ray ray...@cisco.com:

> From my past experience with Spark Data Frame API, the thick client
> approach
> leads to many usability problems.
>
> Ex.
>
> http://apache-ignite-users.70518.x6.nabble.com/Local-node-SEGMENTED-error-causing-node-goes-down-for-no-obvious-reason-td25061.html
>
> I think it makes a lot of sense to change to thin client.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


[jira] [Created] (IGNITE-10325) Spark Data Frame - Thin Client

2018-11-19 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-10325:


 Summary: Spark Data Frame - Thin Client
 Key: IGNITE-10325
 URL: https://issues.apache.org/jira/browse/IGNITE-10325
 Project: Ignite
  Issue Type: Improvement
  Components: spark
Affects Versions: 2.6
Reporter: Nikolay Izhikov


For now, client node required to connect to Ignite cluster from Spark.

We need to add ability to use Thin Client protocol for Spark integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Apache Ignite 2.7. Last Mile

2018-11-18 Thread Nikolay Izhikov
Yes, I think so.

вс, 18 нояб. 2018 г., 20:34 Denis Magda dma...@apache.org:

> Sounds good to me. Are we starting the vote then?
>
> Denis
>
>
>
> On Sun, Nov 18, 2018 at 8:25 AM Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > This issue is the only ticket that blocks 2.7 release.
> >
> > I looked at IGNITE-602 PR and GridToStringBuilder.
> > The code looks complicated for me.
> > And it's not obvious for me how to fix this issue in a short period of
> > time.
> > Especially, code deals with recursion and other things that can lead to
> > very dangerous errors.
> >
> > Let's revert this patch and fix it in calmly.
> > Also, we need additional tests for it.
> >
> > В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет:
> > > Ok, I'll check the issue.
> > > пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk <
> > alexey.goncha...@gmail.com>:
> > > >
> > > > Igniters,
> > > >
> > > > I've just found that S.toString() implementation is broken in
> > ignite-2.7 and master [1]. It leads to a message
> > > > Wrapper [p=Parent [a=0]Child [b=0, super=]]
> > > > being formed instead of
> > > > Wrapper [p=Child [b=0, super=Parent [a=0]]]
> > > > for classes with inheritance that use S.toString(SomeClass.class,
> > this, super.toString()) embedded to some other object.
> > > >
> > > > Dmitrii Ryabov, I've reverted two commits related to IGNITE-602 and
> > IGNITE-9209 tickets locally and it fixes the issue. Can you take a look
> at
> > the issue?
> > > >
> > > > I think this regression essentially makes our logs unreadable in some
> > cases and I would like to get it fixed in ignite-2.7 or revert both
> commits
> > from the release.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-10301
> > > >
> > > > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov :
> > > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > We still have 5 tickets for 2.7:
> > > > >
> > > > > IGNITE-10052Andrew Mashenkov Restart node during TX causes
> > vacuum error.
> > > > > IGNITE-10170Unassigned   .NET: Services.ServicesTestAsync
> > fails
> > > > > IGNITE-10196Maxim Pudov  Remove kafka-clients-*-test
> > dependency
> > > > > IGNITE-10154Andrey Gura  Critical worker liveness check
> > configuration is non-trivial and inconsistent
> > > > > IGNITE-9996 Nikolay Izhikov  Investigate possible performance
> > drop in FSYNC mode for ignite-2.7 compared to ignite-2.6
> > > > >
> > > > >
> > > > > В Чт, 08/11/2018 в 14:25 +0300, Nikolay Izhikov пишет:
> > > > > > I'm OK with this.
> > > > > >
> > > > > > чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:
> > > > > > > Long, long way to release :)
> > > > > > >
> > > > > > > Guys, we have a breaking change in Ignite 2.7 so we must add
> > > > > > > IGNITE-10154 [1] fix to the release.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10154
> > > > > > > On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego  >
> > wrote:
> > > > > > > >
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I've found the following issue: [1]. It is quite local (only
> > affects
> > > > > > > > Ignite C++ Linux build system) but quite critical too. I
> think
> > it
> > > > > > > > should be included in 2.7.
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > > [1] - https://issues.apache.org/jira/browse/IGNITE-10147
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Igor
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 30, 2018 at 4:35 PM Вячеслав Коптилин <
> > slava.kopti...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Nikolay, Igniters,
> > > > > > > > >
> > &g

Re: Apache Ignite 2.7. Last Mile

2018-11-18 Thread Nikolay Izhikov
Hello, Igniters.

This issue is the only ticket that blocks 2.7 release.

I looked at IGNITE-602 PR and GridToStringBuilder.
The code looks complicated for me.
And it's not obvious for me how to fix this issue in a short period of time.
Especially, code deals with recursion and other things that can lead to very 
dangerous errors.

Let's revert this patch and fix it in calmly.
Also, we need additional tests for it.

В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет:
> Ok, I'll check the issue.
> пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk :
> > 
> > Igniters,
> > 
> > I've just found that S.toString() implementation is broken in ignite-2.7 
> > and master [1]. It leads to a message
> > Wrapper [p=Parent [a=0]Child [b=0, super=]]
> > being formed instead of
> > Wrapper [p=Child [b=0, super=Parent [a=0]]]
> > for classes with inheritance that use S.toString(SomeClass.class, this, 
> > super.toString()) embedded to some other object.
> > 
> > Dmitrii Ryabov, I've reverted two commits related to IGNITE-602 and 
> > IGNITE-9209 tickets locally and it fixes the issue. Can you take a look at 
> > the issue?
> > 
> > I think this regression essentially makes our logs unreadable in some cases 
> > and I would like to get it fixed in ignite-2.7 or revert both commits from 
> > the release.
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-10301
> > 
> > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov :
> > > 
> > > Hello, Igniters.
> > > 
> > > We still have 5 tickets for 2.7:
> > > 
> > > IGNITE-10052Andrew Mashenkov Restart node during TX causes vacuum 
> > > error.
> > > IGNITE-10170Unassigned   .NET: Services.ServicesTestAsync fails
> > > IGNITE-10196    Maxim Pudov  Remove kafka-clients-*-test dependency
> > > IGNITE-10154Andrey Gura  Critical worker liveness check 
> > > configuration is non-trivial and inconsistent
> > > IGNITE-9996 Nikolay Izhikov  Investigate possible performance drop in 
> > > FSYNC mode for ignite-2.7 compared to ignite-2.6
> > > 
> > > 
> > > В Чт, 08/11/2018 в 14:25 +0300, Nikolay Izhikov пишет:
> > > > I'm OK with this.
> > > > 
> > > > чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:
> > > > > Long, long way to release :)
> > > > > 
> > > > > Guys, we have a breaking change in Ignite 2.7 so we must add
> > > > > IGNITE-10154 [1] fix to the release.
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10154
> > > > > On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego  wrote:
> > > > > > 
> > > > > > Guys,
> > > > > > 
> > > > > > I've found the following issue: [1]. It is quite local (only affects
> > > > > > Ignite C++ Linux build system) but quite critical too. I think it
> > > > > > should be included in 2.7.
> > > > > > 
> > > > > > What do you think?
> > > > > > 
> > > > > > [1] - https://issues.apache.org/jira/browse/IGNITE-10147
> > > > > > 
> > > > > > Best Regards,
> > > > > > Igor
> > > > > > 
> > > > > > 
> > > > > > On Tue, Oct 30, 2018 at 4:35 PM Вячеслав Коптилин 
> > > > > > 
> > > > > > wrote:
> > > > > > 
> > > > > > > Hello Nikolay, Igniters,
> > > > > > > 
> > > > > > > It seems that we lost the following commit that should be 
> > > > > > > included in
> > > > > > > 'ignite-2.7' branch
> > > > > > > (It looks like the change was not accidentally cherry-picked from 
> > > > > > > 'master'
> > > > > > > to 'ignite-2.7')
> > > > > > >  -
> > > > > > > 
> > > > > > > https://github.com/apache/ignite/commit/6e0ff06f8e309657a16c94da605348d9c3b804ad
> > > > > > > 
> > > > > > > The most important part is the change introduced into 
> > > > > > > GridDhtAtomicCache,
> > > > > > > the fix prevents NullPointerException during cache updates under 
> > > > > > > some
> > > > > > > circumstances.
> > > > > > > So, I propose including the fix into ignite-2.7, at least the 
> >

Re: [ANNOUNCE] Welcome Alexey Plehanov as a new committer

2018-11-17 Thread Nikolay Izhikov
Congratulations, Alexey!

сб, 17 нояб. 2018 г., 12:07 Dmitriy Pavlov dpav...@apache.org:

> The Apache Ignite Project Management Committee (PMC) has invited Alexey
> Plehanov to become a new committer and are happy to announce that he has
> accepted.
>
>
>
> Being a committer enables you to more easily make changes without needing
> to go through the patch submission process
>
>
>
> Alexey is doing an amazing amount of work, contributes ideas, e.g: the idea
> of SQL views for system metrics and parameters, idea of compatible CRC, and
> others. We were impressed by his willingness to help others.
>
>
>
> Alexey, keep your pace!
>
>
>
> Igniters, please join me in welcoming Alexey and congratulating him on his
> new role in the Apache Ignite Community.
>
>
>
> Thanks
>
> Dmitriy Pavlov
>


[jira] [Created] (IGNITE-10256) Yardstick: output benchmark parameters to HTML report

2018-11-14 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-10256:


 Summary: Yardstick: output benchmark parameters to HTML report
 Key: IGNITE-10256
 URL: https://issues.apache.org/jira/browse/IGNITE-10256
 Project: Ignite
  Issue Type: Bug
  Components: yardstick
Affects Versions: 2.6
Reporter: Nikolay Izhikov
 Fix For: 2.8


For now, yardstick doesn't output benchmark parameters to resulting HTML report.
It would be useful to see these parameters into the report.

* benchmark parameter
* jvm parameter
* nodes(server, client) count
* thread count
* etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: JDBC/SQL Contributions

2018-11-12 Thread Nikolay Izhikov
Hello,Peter.

Welcome to Ignite community!
First of all, please, subscribe to dev-list.
Processes of code and doc contribution are described on wiki

https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

> which repo should I clone? It looks like there are two.

https://github.com/apache/ignite


В Вт, 13/11/2018 в 00:42 +, Peter Borissow пишет:
> Dear Devs,I starting experimenting with Ignite using java+jdbc over the 
> weekend. Really cool technology! In my tests, I ran into a couple issues that 
> I might have some time to help fix. For example:
> 
>   (1) Ignite doesn't support auto incrementing keys so you can't use SQL
>   keywords like "BIGSERIAL". Instead, you must explicitly set an ID
>   https://issues.apache.org/jira/browse/IGNITE-5625  
> https://apacheignite.readme.io/docs/id-generator
>  
>   (2) Related to above. You cannot retrieve values via rs.getGeneratedKey()
>   The following statement throws an error:
>   conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
>  
>   (3) ResultSet Concurrency of CONCUR_UPDATABLE throws an exception
>   stmt = conn.createStatement(rs.TYPE_FORWARD_ONLY, rs.CONCUR_UPDATABLE);
>   However, CONCUR_READ_ONLY works.
>  
>   (4) RS metadata is incomplete. Many fields types are returned as "object"
>  
>   (5) Strings must be explicitly cast to CLOB for "text" fields
>   Value conversion failed [from=java.lang.String, to=java.sql.Clob]
>   
> org.apache.ignite.internal.processors.query.h2.dml.DmlUtils.convert(DmlUtils.java:117)
> 
> 
> What is the process for fixing bugs and implementing new features? I don't 
> see the process documented on the site.
> Also, which repo should I clone? It looks like there are two.
> 
> Finally, is there someone I can work with directly to help get started? For 
> example, I see that "pavel-kuznetsov" has been working on the JDBC metadata. 
> Maybe we can collaborate on issues 4+5?
> 
> 
> Thanks in advance,Peter
> 


signature.asc
Description: This is a digitally signed message part


Re: Service grid redesign

2018-11-10 Thread Nikolay Izhikov
Hello, Vladimir.

I'm agree with you.

Can we write the list of reviewers for this feature?
Without a date or similar. 
Just a list of experts who should review this feature.

В Сб, 10/11/2018 в 14:01 +0300, Vladimir Ozerov пишет:
> Igniters,
> 
> This is very huge thing with complex algorithms behind. We should not merge
> it to the product unless several additional thorough reviews are ready,
> irrespectively of how long will it take. We are about quality, not speed.
> 
> сб, 10 нояб. 2018 г. в 1:30, Denis Magda :
> 
> > Vyacheslav,
> > 
> > What are the cases when the service can be redeployed? Affinity, failure,
> > etc., right. It would be good to list all the cases on the wiki and then
> > our tech writers will get everything documented.
> > 
> > --
> > Denis
> > 
> > On Thu, Nov 8, 2018 at 11:06 PM Vyacheslav Daradur 
> > wrote:
> > 
> > > Denis,
> > > 
> > > Services reassignment process takes into account previous assignments
> > > to avoid redundant redeployments.
> > > So, in the described case, ServiceA won't be moved from node1 to node2.
> > > On Fri, Nov 9, 2018 at 4:41 AM Denis Magda  wrote:
> > > > 
> > > > Vyacheslav,
> > > > 
> > > > First of all, thanks for archiving this milestone and rolling out these
> > > 
> > > new
> > > > capabilities.
> > > > 
> > > > Speaking of the topology change events [1], does the new architecture
> > > 
> > > avoid
> > > > a running service redeployment when a new node joins? For instance,
> > 
> > let's
> > > > say I have ServiceA running node1, then node2 joins and I don't want
> > 
> > the
> > > > service to be redeployed to any other node.
> > > > 
> > > > [1]
> > > > 
> > 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584#ServiceGridredesign.Phase1.Implementationdetails.-Topologychange
> > > > 
> > > > --
> > > > Denis
> > > > 
> > > > On Wed, Nov 7, 2018 at 7:04 AM Vyacheslav Daradur  > > > wrote:
> > > > 
> > > > > Dmitriy, I published documentation in wiki:
> > > > > 
> > 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584
> > > > > 
> > > > > Thank you!
> > > > > On Wed, Nov 7, 2018 at 5:10 PM Dmitriy Pavlov  > > > > wrote:
> > > > > > 
> > > > > > Hi I think wiki is better than any attached docs. Could you please
> > > > > 
> > > > > create a
> > > > > > page?
> > > > > > 
> > > > > > ср, 7 нояб. 2018 г., 14:39 Vyacheslav Daradur  > > 
> > > :
> > > > > > 
> > > > > > > I prepared a description of the implemented solution and attached
> > > 
> > > it
> > > > > > > to the issue [1].
> > > > > > > 
> > > > > > > This should help during a review. Should I post the document into
> > > 
> > > wiki
> > > > > or
> > > > > > > IEP?
> > > > > > > 
> > > > > > > I'd like to ask Ignite's experts review the solution [1] [2],
> > > 
> > > please?
> > > > > > > 
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > > > > [2] https://github.com/apache/ignite/pull/4434
> > > > > > > On Wed, Oct 31, 2018 at 5:04 PM Vyacheslav Daradur <
> > > > > 
> > > > > daradu...@gmail.com>
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > Hi, Igniters! Good news!
> > > > > > > > 
> > > > > > > > Service Grid Redesign Phase 1 - is in Patch Available now.
> > > > > > > > 
> > > > > > > > Nikolay Izhikov has reviewed implementation.
> > > > > > > > 
> > > > > > > > However, we need additional review from other Ignite experts.
> > > > > > > > 
> > > > > > > > Here is an umbrella ticket [1] and PR [2].
> > > > > > > > 
> > > > > > > > Could someone step in and do the review?
> > > > > > > > 
> > > > > > > > [1] https://

Re: Apache Ignite 2.7. Last Mile

2018-11-08 Thread Nikolay Izhikov
Hello, Igniters.

We still have 5 tickets for 2.7:

IGNITE-10052Andrew Mashenkov Restart node during TX causes vacuum error.
IGNITE-10170Unassigned   .NET: Services.ServicesTestAsync fails
IGNITE-10196Maxim Pudov  Remove kafka-clients-*-test dependency
IGNITE-10154Andrey Gura  Critical worker liveness check configuration 
is non-trivial and inconsistent
IGNITE-9996 Nikolay Izhikov  Investigate possible performance drop in FSYNC 
mode for ignite-2.7 compared to ignite-2.6


В Чт, 08/11/2018 в 14:25 +0300, Nikolay Izhikov пишет:
> I'm OK with this.
> 
> чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:
> > Long, long way to release :)
> > 
> > Guys, we have a breaking change in Ignite 2.7 so we must add
> > IGNITE-10154 [1] fix to the release.
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-10154
> > On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego  wrote:
> > >
> > > Guys,
> > >
> > > I've found the following issue: [1]. It is quite local (only affects
> > > Ignite C++ Linux build system) but quite critical too. I think it
> > > should be included in 2.7.
> > >
> > > What do you think?
> > >
> > > [1] - https://issues.apache.org/jira/browse/IGNITE-10147
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Tue, Oct 30, 2018 at 4:35 PM Вячеслав Коптилин 
> > > 
> > > wrote:
> > >
> > > > Hello Nikolay, Igniters,
> > > >
> > > > It seems that we lost the following commit that should be included in
> > > > 'ignite-2.7' branch
> > > > (It looks like the change was not accidentally cherry-picked from 
> > > > 'master'
> > > > to 'ignite-2.7')
> > > >  -
> > > >
> > > > https://github.com/apache/ignite/commit/6e0ff06f8e309657a16c94da605348d9c3b804ad
> > > >
> > > > The most important part is the change introduced into 
> > > > GridDhtAtomicCache,
> > > > the fix prevents NullPointerException during cache updates under some
> > > > circumstances.
> > > > So, I propose including the fix into ignite-2.7, at least the change of
> > > > GridDhtAtomicCache.
> > > >
> > > > Thanks,
> > > > Slava.
> > > >
> > > > пн, 29 окт. 2018 г. в 11:20, Nikolay Izhikov :
> > > >
> > > > > Hello, guys.
> > > > >
> > > > > For today we have 11 tickets mapped to 2.7
> > > > >
> > > > > IGNITE-10010 Alexey Goncharuk Node halted if second node was stopped,
> > > > then
> > > > > cache destroyed, then second node returned
> > > > > IGNITE-10015 Alexey Goncharuk Sporadic JVM crash due to restart nodes
> > > > > IGNITE-10013 Unassigned Node restart may lead to NPE in
> > > > > GridDhtPartitionsExchangeFuture
> > > > > IGNITE-9928 Igor Seliverstov MVCC TX: Late affinity assignment 
> > > > > support.
> > > > > IGNITE-9985 Igor Seliverstov MVCC TX: fix backup mappings
> > > > > IGNITE-10007 Sergey Kozlov Deactivation hangs if an open transaction
> > > > exists
> > > > > IGNITE-10004 Andrew Mashenkov Parse error leads to leave the 
> > > > > transaction
> > > > > IGNITE-10024 Ivan Pavlukhin MVCC TX: Stackoverflow during 
> > > > > DhtEnlistFuture
> > > > > mapping
> > > > > IGNITE-9996 Alexey Goncharuk Investigate possible performance drop in
> > > > FSYNC
> > > > > mode for ignite-2.7 compared to ignite-2.6
> > > > > IGNITE-9982 Ivan Pavlukhin SQLLine: can't run with option
> > > > > --autoCommit=false or true
> > > > > IGNITE-9828 Roman Kondakov MVCC: Continuous query failover.
> > > > >
> > > > >
> > > > > пт, 26 окт. 2018 г. в 9:59, Vladimir Ozerov :
> > > > >
> > > > > > Hi Nikolay,
> > > > > >
> > > > > > I do not know. We need to investigate them first. These are all
> > > > > > regressions, so decision about impact and urgency should be made
> > > > > separately
> > > > > > for every ticket.
> > > > > >
> > > > > > On Fri, Oct 26, 2018 at 9:57 AM Nikolay Izhikov 
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, Igniters.

Re: Service grid redesign

2018-11-08 Thread Nikolay Izhikov
Hello, Igniters.

Please, respond if anyone wish to do the additional review of this
improvement.

I think it's ready to be merged, so if noone has time to review, I can
merge the patch.

ср, 7 нояб. 2018 г., 18:04 Vyacheslav Daradur daradu...@gmail.com:

> Dmitriy, I published documentation in wiki:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584
>
> Thank you!
> On Wed, Nov 7, 2018 at 5:10 PM Dmitriy Pavlov 
> wrote:
> >
> > Hi I think wiki is better than any attached docs. Could you please
> create a
> > page?
> >
> > ср, 7 нояб. 2018 г., 14:39 Vyacheslav Daradur :
> >
> > > I prepared a description of the implemented solution and attached it
> > > to the issue [1].
> > >
> > > This should help during a review. Should I post the document into wiki
> or
> > > IEP?
> > >
> > > I'd like to ask Ignite's experts review the solution [1] [2], please?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > [2] https://github.com/apache/ignite/pull/4434
> > > On Wed, Oct 31, 2018 at 5:04 PM Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > > >
> > > > Hi, Igniters! Good news!
> > > >
> > > > Service Grid Redesign Phase 1 - is in Patch Available now.
> > > >
> > > > Nikolay Izhikov has reviewed implementation.
> > > >
> > > > However, we need additional review from other Ignite experts.
> > > >
> > > > Here is an umbrella ticket [1] and PR [2].
> > > >
> > > > Could someone step in and do the review?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > [2] https://github.com/apache/ignite/pull/4434
> > > > On Sat, Aug 18, 2018 at 11:44 AM Denis Mekhanikov <
> dmekhani...@gmail.com>
> > > wrote:
> > > > >
> > > > > Pavel, could you assist?
> > > > >
> > > > > Does it make sense for .Net to specify service class name instead
> of
> > > its
> > > > > implementation?
> > > > >
> > > > > I think, it shouldn't be a problem.
> > > > >
> > > > > Denis
> > > > >
> > > > > On Sat, Aug 18, 2018, 11:33 Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I think that the replacement of serialized instance makes sense
> to me
> > > > > > for Java part.
> > > > > >
> > > > > > But how it should work for .NET client?
> > > > > >
> > > > > > On Tue, Aug 14, 2018 at 4:07 PM Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > On Tue, Aug 14, 2018 at 6:10 AM, Nikita Amelchev <
> > > nsamelc...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello, Igniters.
> > > > > > > >
> > > > > > > > I am working on task [1] that would replace serialized
> service's
> > > > > > instance
> > > > > > > > by service's class name and properties map in
> > > {ServiceConfiguration}.
> > > > > > > >
> > > > > > > > The task describes that we should use
> > > > > > > > {String className} + {Map properties} instead
> > > {Service
> > > > > > > > srvc}.
> > > > > > > >
> > > > > > > > I'd like to clarify the following questions:
> > > > > > > >
> > > > > > > > 1. What about public methods?
> > > > > > > > I suggest to mark them as deprecated and use class name of
> > > provided
> > > > > > > > instance.
> > > > > > > > Also to add deploying methods with new parameters:
> > > > > > > >
> > > > > > > > @Deprecated
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, Service svc)
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName, Map prop)
> > > > > > > >
> > > > > > >
> > > > > > > I think this makes sense, but I would like other committers to
> > > confirm.
> > > > > > > Perhaps Vladimir Ozerov should comment here.
> > > > > > >
> > > > > > >
> > > > > > > > 2. Is {Map properties} parameter mandatory
> when
> > > > > > deploying a
> > > > > > > > service?
> > > > > > > > Is it make sense to add deploying methods without it? For
> > > example:
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName)
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName, Map prop)
> > > > > > > >
> > > > > > >
> > > > > > > I would always ask the user to pass the property map, but would
> > > allow it
> > > > > > to
> > > > > > > be null.
> > > > > > >
> > > > > > > D.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav D.
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Vyacheslav D.
> > >
> > >
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
>


Re: Apache Ignite 2.7. Last Mile

2018-11-08 Thread Nikolay Izhikov
I'm OK with this.

чт, 8 нояб. 2018 г., 13:44 Andrey Gura ag...@apache.org:

> Long, long way to release :)
>
> Guys, we have a breaking change in Ignite 2.7 so we must add
> IGNITE-10154 [1] fix to the release.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10154
> On Tue, Nov 6, 2018 at 6:30 PM Igor Sapego  wrote:
> >
> > Guys,
> >
> > I've found the following issue: [1]. It is quite local (only affects
> > Ignite C++ Linux build system) but quite critical too. I think it
> > should be included in 2.7.
> >
> > What do you think?
> >
> > [1] - https://issues.apache.org/jira/browse/IGNITE-10147
> >
> > Best Regards,
> > Igor
> >
> >
> > On Tue, Oct 30, 2018 at 4:35 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> > wrote:
> >
> > > Hello Nikolay, Igniters,
> > >
> > > It seems that we lost the following commit that should be included in
> > > 'ignite-2.7' branch
> > > (It looks like the change was not accidentally cherry-picked from
> 'master'
> > > to 'ignite-2.7')
> > >  -
> > >
> > >
> https://github.com/apache/ignite/commit/6e0ff06f8e309657a16c94da605348d9c3b804ad
> > >
> > > The most important part is the change introduced into
> GridDhtAtomicCache,
> > > the fix prevents NullPointerException during cache updates under some
> > > circumstances.
> > > So, I propose including the fix into ignite-2.7, at least the change of
> > > GridDhtAtomicCache.
> > >
> > > Thanks,
> > > Slava.
> > >
> > > пн, 29 окт. 2018 г. в 11:20, Nikolay Izhikov :
> > >
> > > > Hello, guys.
> > > >
> > > > For today we have 11 tickets mapped to 2.7
> > > >
> > > > IGNITE-10010 Alexey Goncharuk Node halted if second node was stopped,
> > > then
> > > > cache destroyed, then second node returned
> > > > IGNITE-10015 Alexey Goncharuk Sporadic JVM crash due to restart nodes
> > > > IGNITE-10013 Unassigned Node restart may lead to NPE in
> > > > GridDhtPartitionsExchangeFuture
> > > > IGNITE-9928 Igor Seliverstov MVCC TX: Late affinity assignment
> support.
> > > > IGNITE-9985 Igor Seliverstov MVCC TX: fix backup mappings
> > > > IGNITE-10007 Sergey Kozlov Deactivation hangs if an open transaction
> > > exists
> > > > IGNITE-10004 Andrew Mashenkov Parse error leads to leave the
> transaction
> > > > IGNITE-10024 Ivan Pavlukhin MVCC TX: Stackoverflow during
> DhtEnlistFuture
> > > > mapping
> > > > IGNITE-9996 Alexey Goncharuk Investigate possible performance drop in
> > > FSYNC
> > > > mode for ignite-2.7 compared to ignite-2.6
> > > > IGNITE-9982 Ivan Pavlukhin SQLLine: can't run with option
> > > > --autoCommit=false or true
> > > > IGNITE-9828 Roman Kondakov MVCC: Continuous query failover.
> > > >
> > > >
> > > > пт, 26 окт. 2018 г. в 9:59, Vladimir Ozerov :
> > > >
> > > > > Hi Nikolay,
> > > > >
> > > > > I do not know. We need to investigate them first. These are all
> > > > > regressions, so decision about impact and urgency should be made
> > > > separately
> > > > > for every ticket.
> > > > >
> > > > > On Fri, Oct 26, 2018 at 9:57 AM Nikolay Izhikov <
> nizhi...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hello, Igniters.
> > > > > >
> > > > > > We have *9* tickets mapped to 2.7 today
> > > > > >
> > > > > > Vladimir, do you think 1 week delay will be enough to resolve all
> > > this
> > > > > > tickets?
> > > > > >
> > > > > > IGNITE-9828 Roman Kondakov  MVCC: Continuous query
> > > > failover.
> > > > > > IGNITE-9928 Igor SeliverstovMVCC TX: Late affinity
> > > > assignment
> > > > > > support.
> > > > > > IGNITE-9985 Igor SeliverstovMVCC TX: fix backup
> mappings
> > > > > > IGNITE-9982 Ivan Pavlukhin  SQLLine: can't run with
> > > option
> > > > > > --autoCommit=false or true
> > > > > > IGNITE-10010Unassigned  Node halted if table was
> > > > dropped
> > > > > > IGNITE-10013Unas

Re: IGNITE-2.7. New Features

2018-11-05 Thread Nikolay Izhikov
t; * Fixed situation when data may be returned from cache partitoins in LOST
> state even when PartitionLossPolicy doesn't permit it
> * Fixed "Caches have distinct sets of data nodes" during SQL JOIN query
> execution between REPLICATED and PARTITIONED caches
> * Fixed wrong result for sql queries when item size exceeds page size
> * Fixed error during SQL query from client node with local flag set to
> "true"
> * Fixed handling UUID as column type
>
> JDBC:
> * Implemented DataSource interface for thin driver
>
> ODBC:
> * Added streaming mode support
> * Fixed crash in Linux when there is more than 1023 open file descriptors
> * Fixed bug that prevented cursors on server from being closed
> * Fixed segmentation fault when reusing a closed connection
>
> Web Console:
> * Added new metrics: WAL and Data size on disk
> * Added support for "collocated" query mode on Query screen
> * Added support for Java 9+ for Web Agent.
> * Added ability to show/hide password field value
> * Implemented execution of selected part of SQL query
> * Implemented explain of selected part of SQL query
> * Implemented connection to secured cluster
> * Implemented responsive full screen layout
> * Split "Sign In" page to three separate pages
> * UI updated to modern look and feel
> * Improved backend stability
> * Fixed fail when working with web sockets
>
> REST:
> * Added option IGNITE_REST_GETALL_AS_ARRAY for array format in "getAll"
> call
>
> Visor:
> * Added output of node "Consistent ID"
> * Visor now collects information about cache groups instead of separate
> caches to reduce memory consumption
> * Improved help for "start" command
> * Fixed output of cache metrics
>
> Control utility:
> * Added information about transaction start time
> * Added command to collect information about distribution of partitions
> * Added command to reset lost partitions
> * Added support for empty label (control.sh --tx label null)
> * Added atomicity mode to utility output.
> * Added orphaned local and remote transactions and ability to rollback them
> * Added "--dump" flag to dump current partition state to file.
> * Renamed command argument '--force' to '--yes'
> * Removed "initOrder" and "loc keys" from info
> * Fixed control utility hanging when connected to a joining node with PME
>
> ML:
> * Added TensorFlow integration
> * Added Estimator API support to TensorFlow cluster on top of Apache Ignite
> * Added ANN algorithm based on ACD concept
> * Added Random Forest algorithm
> * Added OneHotEncoder for categorical features
> * Added model estimation
> * Added K-fold cross validation for ML models
> * Added splitter for splitting dataset on test and train subsets
> * Added ability of filtering data during datasets creation
> * Added encoding categorical features with One-of-K Encoder
> * Added MinMax scaler preprocessor
> * Added gradient boosting for trees
> * Added indexing for decision trees
> * Added GDB convergence by error support
> * Added ability to build pipeline of data preprocessing and model training
> * Added ability to start and maintain TensorFlow cluster on top of Apache
> Ignite
> * Added support of Multi-Class for Logistic Regression
> * Implemented distributed binary logistic regression
>
> De[endency updates:
> * Apache Camel updated to 2.22.0
> * Apache Commons Beanutils updated to 1.9.3
> * Apache Hadoop Yarn updated to 2.7.7
> * Apache Kafka updated to 1.1.0
> * Apache Lucene updated to 7.4.0
> * Apache Mesos updated to 1.5.0
> * Apache Tomcat updated to 9.0.10
> * Apache Zookeeper updated to 3.4.13
> * Guava updated to 25.1-jre
> * Jackson Databind updated to 2.9.6
> * Jackson 1 usages replaced with Jackson 2
> * JCraft updated to 0.1.54
> * H2 version updated to 1.4.197
> * Log4j 2.x updated to 2.11.0
> * Netty updated to 4.1.27.Final
> * RocketMQ updated to 4.3.0
> * Scala 2.10.x was updated to 2.10.7
> * Scala 2.11.x updated to 2.11.12
>
> On Fri, Nov 2, 2018 at 6:47 PM David Harvey  wrote:
>
> > IGNITE-9365 Force backups to different AWS availability zones using only
> > Spring XML
> >
> > This deserves documentation enhancements:
> > https://apacheignite.readme.io/docs/affinity-collocation
> >
> > What is the mechanism for proposing an update for 2.7?
> >
> >
> > On Fri, Nov 2, 2018 at 6:31 AM Andrey Kuznetsov 
> wrote:
> >
> > > Great news!
> > >
> > > Future release is about to contain mission critical Ignite workers
> > liveness
> > > monitoring, introduced in IGNITE-6587.
> > >
> > >
> > > пт, 2 нояб. 2018 г. в 13:23, Nikolay Izhikov :
> > >
> > > > Hello, Guys.
> > > >
> > > > Good news! We have 2 final tickets for 2.7.
> > > > So release date is very near!
> > > >
> > > > Let's collect new features and improvements of Ignite 2.7 and
> includes
> > it
> > > > to release notes and other documents.
> > > >
> > > > Can you answer and describe your contributions?
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > >   Andrey Kuznetsov.
> > >
> >
>


IGNITE-2.7. New Features

2018-11-02 Thread Nikolay Izhikov
Hello, Guys.

Good news! We have 2 final tickets for 2.7.
So release date is very near!

Let's collect new features and improvements of Ignite 2.7 and includes it to 
release notes and other documents.

Can you answer and describe your contributions?


signature.asc
Description: This is a digitally signed message part


Re: How to run TDE tests locally?

2018-10-29 Thread Nikolay Izhikov
Hello, Pavel.

Found your environment in the end of the mail.

> My environment: JDK 1.8.0_151-b12, Ubuntu 17.10

Seems, you should enable JCE [1], [2] or update your JDK.
Actually, I don't executed any manual steps on my local environment.

host:~/src/ignite:[master]$ java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
host:~/src/ignite:[master]$ uname -a
Linux Host 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux

[1] https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html, 
section "New Security property to control crypto policy"
[2] 
https://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

В Пн, 29/10/2018 в 15:45 +0300, Pavel Kovalenko пишет:
> Nikolay,
> 
> JDK 1.8.0_151-b12
> 
> пн, 29 окт. 2018 г. в 15:31, Nikolay Izhikov :
> 
> > Hello, Pavel.
> > 
> > What jdk version do you use?
> > 
> > В Пн, 29/10/2018 в 15:30 +0300, Pavel Kovalenko пишет:
> > > Hello Igniters,
> > > 
> > > I have a problem with running TDE tests on my local machine.
> > > 
> > > When I start e.g. EncryptedCacheDestroyTest I get following exception:
> > > 
> > > > javax.cache.CacheException: class
> > > > > org.apache.ignite.IgniteCheckedException: Illegal key size
> > > > 
> > > > 
> > > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.IgniteKernal.createCache(IgniteKernal.java:2942)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.encryption.AbstractEncryptionTest.createEncryptedCache(AbstractEncryptionTest.java:173)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.encryption.AbstractEncryptionTest.createEncryptedCache(AbstractEncryptionTest.java:162)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.encryption.EncryptedCacheDestroyTest.testEncryptedCacheFromGroupDestroy(EncryptedCacheDestroyTest.java:72)
> > > > 
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > 
> > > > at
> > > > > 
> > 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > 
> > > > at
> > > > > 
> > 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > 
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > 
> > > > at junit.framework.TestCase.runTest(TestCase.java:176)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2206)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:144)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2121)
> > > > 
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 
> > > > Caused by: class org.apache.ignite.IgniteCheckedException: Illegal key
> > 
> > size
> > > > 
> > > > at
> > 
> > org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7431)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.util.future.GridFinishedFuture.get(GridFinishedFuture.java:102)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.internal.IgniteKernal.createCache(IgniteKernal.java:2937)
> > > > 
> > > > ... 12 more
> > > > 
> > > > Caused by: class org.apache.ignite.spi.IgniteSpiException: Illegal key
> > 
> > size
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.doEncryption(KeystoreEncryptionSpi.java:288)
> > > > 
> > > > at
> > > > > 
> > 
> > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.encr

Re: How to run TDE tests locally?

2018-10-29 Thread Nikolay Izhikov
Hello, Pavel.

What jdk version do you use?

В Пн, 29/10/2018 в 15:30 +0300, Pavel Kovalenko пишет:
> Hello Igniters,
> 
> I have a problem with running TDE tests on my local machine.
> 
> When I start e.g. EncryptedCacheDestroyTest I get following exception:
> 
> > javax.cache.CacheException: class
> > > org.apache.ignite.IgniteCheckedException: Illegal key size
> > 
> > 
> > > at
> > > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> > 
> > at
> > > org.apache.ignite.internal.IgniteKernal.createCache(IgniteKernal.java:2942)
> > 
> > at
> > > org.apache.ignite.internal.encryption.AbstractEncryptionTest.createEncryptedCache(AbstractEncryptionTest.java:173)
> > 
> > at
> > > org.apache.ignite.internal.encryption.AbstractEncryptionTest.createEncryptedCache(AbstractEncryptionTest.java:162)
> > 
> > at
> > > org.apache.ignite.internal.encryption.EncryptedCacheDestroyTest.testEncryptedCacheFromGroupDestroy(EncryptedCacheDestroyTest.java:72)
> > 
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 
> > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > 
> > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > 
> > at junit.framework.TestCase.runTest(TestCase.java:176)
> > 
> > at
> > > org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2206)
> > 
> > at
> > > org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:144)
> > 
> > at
> > > org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2121)
> > 
> > at java.lang.Thread.run(Thread.java:748)
> > 
> > Caused by: class org.apache.ignite.IgniteCheckedException: Illegal key size
> > 
> > at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7431)
> > 
> > at
> > > org.apache.ignite.internal.util.future.GridFinishedFuture.get(GridFinishedFuture.java:102)
> > 
> > at
> > > org.apache.ignite.internal.IgniteKernal.createCache(IgniteKernal.java:2937)
> > 
> > ... 12 more
> > 
> > Caused by: class org.apache.ignite.spi.IgniteSpiException: Illegal key size
> > 
> > at
> > > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.doEncryption(KeystoreEncryptionSpi.java:288)
> > 
> > at
> > > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.encrypt(KeystoreEncryptionSpi.java:211)
> > 
> > at
> > > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.encryptKey(KeystoreEncryptionSpi.java:300)
> > 
> > at
> > > org.apache.ignite.internal.managers.encryption.GridEncryptionManager.createKeys(GridEncryptionManager.java:753)
> > 
> > at
> > > org.apache.ignite.internal.managers.encryption.GridEncryptionManager.generateKeys(GridEncryptionManager.java:608)
> > 
> > at
> > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.generateEncryptionKeysAndStartCacheAfter(GridCacheProcessor.java:3650)
> > 
> > at
> > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3632)
> > 
> > at
> > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3543)
> > 
> > at
> > > org.apache.ignite.internal.IgniteKernal.createCache(IgniteKernal.java:2932)
> > 
> > ... 12 more
> > 
> > Caused by: java.security.InvalidKeyException: Illegal key size
> > 
> > at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
> > 
> > at javax.crypto.Cipher.init(Cipher.java:1393)
> > 
> > at javax.crypto.Cipher.init(Cipher.java:1327)
> > 
> > at
> > > org.apache.ignite.spi.encryption.keystore.KeystoreEncryptionSpi.doEncryption(KeystoreEncryptionSpi.java:282)
> > 
> > ... 20 more
> > 
> > 
> > > 
> 
> What I should do to avoid the exception above?
> 
> My environment: JDK 1.8.0_151-b12, Ubuntu 17.10


signature.asc
Description: This is a digitally signed message part


Re: [DISCUSSION] TDE. Phase-2. Master key rotation.

2018-10-29 Thread Nikolay Izhikov
Hello, Igniters.

We had private discussion of this design with Anton Vinogradov, Vladimir 
Ozerov, Alexey Goncharyuk.
Design need to be improved in the following ways:

1. Clear recovery logic should be written.

2. Clear process of atomic cache keys and master key change should be written.

3. We should design master key rotation process in the way it would be easy to 
implement EncryptionSPI for some enterprise encryption provider(AWS Key 
Management Service as an example)
   Which aspect of EncryptionSPI should be improved to make this process 
obvious.


В Ср, 24/10/2018 в 17:19 +0300, Nikolay Izhikov пишет:
> Hello.
> 
> Deisgn updated [1]
> 
> Please, share your feedback
> 
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
> 
> 
> В Вт, 23/10/2018 в 21:49 +0300, Nikolay Izhikov пишет:
> > Hello, Anton.
> > 
> > Thank you for your very usefull feedback!
> > 
> > I accept your proposals.
> > Seems it makes this feature more robust and clear.
> > 
> > Will update design in confluence in a couple of hours.
> > 
> > В Вт, 23/10/2018 в 19:18 +0300, Anton Vinogradov пишет:
> > > Nikolay,
> > > 
> > > I have some comments.
> > > 
> > > 1) Master key setup and removal is a responsibility of system 
> > > administrator.
> > > No matter how he will set a new master key or remove an old.
> > > The only need it to have both keys, new and old, installed before starting
> > > the rotation process.
> > > 
> > > 2) Master Key rotation is a process of cache's keys re-encryption.
> > > 
> > > So, we should send a message contains a new master key id only.
> > > We also have to check that "Master Key rotation" allowed to the user by
> > > checking it has SecurityPermission#ADMIN_OPS permission.
> > > 
> > > During this message handling, we have to re-encrypt keys and to set new
> > > master key id.
> > > 
> > > 3) We should provide recovery mode for nodes unexpectedly leaved cluster
> > > during "Master Key rotation" process.
> > > We have to have a special "node start" command allows to change node's
> > > master key before joining the cluster.
> > > 
> > > пн, 22 окт. 2018 г. в 22:39, Nikolay Izhikov :
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > As you may know, we successfully implement TDE. Phase-1 feature. [1]
> > > > This improvement allows users to use an encrypted cache.
> > > > 
> > > > To make TDE production ready I propose to extend it with two things:
> > > > 
> > > > * Master key rotation.
> > > > * Cache key rotation.
> > > > 
> > > > Such features required by many security standards such as PCI DSS [2] 
> > > > and
> > > > GDPR [3]
> > > > 
> > > > I think it would be easier to discuss, implement and review both 
> > > > features
> > > > separately.
> > > > So my plan is the following:
> > > > 
> > > > * TDE. Phase-2 - Master key rotation [4]
> > > > * TDE. Phase-3 - Cache key rotation. [5]
> > > > 
> > > > I prepared designs for both parts.
> > > > I want to specifically discuss Phase-2 design.
> > > > Phase-3 design state is [EARLY DRAFT].
> > > > I propose to use Phase-3 design as a reference to make sure we have a
> > > > consistent view of all aspects of TDE
> > > > and can be implemented without significant changes in earlier parts.
> > > > 
> > > > Below, my design.
> > > > Following changes will be made in confluence [4].
> > > > Please, share your feedback.
> > > > 
> > > > *TDE. PHASE-2. MASTER KEY ROTATION*
> > > > Key rotation required in case of it compromising or at the end 
> > > > of
> > > > crypto period(key validity period).
> > > > 
> > > > Goal:
> > > > To implement the ability to rotate master encryption key.
> > > > 
> > > > New processes:
> > > > 1. Master key rotation.
> > > > 2. Removal of a master key.
> > > > 
> > > > New administrator commands:
> > > > 1. Master keys view: node -> master key hash
> > > > 2. Cache group keys view: node -> group name -> encryption key
> > > > hash
> > &

Re: Apache Ignite 2.7. Last Mile

2018-10-29 Thread Nikolay Izhikov
Hello, guys.

For today we have 11 tickets mapped to 2.7

IGNITE-10010 Alexey Goncharuk Node halted if second node was stopped, then
cache destroyed, then second node returned
IGNITE-10015 Alexey Goncharuk Sporadic JVM crash due to restart nodes
IGNITE-10013 Unassigned Node restart may lead to NPE in
GridDhtPartitionsExchangeFuture
IGNITE-9928 Igor Seliverstov MVCC TX: Late affinity assignment support.
IGNITE-9985 Igor Seliverstov MVCC TX: fix backup mappings
IGNITE-10007 Sergey Kozlov Deactivation hangs if an open transaction exists
IGNITE-10004 Andrew Mashenkov Parse error leads to leave the transaction
IGNITE-10024 Ivan Pavlukhin MVCC TX: Stackoverflow during DhtEnlistFuture
mapping
IGNITE-9996 Alexey Goncharuk Investigate possible performance drop in FSYNC
mode for ignite-2.7 compared to ignite-2.6
IGNITE-9982 Ivan Pavlukhin SQLLine: can't run with option
--autoCommit=false or true
IGNITE-9828 Roman Kondakov MVCC: Continuous query failover.


пт, 26 окт. 2018 г. в 9:59, Vladimir Ozerov :

> Hi Nikolay,
>
> I do not know. We need to investigate them first. These are all
> regressions, so decision about impact and urgency should be made separately
> for every ticket.
>
> On Fri, Oct 26, 2018 at 9:57 AM Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > We have *9* tickets mapped to 2.7 today
> >
> > Vladimir, do you think 1 week delay will be enough to resolve all this
> > tickets?
> >
> > IGNITE-9828 Roman Kondakov  MVCC: Continuous query failover.
> > IGNITE-9928 Igor SeliverstovMVCC TX: Late affinity assignment
> > support.
> > IGNITE-9985 Igor SeliverstovMVCC TX: fix backup mappings
> > IGNITE-9982 Ivan Pavlukhin  SQLLine: can't run with option
> > --autoCommit=false or true
> > IGNITE-10010Unassigned  Node halted if table was dropped
> > IGNITE-10013Unassigned  Node restart may lead to NPE in
> > GridDhtPartitionsExchangeFuture
> > IGNITE-9996 Unassigned  Investigate possible performance
> > drop in FSYNC mode for ignite-2.7 compared to ignite-2.6
> > IGNITE-10007Unassigned  Deactivation hangs if an open
> > transaction exists
> > IGNITE-10004Unassigned  Parse error leads to leave the
> > transaction
> >
> > В Ср, 24/10/2018 в 12:30 +0300, Nikolay Izhikov пишет:
> > > Hello, Igniters.
> > >
> > > We have 3 ticket mapped to 2.7 today:
> > >
> > > Igor Seliverstov - IGNITE-9892 - MVCC: Exchange hangs on mvcc
> > coordinator fail
> > > Roman Kondakov   - IGNITE-9828 - MVCC: Continuous query failover.
> > > Roman Kondakov   - IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
> > >
> > > В Вт, 23/10/2018 в 15:01 +0300, Nikolay Izhikov пишет:
> > > > Hello, Dmitriy.
> > > >
> > > > I'm OK with including this patch to 2.7.
> > > >
> > > > Can you ensure it completely fix the issue?
> > > > I left comment in ticket.
> > > >
> > > >
> >
> https://issues.apache.org/jira/browse/IGNITE-9854?focusedCommentId=16660516=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16660516
> > > >
> > > >
> > > > В Вт, 23/10/2018 в 13:10 +0300, Dmitriy Govorukhin пишет:
> > > > > Nikolay,
> > > > >
> > > > > I have an issue which I want to include in 2.7 release,
> > > > > https://issues.apache.org/jira/browse/IGNITE-9854
> > > > > It is a very small fix but very important, it protects us from NPE
> > in some
> > > > > race scenario.
> > > > > Changes already in master, but the issue still not resolve, need
> your
> > > > > approval for cherry-picking changes to ignite-2.7.
> > > > >
> > > > > On Tue, Oct 23, 2018 at 10:51 AM Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > There are still tickets in the scope as we continue finding new
> > issues
> > > > > > during QA. I propose the following plan: if there are still
> opened
> > issues
> > > > > > by Friday, then shift vote date for 1 week, to 2nd November. This
> > is needed
> > > > > > to ensure that product quality is sufficient. But if the backlog
> > is empty
> > > > > > by Friday, we can go ahead with vote.
> > > > > >
> > > > > > Thoughts?
> > > > > >
&

Re: Apache Ignite 2.7. Last Mile

2018-10-26 Thread Nikolay Izhikov
Hello, Igniters.

We have *9* tickets mapped to 2.7 today

Vladimir, do you think 1 week delay will be enough to resolve all this tickets?

IGNITE-9828 Roman Kondakov  MVCC: Continuous query failover.
IGNITE-9928 Igor SeliverstovMVCC TX: Late affinity assignment 
support.
IGNITE-9985 Igor SeliverstovMVCC TX: fix backup mappings
IGNITE-9982 Ivan Pavlukhin  SQLLine: can't run with option 
--autoCommit=false or true
IGNITE-10010Unassigned  Node halted if table was dropped
IGNITE-10013Unassigned  Node restart may lead to NPE in 
GridDhtPartitionsExchangeFuture
IGNITE-9996 Unassigned  Investigate possible performance drop 
in FSYNC mode for ignite-2.7 compared to ignite-2.6
IGNITE-10007Unassigned  Deactivation hangs if an open 
transaction exists
IGNITE-10004Unassigned  Parse error leads to leave the 
transaction

В Ср, 24/10/2018 в 12:30 +0300, Nikolay Izhikov пишет:
> Hello, Igniters.
> 
> We have 3 ticket mapped to 2.7 today:
> 
> Igor Seliverstov - IGNITE-9892 - MVCC: Exchange hangs on mvcc coordinator fail
> Roman Kondakov   - IGNITE-9828 - MVCC: Continuous query failover.
> Roman Kondakov   - IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
> 
> В Вт, 23/10/2018 в 15:01 +0300, Nikolay Izhikov пишет:
> > Hello, Dmitriy.
> > 
> > I'm OK with including this patch to 2.7.
> > 
> > Can you ensure it completely fix the issue?
> > I left comment in ticket.
> > 
> > https://issues.apache.org/jira/browse/IGNITE-9854?focusedCommentId=16660516=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16660516
> > 
> > 
> > В Вт, 23/10/2018 в 13:10 +0300, Dmitriy Govorukhin пишет:
> > > Nikolay,
> > > 
> > > I have an issue which I want to include in 2.7 release,
> > > https://issues.apache.org/jira/browse/IGNITE-9854
> > > It is a very small fix but very important, it protects us from NPE in some
> > > race scenario.
> > > Changes already in master, but the issue still not resolve, need your
> > > approval for cherry-picking changes to ignite-2.7.
> > > 
> > > On Tue, Oct 23, 2018 at 10:51 AM Vladimir Ozerov 
> > > wrote:
> > > 
> > > > Igniters,
> > > > 
> > > > There are still tickets in the scope as we continue finding new issues
> > > > during QA. I propose the following plan: if there are still opened 
> > > > issues
> > > > by Friday, then shift vote date for 1 week, to 2nd November. This is 
> > > > needed
> > > > to ensure that product quality is sufficient. But if the backlog is 
> > > > empty
> > > > by Friday, we can go ahead with vote.
> > > > 
> > > > Thoughts?
> > > > 
> > > > On Tue, Oct 23, 2018 at 10:41 AM Nikolay Izhikov 
> > > > wrote:
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > We have 7 tickets mapped to 2.7 today.
> > > > > 
> > > > > Igov Seliverstov  - IGNITE-9892 - MVCC: Exchange hangs on mvcc
> > > > 
> > > > coordinator
> > > > > fail
> > > > > Igor Seliverstov  - IGNITE-9911 -
> > > > > 
> > > > 
> > > > CacheMvccSelectForUpdateQueryAbstractTest#testSelectForUpdateAfterAbortedTx
> > > > > periodically hangs
> > > > > Roman Kondakov- IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
> > > > > Roman Kondakov- IGNITE-9663 - MVCC: Data node failure can cause TX
> > > > > hanging.
> > > > > Vladimir Ozerov   - IGNITE-9960 - SQL: Revert and reopen lazy flag
> > > > > optimization (IGNITE-9171)
> > > > > Pavel Petroshenko - IGNITE-9951 - thin php: Date data type cut nanos
> > > > > Peter Ivanov  - IGNITE-9953 - Dropping hadoop accelerator 
> > > > > downloads
> > > > > 
> > > > > 
> > > > > В Вс, 21/10/2018 в 11:23 +0300, Nikolay Izhikov пишет:
> > > > > > Hell, Denis.
> > > > > > 
> > > > > > I just filter all 2.7 tickets without documentation.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > 
> > > > https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(docum

Re: Code inspection

2018-10-26 Thread Nikolay Izhikov
Hello, Maxim.

Great job! Thank you!
Igniters, let's add this suite to the Run All!

It can help us improve code quality and check code style without human eyes.


В Пт, 26/10/2018 в 00:47 +0300, Maxim Muzafarov пишет:
> Igniters,
> 
> The new `Inspections: Core` suite [2] configured on TeamCity (pass
> successfully with - Inspections total: 0, errors: 0 ).
> The next rules are enabled for this suite:
>  - `Missorted modifiers`;
>  - `'size() == 0' replaceable with 'isEmpty()'`;
>  - `Add missing @Override annotation`;
>  - `Fix unused imports`;
> 
> Let's incule it to the `Run::All` group on TC, so we will check these rules
> automatically for each PR.
> Any objections?
> 
> 
> Talking about the details,
> 
> - the issue [1] with adding an inspections configuration for TC have PA
> status;
> - the new configuration ignite_inspections_teamcity.xml added to PR;
> - four rules which are already fixed in the master branch enabled in config;
> - the `Inspections:Core` suite configured to use the inspections
> configuration from the local branch;
> - the example `how to use inspections from the command line` added.
> 
> 
> Petr, Nikolay,
> 
> Thank you for your support!
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-9983
> [2]
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore_IgniteTests24Java8=pull%2F5059%2Fhead=buildTypeStatusDiv
> [3]
> https://issues.apache.org/jira/browse/IGNITE-9983?focusedCommentId=16662323=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16662323
> 
> On Tue, 23 Oct 2018 at 19:16 Nikolay Izhikov  wrote:
> 
> > Hello, Maxim.
> > 
> > +1 from me.
> > 
> > I vote to enable static inspections for the Ignite codebase.
> > Thank you for that contributions!
> > 
> > В Вт, 23/10/2018 в 19:09 +0300, Maxim Muzafarov пишет:
> > > Igniters,
> > > 
> > > I've fixed some issues according to the inspections.xml configuration:
> > >  - `Missorted modifiers`;
> > >  - `'size() == 0' replaceable with 'isEmpty()'`;
> > >  - `Add missing @Override annotation`;
> > > These one have `In progress` state:
> > >  - `Fix unused imports`;
> > >  - `Remove unnecessary @SuppressWarnings annotation`;
> > > 
> > > The list of issues related to the current Code Inspections changes can be
> > > found [1]
> > > with using label `inspections`. So, to move forward and not lose current
> > > changes I
> > > propose to:
> > >  - Create the new configuration idea\ignite_inspections_teamcity.xml (I
> > > will file a new issue for it);
> > >  - Tune `Inspections: Core` Suite to use this configuration profile (It
> > > will run with each PR);
> > >  - In the case with fixing a new inspection rule enable it this
> > 
> > inspection
> > > configuration.
> > > 
> > > This will allow us to move forward in small steps and at some point of
> > 
> > time
> > > in future we will switch
> > > this ignite_inspections_teamcity.xml with the
> > > default ignite_inspections.xml.
> > > 
> > > Thoughts?
> > > Pert Ivanov, will you help to tune `Inspections: Core` suite?
> > > 
> > > [1]
> > > 
> > 
> > https://issues.apache.org/jira/browse/IGNITE-9923?jql=project%20%3D%20Ignite%20AND%20labels%20%3D%20inspections
> > > 
> > > On Sat, 25 Aug 2018 at 00:54 Dmitriy Pavlov 
> > 
> > wrote:
> > > 
> > > > IntelliJ Idea shows missing @Override annotation on my installation.
> > 
> > Not
> > > > sure it comes from our inspection or not.
> > > > 
> > > > Anyway, count on me.
> > > > 
> > > > пт, 24 авг. 2018 г. в 9:25, Maxim Muzafarov :
> > > > 
> > > > > Folks,
> > > > > 
> > > > > I think we can make a small step further with Ignite Inspections.
> > > > > 
> > > > > I've created these tickets [1], [2] for myself according to
> > 
> > previously
> > > > > added
> > > > > `idea/ignite_inspections.xml` and I plan to complete them.
> > > > > 
> > > > > Who will help me with review and merge?
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9311 - Add missing
> > > > > @Override annotation
> > > > > [2] https://issues.apache.org/jira/browse/IGNITE-9312 - Remove
> > > > 
> > > > unnecessary
> > > > > @Su

Re: [DISCUSSION] TDE. Phase-2. Master key rotation.

2018-10-24 Thread Nikolay Izhikov
Hello.

Deisgn updated [1]

Please, share your feedback

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381


В Вт, 23/10/2018 в 21:49 +0300, Nikolay Izhikov пишет:
> Hello, Anton.
> 
> Thank you for your very usefull feedback!
> 
> I accept your proposals.
> Seems it makes this feature more robust and clear.
> 
> Will update design in confluence in a couple of hours.
> 
> В Вт, 23/10/2018 в 19:18 +0300, Anton Vinogradov пишет:
> > Nikolay,
> > 
> > I have some comments.
> > 
> > 1) Master key setup and removal is a responsibility of system administrator.
> > No matter how he will set a new master key or remove an old.
> > The only need it to have both keys, new and old, installed before starting
> > the rotation process.
> > 
> > 2) Master Key rotation is a process of cache's keys re-encryption.
> > 
> > So, we should send a message contains a new master key id only.
> > We also have to check that "Master Key rotation" allowed to the user by
> > checking it has SecurityPermission#ADMIN_OPS permission.
> > 
> > During this message handling, we have to re-encrypt keys and to set new
> > master key id.
> > 
> > 3) We should provide recovery mode for nodes unexpectedly leaved cluster
> > during "Master Key rotation" process.
> > We have to have a special "node start" command allows to change node's
> > master key before joining the cluster.
> > 
> > пн, 22 окт. 2018 г. в 22:39, Nikolay Izhikov :
> > 
> > > Hello, Igniters.
> > > 
> > > As you may know, we successfully implement TDE. Phase-1 feature. [1]
> > > This improvement allows users to use an encrypted cache.
> > > 
> > > To make TDE production ready I propose to extend it with two things:
> > > 
> > > * Master key rotation.
> > > * Cache key rotation.
> > > 
> > > Such features required by many security standards such as PCI DSS [2] and
> > > GDPR [3]
> > > 
> > > I think it would be easier to discuss, implement and review both features
> > > separately.
> > > So my plan is the following:
> > > 
> > > * TDE. Phase-2 - Master key rotation [4]
> > > * TDE. Phase-3 - Cache key rotation. [5]
> > > 
> > > I prepared designs for both parts.
> > > I want to specifically discuss Phase-2 design.
> > > Phase-3 design state is [EARLY DRAFT].
> > > I propose to use Phase-3 design as a reference to make sure we have a
> > > consistent view of all aspects of TDE
> > > and can be implemented without significant changes in earlier parts.
> > > 
> > > Below, my design.
> > > Following changes will be made in confluence [4].
> > > Please, share your feedback.
> > > 
> > > *TDE. PHASE-2. MASTER KEY ROTATION*
> > > Key rotation required in case of it compromising or at the end of
> > > crypto period(key validity period).
> > > 
> > > Goal:
> > > To implement the ability to rotate master encryption key.
> > > 
> > > New processes:
> > > 1. Master key rotation.
> > > 2. Removal of a master key.
> > > 
> > > New administrator commands:
> > > 1. Master keys view: node -> master key hash
> > > 2. Cache group keys view: node -> group name -> encryption key
> > > hash
> > > 
> > > MASTER KEY ROTATION:
> > > Process start:
> > > Administrator initiates key rotation via  some kind of
> > > user interface(CLI, Visor, Web Console, JMX, etc).
> > > 
> > > Process description:
> > > Message is sent by discovery.
> > > A Message should contain:
> > > * Master cache key encrypted with the current
> > > master key.
> > > 
> > > When server node processed message following actions are
> > > executed:
> > > * Blocks creation of encrypted cache key.
> > > * Encrypt cache group keys with new master key.
> > > * Unblock creation of encrypted cache key.
> > > 
> > > New joining node should also change the current master key
> > > with the new one.
> > > 
> > > Process completion:
> > > The administrator initiates process completion via the
&

Re: [DISCUSSION] Spark Data Frame through Thin Client

2018-10-24 Thread Nikolay Izhikov
Hello, Valentin.

> What I don't agree with is that replacing thick client with thin client is a 
> way to fix usability issues. 

I think it will fix some of them.

> will potentially compromise the performance

As I mentioned earlier, I want to provide easy way to play with integration.
For maximum performance one should use client nodes.

> What is the difference between thin and thick client from this point of view?

We need only 1 jar file.
All config options we need is list of ip addressed.

> I'm not arguing there are usability issues with thick client. 
> I'm just suggesting to fix those issues first, before we jump reworking the 
> implementation.

> My suggestion is to look at usability issues and try to fix them without 
> getting rid of thick client.

I agree, let's do it!
Can you create some tickets?
I'm ready to look at it and contribute a fix.

В Вт, 23/10/2018 в 19:31 -0700, Valentin Kulichenko пишет:
> Nikolay,
> 
> Please see my comments below. Actually, I haven't made most of the
> assumptions that you mentioned, and I generally agree with you. What I
> don't agree with is that replacing thick client with thin client is a way
> to fix usability issues. Thin client is not going to be issue-free either,
> but will potentially compromise the performance, as well as functionality
> (like streaming, as Stephen mentioned). My suggestion is to look at
> usability issues and try to fix them without getting rid of thick client.
> 
> -Val
> 
> On Sun, Oct 21, 2018 at 1:08 AM Nikolay Izhikov  wrote:
> 
> > Valentin.
> > 
> > Seems, You made several suggestions, which is not always true, from my
> > point of view:
> > 
> > 1. "We have access to Spark cluster installation to perform deployment
> > steps" - this is not true in cloud or enterprise environment.
> > 
> 
> Can you please elaborate on this? What is the difference between thin and
> thick client from this point of view? I understand that the latter would
> generally be more complicated, but how would one use thin client without
> deploying a JAR?
> 
> 
> > 
> > 2. "Spark cluster is used only for Ignite integration".
> > From what I know computational resources for big Spark cluster is divided
> > by many business divisions.
> > And it is not convenient to perform some deployment steps on this cluster.
> > 
> 
> Same as #1. Regardless how we use the Spark cluster, we need to deploy a
> JAR in case of thin client, no?
> 
> 
> > 
> > 3. "When Ignite + Spark are used in real production it's OK to have
> > reasonable deployment overhead"
> > What about developer who want to play with this integration?
> > And want to do it quickly to see how it works in real life examples.
> > Can we do his life much easier?
> > 
> 
> We can and we should :) I'm not arguing there are usability issues with
> thick client. I'm just suggesting to fix those issues first, before we jump
> reworking the implementation.
> 
> 
> > 
> > > First of all, they will exist with thin client either.
> > 
> > Spark have an ability to deploy jars on worker and add it to application
> > tasks classpath.
> > For 2.6 we must deploy 11 additional jars to start using Ignite.
> > Please, see my example on the bottom of documentation page [1]
> > 
> > Does cache-api-1.0.0.jar and h2-1.4.195.jar seems like obvious
> > dependencies for Ignite integration for you?
> > And for our users? :)
> > 
> 
> No, this is not obvious. Absolutely, this is a usability issue and we
> should think how to make user's life easier.
> 
> 
> > 
> > Actually, list of dependencies will be changed in 2.7 - new version of
> > jcache, new version of h2
> > So user should change it in code or perform additional deployment steps.
> > 
> > It overkill for me.
> > 
> > On the other hand - thin client requires only 1 jar.
> > Moreover, thin client protocol have the backward compatibility.
> > So thin client will perform correctly when Ignite cluster will be updated
> > from 2.6 to 2.7.
> > So, with Spark integration via thin client we will be able to update
> > Ignite cluster and Spark integration separately.
> > For now, we should do it in one big step.
> > 
> > What do you think?
> > 
> > [1] https://apacheignite-fs.readme.io/docs/installation-deployment
> > 
> > В Сб, 20/10/2018 в 18:33 -0700, Valentin Kulichenko пишет:
> > > Guys,
> > > 
> > > From my experience, Ignite and Spark clusters typically run in the same
> > > environment, which makes client node a more preferable option. 

Re: Apache Ignite 2.7. Last Mile

2018-10-24 Thread Nikolay Izhikov
Hello, Igniters.

We have 3 ticket mapped to 2.7 today:

Igor Seliverstov - IGNITE-9892 - MVCC: Exchange hangs on mvcc coordinator fail
Roman Kondakov   - IGNITE-9828 - MVCC: Continuous query failover.
Roman Kondakov   - IGNITE-9928 - MVCC TX: Bug in SQL query mapping.

В Вт, 23/10/2018 в 15:01 +0300, Nikolay Izhikov пишет:
> Hello, Dmitriy.
> 
> I'm OK with including this patch to 2.7.
> 
> Can you ensure it completely fix the issue?
> I left comment in ticket.
> 
> https://issues.apache.org/jira/browse/IGNITE-9854?focusedCommentId=16660516=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16660516
> 
> 
> В Вт, 23/10/2018 в 13:10 +0300, Dmitriy Govorukhin пишет:
> > Nikolay,
> > 
> > I have an issue which I want to include in 2.7 release,
> > https://issues.apache.org/jira/browse/IGNITE-9854
> > It is a very small fix but very important, it protects us from NPE in some
> > race scenario.
> > Changes already in master, but the issue still not resolve, need your
> > approval for cherry-picking changes to ignite-2.7.
> > 
> > On Tue, Oct 23, 2018 at 10:51 AM Vladimir Ozerov 
> > wrote:
> > 
> > > Igniters,
> > > 
> > > There are still tickets in the scope as we continue finding new issues
> > > during QA. I propose the following plan: if there are still opened issues
> > > by Friday, then shift vote date for 1 week, to 2nd November. This is 
> > > needed
> > > to ensure that product quality is sufficient. But if the backlog is empty
> > > by Friday, we can go ahead with vote.
> > > 
> > > Thoughts?
> > > 
> > > On Tue, Oct 23, 2018 at 10:41 AM Nikolay Izhikov 
> > > wrote:
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > We have 7 tickets mapped to 2.7 today.
> > > > 
> > > > Igov Seliverstov  - IGNITE-9892 - MVCC: Exchange hangs on mvcc
> > > 
> > > coordinator
> > > > fail
> > > > Igor Seliverstov  - IGNITE-9911 -
> > > > 
> > > 
> > > CacheMvccSelectForUpdateQueryAbstractTest#testSelectForUpdateAfterAbortedTx
> > > > periodically hangs
> > > > Roman Kondakov- IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
> > > > Roman Kondakov- IGNITE-9663 - MVCC: Data node failure can cause TX
> > > > hanging.
> > > > Vladimir Ozerov   - IGNITE-9960 - SQL: Revert and reopen lazy flag
> > > > optimization (IGNITE-9171)
> > > > Pavel Petroshenko - IGNITE-9951 - thin php: Date data type cut nanos
> > > > Peter Ivanov  - IGNITE-9953 - Dropping hadoop accelerator downloads
> > > > 
> > > > 
> > > > В Вс, 21/10/2018 в 11:23 +0300, Nikolay Izhikov пишет:
> > > > > Hell, Denis.
> > > > > 
> > > > > I just filter all 2.7 tickets without documentation.
> > > > > 
> > > > > 
> > > > > 
> > > 
> > > https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(documentation)))%20ORDER%20BY%20%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> > > > > 
> > > > > I've added this view to the release page.
> > > > > See sesction "Unresolved tickets(without documentation)".
> > > > > 
> > > > > В Сб, 20/10/2018 в 16:04 -0700, Denis Magda пишет:
> > > > > > Nikolay,
> > > > > > 
> > > > > > Where do you track those 8 blockers? Can't find them on this wiki
> > > 
> > > page:
> > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
> > > > > > 
> > > > > > --
> > > > > > Denis
> > > > > > 
> > > > > > On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org>
> > > > > > wrote:
> > > > > > 
> > > > > > > Hello, Denis.
> > > > > > > 
> > > > > > > As a first time release manager I'm trying to rely on Ignite
> > > 
> > > veterans
> > > > > > > opinion.
> > > > > > > Guys told me that we must fix all blockers and only after it make
> > > 
> > > the
> > > > > > > release.

Re: [DISCUSSION] TDE. Phase-2. Master key rotation.

2018-10-23 Thread Nikolay Izhikov
Hello, Anton.

Thank you for your very usefull feedback!

I accept your proposals.
Seems it makes this feature more robust and clear.

Will update design in confluence in a couple of hours.

В Вт, 23/10/2018 в 19:18 +0300, Anton Vinogradov пишет:
> Nikolay,
> 
> I have some comments.
> 
> 1) Master key setup and removal is a responsibility of system administrator.
> No matter how he will set a new master key or remove an old.
> The only need it to have both keys, new and old, installed before starting
> the rotation process.
> 
> 2) Master Key rotation is a process of cache's keys re-encryption.
> 
> So, we should send a message contains a new master key id only.
> We also have to check that "Master Key rotation" allowed to the user by
> checking it has SecurityPermission#ADMIN_OPS permission.
> 
> During this message handling, we have to re-encrypt keys and to set new
> master key id.
> 
> 3) We should provide recovery mode for nodes unexpectedly leaved cluster
> during "Master Key rotation" process.
> We have to have a special "node start" command allows to change node's
> master key before joining the cluster.
> 
> пн, 22 окт. 2018 г. в 22:39, Nikolay Izhikov :
> 
> > Hello, Igniters.
> > 
> > As you may know, we successfully implement TDE. Phase-1 feature. [1]
> > This improvement allows users to use an encrypted cache.
> > 
> > To make TDE production ready I propose to extend it with two things:
> > 
> > * Master key rotation.
> > * Cache key rotation.
> > 
> > Such features required by many security standards such as PCI DSS [2] and
> > GDPR [3]
> > 
> > I think it would be easier to discuss, implement and review both features
> > separately.
> > So my plan is the following:
> > 
> > * TDE. Phase-2 - Master key rotation [4]
> > * TDE. Phase-3 - Cache key rotation. [5]
> > 
> > I prepared designs for both parts.
> > I want to specifically discuss Phase-2 design.
> > Phase-3 design state is [EARLY DRAFT].
> > I propose to use Phase-3 design as a reference to make sure we have a
> > consistent view of all aspects of TDE
> > and can be implemented without significant changes in earlier parts.
> > 
> > Below, my design.
> > Following changes will be made in confluence [4].
> > Please, share your feedback.
> > 
> > *TDE. PHASE-2. MASTER KEY ROTATION*
> > Key rotation required in case of it compromising or at the end of
> > crypto period(key validity period).
> > 
> > Goal:
> > To implement the ability to rotate master encryption key.
> > 
> > New processes:
> > 1. Master key rotation.
> > 2. Removal of a master key.
> > 
> > New administrator commands:
> > 1. Master keys view: node -> master key hash
> > 2. Cache group keys view: node -> group name -> encryption key
> > hash
> > 
> > MASTER KEY ROTATION:
> > Process start:
> > Administrator initiates key rotation via  some kind of
> > user interface(CLI, Visor, Web Console, JMX, etc).
> > 
> > Process description:
> > Message is sent by discovery.
> > A Message should contain:
> > * Master cache key encrypted with the current
> > master key.
> > 
> > When server node processed message following actions are
> > executed:
> > * Blocks creation of encrypted cache key.
> > * Encrypt cache group keys with new master key.
> > * Unblock creation of encrypted cache key.
> > 
> > New joining node should also change the current master key
> > with the new one.
> > 
> > Process completion:
> > The administrator initiates process completion via the
> > interface by using “master key removal” command.
> > Design assumes an administrator will check that all nodes
> > successfully change master key and all required nodes are alive.
> > 
> > MASTER KEY REMOVAL:
> > Process start:
> > Administrator initiates process via some kind of user
> > interface(CLI, Visor, WebConsole, JMX, etc),
> > 
> > Process description:
> > Message is sent by discovery.
> > Message should contain:
> > * New master key hash.
> > When a server node processed message following actions

Re: Code inspection

2018-10-23 Thread Nikolay Izhikov
Hello, Maxim.

+1 from me.

I vote to enable static inspections for the Ignite codebase.
Thank you for that contributions!

В Вт, 23/10/2018 в 19:09 +0300, Maxim Muzafarov пишет:
> Igniters,
> 
> I've fixed some issues according to the inspections.xml configuration:
>  - `Missorted modifiers`;
>  - `'size() == 0' replaceable with 'isEmpty()'`;
>  - `Add missing @Override annotation`;
> These one have `In progress` state:
>  - `Fix unused imports`;
>  - `Remove unnecessary @SuppressWarnings annotation`;
> 
> The list of issues related to the current Code Inspections changes can be
> found [1]
> with using label `inspections`. So, to move forward and not lose current
> changes I
> propose to:
>  - Create the new configuration idea\ignite_inspections_teamcity.xml (I
> will file a new issue for it);
>  - Tune `Inspections: Core` Suite to use this configuration profile (It
> will run with each PR);
>  - In the case with fixing a new inspection rule enable it this inspection
> configuration.
> 
> This will allow us to move forward in small steps and at some point of time
> in future we will switch
> this ignite_inspections_teamcity.xml with the
> default ignite_inspections.xml.
> 
> Thoughts?
> Pert Ivanov, will you help to tune `Inspections: Core` suite?
> 
> [1]
> https://issues.apache.org/jira/browse/IGNITE-9923?jql=project%20%3D%20Ignite%20AND%20labels%20%3D%20inspections
> 
> On Sat, 25 Aug 2018 at 00:54 Dmitriy Pavlov  wrote:
> 
> > IntelliJ Idea shows missing @Override annotation on my installation. Not
> > sure it comes from our inspection or not.
> > 
> > Anyway, count on me.
> > 
> > пт, 24 авг. 2018 г. в 9:25, Maxim Muzafarov :
> > 
> > > Folks,
> > > 
> > > I think we can make a small step further with Ignite Inspections.
> > > 
> > > I've created these tickets [1], [2] for myself according to previously
> > > added
> > > `idea/ignite_inspections.xml` and I plan to complete them.
> > > 
> > > Who will help me with review and merge?
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9311 - Add missing
> > > @Override annotation
> > > [2] https://issues.apache.org/jira/browse/IGNITE-9312 - Remove
> > 
> > unnecessary
> > > @SuppressWarnings annotation
> > > 
> > > On Thu, 16 Aug 2018 at 19:53 Dmitriy Pavlov 
> > 
> > wrote:
> > > 
> > > > Hi Pavel,
> > > > 
> > > > Thank you for noticing and bringing it here. I've fixed TC failure.
> > > > 
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > > 
> > > > чт, 16 авг. 2018 г. в 0:10, Pavel Pereslegin :
> > > > 
> > > > > Hello Igniters.
> > > > > 
> > > > > It seems that "idea/ignite_inspections.xml" should be excluded from
> > > > > "check-licenses" maven profile, because "_Licenses Headers_"
> > > > > configuration always fails now [1] on TeamCity.
> > > > > 
> > > > > [1]
> > > > > 
> > 
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_LicensesHeaders=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E
> > > > > ср, 15 авг. 2018 г. в 20:49, Dmitriy Pavlov :
> > > > > > 
> > > > > > I've updated wiki page
> > > > > > 
> > 
> > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-C.CodeInspection
> > > > > > with
> > > > > > reference to settings.xml placement in the project.
> > > > > > 
> > > > > > It is only advice, so I hope you don't mind having this reference.
> > > > > > 
> > > > > > ср, 15 авг. 2018 г. в 16:45, Dmitriy Pavlov  > > 
> > > :
> > > > > > 
> > > > > > > Hi Maxim,
> > > > > > > 
> > > > > > > Thank you for stepping in. I've committed the first version here
> > > > > > > 'idea/ignite_inspections.xml'. We can move it to project default
> > > > 
> > > > later
> > > > > when
> > > > > > > all inspection problems are fixed.
> > > > > > > Commit:
> > > > > > > 
> > > > > > > 
> > 
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3e0f04edf7cc0aa1631fbd1b9af1e9b87b697eb1
> > > > > > > 
> > > > > > > 
> > > > > > > Igniters,
> > > > > > > 
> > > > > > > you can enable this profile using the following steps:
> > 
> > Inspections
> > > > > > > (icon)->Configure inspections->(settings button)->Import
> > > > > 
> > > > > Profile->select
> > > > > > > file and import.
> > > > > > > 
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > > 
> > > > > > > вт, 14 авг. 2018 г. в 16:31, Maxim Muzafarov  > > 
> > > :
> > > > > > > 
> > > > > > > > Dmitry and other Igniters,
> > > > > > > > 
> > > > > > > > Previously you has suggested to commit `Code Inspections` into
> > > > 
> > > > Ignite
> > > > > > > > codebase.
> > > > > > > > It makes sense for me. I think it's the easiest way to share
> > 
> > this
> > > > > profile
> > > > > > > > among community
> > > > > > > > members and this inspection can be set as for the project level.
> > > > > > > > So, I suggest:
> > > > > > > > 
> > > > > > > > 1) According to Jetbrains documentation [1] the inspection
> > 
> > profile
> > > > > can be
> > > > > > > > placed to
> > > > > > > > 

Re: Apache Ignite 2.7. Last Mile

2018-10-23 Thread Nikolay Izhikov
Hello, Dmitriy.

I'm OK with including this patch to 2.7.

Can you ensure it completely fix the issue?
I left comment in ticket.

https://issues.apache.org/jira/browse/IGNITE-9854?focusedCommentId=16660516=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16660516


В Вт, 23/10/2018 в 13:10 +0300, Dmitriy Govorukhin пишет:
> Nikolay,
> 
> I have an issue which I want to include in 2.7 release,
> https://issues.apache.org/jira/browse/IGNITE-9854
> It is a very small fix but very important, it protects us from NPE in some
> race scenario.
> Changes already in master, but the issue still not resolve, need your
> approval for cherry-picking changes to ignite-2.7.
> 
> On Tue, Oct 23, 2018 at 10:51 AM Vladimir Ozerov 
> wrote:
> 
> > Igniters,
> > 
> > There are still tickets in the scope as we continue finding new issues
> > during QA. I propose the following plan: if there are still opened issues
> > by Friday, then shift vote date for 1 week, to 2nd November. This is needed
> > to ensure that product quality is sufficient. But if the backlog is empty
> > by Friday, we can go ahead with vote.
> > 
> > Thoughts?
> > 
> > On Tue, Oct 23, 2018 at 10:41 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > We have 7 tickets mapped to 2.7 today.
> > > 
> > > Igov Seliverstov  - IGNITE-9892 - MVCC: Exchange hangs on mvcc
> > 
> > coordinator
> > > fail
> > > Igor Seliverstov  - IGNITE-9911 -
> > > 
> > 
> > CacheMvccSelectForUpdateQueryAbstractTest#testSelectForUpdateAfterAbortedTx
> > > periodically hangs
> > > Roman Kondakov- IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
> > > Roman Kondakov- IGNITE-9663 - MVCC: Data node failure can cause TX
> > > hanging.
> > > Vladimir Ozerov   - IGNITE-9960 - SQL: Revert and reopen lazy flag
> > > optimization (IGNITE-9171)
> > > Pavel Petroshenko - IGNITE-9951 - thin php: Date data type cut nanos
> > > Peter Ivanov  - IGNITE-9953 - Dropping hadoop accelerator downloads
> > > 
> > > 
> > > В Вс, 21/10/2018 в 11:23 +0300, Nikolay Izhikov пишет:
> > > > Hell, Denis.
> > > > 
> > > > I just filter all 2.7 tickets without documentation.
> > > > 
> > > > 
> > > > 
> > 
> > https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(documentation)))%20ORDER%20BY%20%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> > > > 
> > > > I've added this view to the release page.
> > > > See sesction "Unresolved tickets(without documentation)".
> > > > 
> > > > В Сб, 20/10/2018 в 16:04 -0700, Denis Magda пишет:
> > > > > Nikolay,
> > > > > 
> > > > > Where do you track those 8 blockers? Can't find them on this wiki
> > 
> > page:
> > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
> > > > > 
> > > > > --
> > > > > Denis
> > > > > 
> > > > > On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov <
> > 
> > nizhi...@apache.org>
> > > > > wrote:
> > > > > 
> > > > > > Hello, Denis.
> > > > > > 
> > > > > > As a first time release manager I'm trying to rely on Ignite
> > 
> > veterans
> > > > > > opinion.
> > > > > > Guys told me that we must fix all blockers and only after it make
> > 
> > the
> > > > > > release.
> > > > > > 
> > > > > > Let's fix them all.
> > > > > > What do you think?
> > > > > > 
> > > > > > > When are we sending a release candidate for vote?
> > > > > > 
> > > > > > When all blocker bugs will be fixed.
> > > > > > Currently, we have *8*.
> > > > > > 
> > > > > > В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> > > > > > > Guys, as a side observer of the current release, this all looks
> > > 
> > > like a
> > > > > > > never ending story :)
> > > > > > > 
> > > > > > > When are we sending a release candidate for vo

Re: Apache Ignite 2.7. Last Mile

2018-10-23 Thread Nikolay Izhikov
Hello, Igniters.

We have 7 tickets mapped to 2.7 today.

Igov Seliverstov  - IGNITE-9892 - MVCC: Exchange hangs on mvcc coordinator fail
Igor Seliverstov  - IGNITE-9911 - 
CacheMvccSelectForUpdateQueryAbstractTest#testSelectForUpdateAfterAbortedTx 
periodically hangs
Roman Kondakov- IGNITE-9928 - MVCC TX: Bug in SQL query mapping.
Roman Kondakov- IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
Vladimir Ozerov   - IGNITE-9960 - SQL: Revert and reopen lazy flag optimization 
(IGNITE-9171)
Pavel Petroshenko - IGNITE-9951 - thin php: Date data type cut nanos
Peter Ivanov  - IGNITE-9953 - Dropping hadoop accelerator downloads


В Вс, 21/10/2018 в 11:23 +0300, Nikolay Izhikov пишет:
> Hell, Denis.
> 
> I just filter all 2.7 tickets without documentation.
> 
> 
> https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(documentation)))%20ORDER%20BY%20%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> 
> I've added this view to the release page.
> See sesction "Unresolved tickets(without documentation)".
> 
> В Сб, 20/10/2018 в 16:04 -0700, Denis Magda пишет:
> > Nikolay,
> > 
> > Where do you track those 8 blockers? Can't find them on this wiki page:
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
> > 
> > --
> > Denis
> > 
> > On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Denis.
> > > 
> > > As a first time release manager I'm trying to rely on Ignite veterans
> > > opinion.
> > > Guys told me that we must fix all blockers and only after it make the
> > > release.
> > > 
> > > Let's fix them all.
> > > What do you think?
> > > 
> > > > When are we sending a release candidate for vote?
> > > 
> > > When all blocker bugs will be fixed.
> > > Currently, we have *8*.
> > > 
> > > В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> > > > Guys, as a side observer of the current release, this all looks like a
> > > > never ending story :)
> > > > 
> > > > When are we sending a release candidate for vote?
> > > > 
> > > > --
> > > > Denis
> > > > 
> > > > On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov 
> > > 
> > > wrote:
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > We have 6 tickets for 2.7
> > > > > 
> > > > > Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> > > > > Igor Seliverstov - IGNITE-9911
> > > > > Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
> > > > > 
> > > > > В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > > > > > I have got one more potential 2.7 blocker [1] with straightforward
> > > 
> > > fix. I
> > > > > > beleive it will not break any production use case, but it leads to
> > > 
> > > test
> > > > > > suite hang, thus affecting other urgent issues.
> > > > > > 
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> > > > > > 
> > > > > > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> > > > > > 
> > > > > > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> > > > > 
> > > > > quite
> > > > > > > important.
> > > > > > > 
> > > > > > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > > > > > > 
> > > > > > > > JFYI
> > > > > > > > 
> > > > > > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > > > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn <
> > > 
> > > ptupit...@apache.org
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > Thank you. Fix has been merged to master and cherry-picked to
> > > > > > > 
> > > > > > > ignite-2.7.
> > > > > > > > > 
> > > > > > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> > > > >

[DISCUSSION] TDE. Phase-2. Master key rotation.

2018-10-22 Thread Nikolay Izhikov
Hello, Igniters.

As you may know, we successfully implement TDE. Phase-1 feature. [1]
This improvement allows users to use an encrypted cache.

To make TDE production ready I propose to extend it with two things:

* Master key rotation.
* Cache key rotation.

Such features required by many security standards such as PCI DSS [2] and GDPR 
[3]

I think it would be easier to discuss, implement and review both features 
separately.
So my plan is the following:

* TDE. Phase-2 - Master key rotation [4]
* TDE. Phase-3 - Cache key rotation. [5]

I prepared designs for both parts.
I want to specifically discuss Phase-2 design. 
Phase-3 design state is [EARLY DRAFT].
I propose to use Phase-3 design as a reference to make sure we have a 
consistent view of all aspects of TDE
and can be implemented without significant changes in earlier parts.

Below, my design.
Following changes will be made in confluence [4].
Please, share your feedback.

*TDE. PHASE-2. MASTER KEY ROTATION*
Key rotation required in case of it compromising or at the end of 
crypto period(key validity period). 

Goal:
To implement the ability to rotate master encryption key. 

New processes: 
1. Master key rotation.
2. Removal of a master key.  

New administrator commands: 
1. Master keys view: node -> master key hash 
2. Cache group keys view: node -> group name -> encryption key hash 

MASTER KEY ROTATION: 
Process start: 
Administrator initiates key rotation via  some kind of user 
interface(CLI, Visor, Web Console, JMX, etc). 

Process description: 
Message is sent by discovery. 
A Message should contain: 
* Master cache key encrypted with the current master 
key. 

When server node processed message following actions are 
executed: 
* Blocks creation of encrypted cache key. 
* Encrypt cache group keys with new master key. 
* Unblock creation of encrypted cache key. 

New joining node should also change the current master key with 
the new one. 

Process completion: 
The administrator initiates process completion via the 
interface by using “master key removal” command. 
Design assumes an administrator will check that all nodes 
successfully change master key and all required nodes are alive. 

MASTER KEY REMOVAL:
Process start: 
Administrator initiates process via some kind of user 
interface(CLI, Visor, WebConsole, JMX, etc), 

Process description: 
Message is sent by discovery. 
Message should contain: 
* New master key hash. 
When a server node processed message following actions are 
executed: 
Received master key hash compared with known master key hash. 
Previous master key removed using configured EncryptionSPI. 

NEW COMMANDS:
Master key hashes. 
Input: nothing 
Output: List of Tuples3 
* Node ID 
* Current key hash 
* Previous key hash or null. 
Cache key hashes. 
Input: cache id. 
Output: List of Tuples3 
* Node ID 
* Current key hash 
* Previous key hash or null. 

[1] https://issues.apache.org/jira/browse/IGNITE-8260
[2] https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf
[3] https://gdpr-info.eu/
[4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
[5] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384


signature.asc
Description: This is a digitally signed message part


Re: [DISCUSSION] Spark Data Frame through Thin Client

2018-10-22 Thread Nikolay Izhikov
Hello, Stephen.

I suggest thin client deployment as a second option together with existing 
integration that use Client Node.

> I’m thinking specifically about better support for Spark Streaming, where the 
> lack  of continuous query support in thin clients removes a significant 
> optimisation option. 

It's very interesting.
Can you share you thoughts?
What can be improved in Spark integration?

В Пн, 22/10/2018 в 10:22 +0100, Stephen Darlington пишет:
> Are you suggesting making the Thin Client deployment an option or as a 
> replacement for the thick-client? If the latter, do we risk making future 
> desirable changes more difficult (or impossible)? I’m thinking specifically 
> about better support for Spark Streaming, where the lack  of continuous query 
> support in thin clients removes a significant optimisation option. I’m sure 
> there are other use cases.
> 
> Regards,
> Stephen
> 
> > On 21 Oct 2018, at 09:08, Nikolay Izhikov  wrote:
> > 
> > Valentin.
> > 
> > Seems, You made several suggestions, which is not always true, from my 
> > point of view:
> > 
> > 1. "We have access to Spark cluster installation to perform deployment 
> > steps" - this is not true in cloud or enterprise environment.
> > 
> > 2. "Spark cluster is used only for Ignite integration".
> > From what I know computational resources for big Spark cluster is divided 
> > by many business divisions.
> > And it is not convenient to perform some deployment steps on this cluster.
> > 
> > 3. "When Ignite + Spark are used in real production it's OK to have 
> > reasonable deployment overhead"
> > What about developer who want to play with this integration?
> > And want to do it quickly to see how it works in real life examples.
> > Can we do his life much easier?
> > 
> > > First of all, they will exist with thin client either.
> > 
> > Spark have an ability to deploy jars on worker and add it to application 
> > tasks classpath.
> > For 2.6 we must deploy 11 additional jars to start using Ignite.
> > Please, see my example on the bottom of documentation page [1]
> > 
> > Does cache-api-1.0.0.jar and h2-1.4.195.jar seems like obvious dependencies 
> > for Ignite integration for you?
> > And for our users? :)
> > 
> > Actually, list of dependencies will be changed in 2.7 - new version of 
> > jcache, new version of h2
> > So user should change it in code or perform additional deployment steps.
> > 
> > It overkill for me.
> > 
> > On the other hand - thin client requires only 1 jar.
> > Moreover, thin client protocol have the backward compatibility.
> > So thin client will perform correctly when Ignite cluster will be updated 
> > from 2.6 to 2.7.
> > So, with Spark integration via thin client we will be able to update Ignite 
> > cluster and Spark integration separately.
> > For now, we should do it in one big step.
> > 
> > What do you think?
> > 
> > [1] https://apacheignite-fs.readme.io/docs/installation-deployment
> > 
> > В Сб, 20/10/2018 в 18:33 -0700, Valentin Kulichenko пишет:
> > > Guys,
> > > 
> > > From my experience, Ignite and Spark clusters typically run in the same
> > > environment, which makes client node a more preferable option. Mainly,
> > > because of performance. BTW, I doubt partition-awareness on thin client
> > > will help either, because in dataframes we only run SQL queries and I
> > > believe thin client will execute them through a proxy anyway. But correct
> > > me if I’m wrong.
> > > 
> > > Either way, it sounds like we just have usability issues with Ignite/Spark
> > > integration. Why don’t we concentrate on fixing them then? For example, #3
> > > can be fixed by loading XML content on master and then distributing it to
> > > workers, instead of loading on every worker independently. Then there are
> > > certain procedures like deploying JARs, etc. First of all, they will exist
> > > with thin client either. Second of all, I’m sure there are ways to 
> > > simplify
> > > this procedures and make integration easier. My opinion is that working on
> > > such improvements is going to add more value than another implementation
> > > based on thin client.
> > > 
> > > -Val
> > > 
> > > On Sat, Oct 20, 2018 at 4:03 PM Denis Magda  wrote:
> > > 
> > > > Hello Nikolay,
> > > > 
> > > > Your proposal sounds reasonable. However, I would suggest us to wait 
> > > > while
>

Re: Apache Ignite 2.7. Last Mile

2018-10-21 Thread Nikolay Izhikov
Hell, Denis.

I just filter all 2.7 tickets without documentation.


https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(documentation)))%20ORDER%20BY%20%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20

I've added this view to the release page.
See sesction "Unresolved tickets(without documentation)".

В Сб, 20/10/2018 в 16:04 -0700, Denis Magda пишет:
> Nikolay,
> 
> Where do you track those 8 blockers? Can't find them on this wiki page:
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
> 
> --
> Denis
> 
> On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov 
> wrote:
> 
> > Hello, Denis.
> > 
> > As a first time release manager I'm trying to rely on Ignite veterans
> > opinion.
> > Guys told me that we must fix all blockers and only after it make the
> > release.
> > 
> > Let's fix them all.
> > What do you think?
> > 
> > > When are we sending a release candidate for vote?
> > 
> > When all blocker bugs will be fixed.
> > Currently, we have *8*.
> > 
> > В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> > > Guys, as a side observer of the current release, this all looks like a
> > > never ending story :)
> > > 
> > > When are we sending a release candidate for vote?
> > > 
> > > --
> > > Denis
> > > 
> > > On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov 
> > 
> > wrote:
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > We have 6 tickets for 2.7
> > > > 
> > > > Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> > > > Igor Seliverstov - IGNITE-9911
> > > > Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
> > > > 
> > > > В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > > > > I have got one more potential 2.7 blocker [1] with straightforward
> > 
> > fix. I
> > > > > beleive it will not break any production use case, but it leads to
> > 
> > test
> > > > > suite hang, thus affecting other urgent issues.
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> > > > > 
> > > > > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> > > > > 
> > > > > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> > > > 
> > > > quite
> > > > > > important.
> > > > > > 
> > > > > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > > > > > 
> > > > > > > JFYI
> > > > > > > 
> > > > > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn <
> > 
> > ptupit...@apache.org
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > Thank you. Fix has been merged to master and cherry-picked to
> > > > > > 
> > > > > > ignite-2.7.
> > > > > > > > 
> > > > > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> > > > 
> > > > nizhi...@apache.org>
> > > > > > > 
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Pavel.
> > > > > > > > > 
> > > > > > > > > Ok, I agree to include this ticket into 2.7
> > > > > > > > > Let's do it.
> > > > > > > > > 
> > > > > > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > > Nikolay,
> > > > > > > > > > 
> > > > > > > > > > It completely breaks a major feature under certain
> > 
> > conditions.
> > > > 
> > > > I
> > > > > > > 
> > > > > > > would
> > > > > > > > > > consider it a blocker.
> > > > > > > > > > 
> > > > > > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > > > > > 
> > > > >

Re: [DISCUSSION] Spark Data Frame through Thin Client

2018-10-21 Thread Nikolay Izhikov
Valentin.

Seems, You made several suggestions, which is not always true, from my point of 
view:

1. "We have access to Spark cluster installation to perform deployment steps" - 
this is not true in cloud or enterprise environment.

2. "Spark cluster is used only for Ignite integration".
From what I know computational resources for big Spark cluster is divided by 
many business divisions.
And it is not convenient to perform some deployment steps on this cluster.

3. "When Ignite + Spark are used in real production it's OK to have reasonable 
deployment overhead"
What about developer who want to play with this integration?
And want to do it quickly to see how it works in real life examples.
Can we do his life much easier?

> First of all, they will exist with thin client either.

Spark have an ability to deploy jars on worker and add it to application tasks 
classpath.
For 2.6 we must deploy 11 additional jars to start using Ignite.
Please, see my example on the bottom of documentation page [1]

Does cache-api-1.0.0.jar and h2-1.4.195.jar seems like obvious dependencies for 
Ignite integration for you?
And for our users? :)

Actually, list of dependencies will be changed in 2.7 - new version of jcache, 
new version of h2
So user should change it in code or perform additional deployment steps.

It overkill for me.

On the other hand - thin client requires only 1 jar.
Moreover, thin client protocol have the backward compatibility.
So thin client will perform correctly when Ignite cluster will be updated from 
2.6 to 2.7.
So, with Spark integration via thin client we will be able to update Ignite 
cluster and Spark integration separately.
For now, we should do it in one big step.

What do you think?

[1] https://apacheignite-fs.readme.io/docs/installation-deployment

В Сб, 20/10/2018 в 18:33 -0700, Valentin Kulichenko пишет:
> Guys,
> 
> From my experience, Ignite and Spark clusters typically run in the same
> environment, which makes client node a more preferable option. Mainly,
> because of performance. BTW, I doubt partition-awareness on thin client
> will help either, because in dataframes we only run SQL queries and I
> believe thin client will execute them through a proxy anyway. But correct
> me if I’m wrong.
> 
> Either way, it sounds like we just have usability issues with Ignite/Spark
> integration. Why don’t we concentrate on fixing them then? For example, #3
> can be fixed by loading XML content on master and then distributing it to
> workers, instead of loading on every worker independently. Then there are
> certain procedures like deploying JARs, etc. First of all, they will exist
> with thin client either. Second of all, I’m sure there are ways to simplify
> this procedures and make integration easier. My opinion is that working on
> such improvements is going to add more value than another implementation
> based on thin client.
> 
> -Val
> 
> On Sat, Oct 20, 2018 at 4:03 PM Denis Magda  wrote:
> 
> > Hello Nikolay,
> > 
> > Your proposal sounds reasonable. However, I would suggest us to wait while
> > partition-awareness is supported for Java thin client first. With that
> > feature, the client can connect to any node directly while presently all
> > the communication goes through a proxy (a node the client is connected to).
> > All of that is bad for performance.
> > 
> > 
> > Vladimir, how hard would it be to support the partition-awareness for Java
> > client? Probably, Nikolay can take over.
> > 
> > --
> > Denis
> > 
> > 
> > On Sat, Oct 20, 2018 at 2:09 PM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > Currently, Spark Data Frame integration implemented via client node
> > > connection.
> > > Whenever we need to retrieve some data into Spark worker(or master) from
> > > Ignite we start a client node.
> > > 
> > > It has several major disadvantages:
> > > 
> > > 1. We should copy whole Ignite distribution on to each Spark
> > > worker [1]
> > > 2. We should copy whole Ignite distribution on to Spark master to
> > > get catalogue works.
> > > 3. We should have the same absolute path to Ignite configuration
> > > file on every worker and provide it during data frame construction [2]
> > > 4. We should additionally configure Spark workerks classpath to
> > > include Ignite libraries.
> > > 
> > > For now, almost all operation we need to do in Spark Data Frame
> > > integration is supported by Java Thin Client.
> > > * obtain the list of caches.
> > > * get cache configuration.
> > > * execute SQL q

[DISCUSSION] Spark Data Frame through Thin Client

2018-10-20 Thread Nikolay Izhikov
Hello, Igniters.

Currently, Spark Data Frame integration implemented via client node connection.
Whenever we need to retrieve some data into Spark worker(or master) from Ignite 
we start a client node.

It has several major disadvantages:

1. We should copy whole Ignite distribution on to each Spark worker [1]
2. We should copy whole Ignite distribution on to Spark master to get 
catalogue works.
3. We should have the same absolute path to Ignite configuration file 
on every worker and provide it during data frame construction [2]
4. We should additionally configure Spark workerks classpath to include 
Ignite libraries.

For now, almost all operation we need to do in Spark Data Frame integration is 
supported by Java Thin Client.
* obtain the list of caches.
* get cache configuration.
* execute SQL query.
* stream data to the table - don't support by the thin client for now, 
but can be implemented using simple SQL INSERT statements.

Advantages of usage Java Thin Client in Spark integration(they all known from 
Java Thin Client advantages):
1. Easy to configure: only IP addresses of server nodes are required.
2. Easy to deploy: only 1 additional jar required. No server 
side(Ignite worker) configuration required.

I propose to implement Spark Data Frame integration through Java Thin Client.

Thoughts?

[1] https://apacheignite-fs.readme.io/docs/installation-deployment
[2] 
https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-ignite-dataframe-options


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7. Last Mile

2018-10-20 Thread Nikolay Izhikov
Hello, Denis.

As a first time release manager I'm trying to rely on Ignite veterans opinion.
Guys told me that we must fix all blockers and only after it make the release.

Let's fix them all.
What do you think?

> When are we sending a release candidate for vote?

When all blocker bugs will be fixed.
Currently, we have *8*.

В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> Guys, as a side observer of the current release, this all looks like a
> never ending story :)
> 
> When are we sending a release candidate for vote?
> 
> --
> Denis
> 
> On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov  wrote:
> 
> > Hello, Igniters.
> > 
> > We have 6 tickets for 2.7
> > 
> > Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> > Igor Seliverstov - IGNITE-9911
> > Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
> > 
> > В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > > I have got one more potential 2.7 blocker [1] with straightforward fix. I
> > > beleive it will not break any production use case, but it leads to test
> > > suite hang, thus affecting other urgent issues.
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> > > 
> > > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> > > 
> > > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> > 
> > quite
> > > > important.
> > > > 
> > > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > > > 
> > > > > JFYI
> > > > > 
> > > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn  > > > > wrote:
> > > > > > 
> > > > > > Thank you. Fix has been merged to master and cherry-picked to
> > > > 
> > > > ignite-2.7.
> > > > > > 
> > > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> > 
> > nizhi...@apache.org>
> > > > > 
> > > > > wrote:
> > > > > > 
> > > > > > > Pavel.
> > > > > > > 
> > > > > > > Ok, I agree to include this ticket into 2.7
> > > > > > > Let's do it.
> > > > > > > 
> > > > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > > > Nikolay,
> > > > > > > > 
> > > > > > > > It completely breaks a major feature under certain conditions.
> > 
> > I
> > > > > 
> > > > > would
> > > > > > > > consider it a blocker.
> > > > > > > > 
> > > > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > > > 
> > > > nizhi...@apache.org
> > > > > > 
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hello, Pavel.
> > > > > > > > > 
> > > > > > > > > Is it a blocker?
> > > > > > > > > 
> > > > > > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > > Hi Igniters,
> > > > > > > > > > 
> > > > > > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > > > > > The fix is ready, I'm waiting for TC run.
> > > > > > > > > > 
> > > > > > > > > > Pavel
> > > > > > > > > > 
> > > > > > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> > > > > 
> > > > > vololo...@gmail.com>
> > > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi NIkolay,
> > > > > > > > > > > 
> > > > > > > > > > > Thank you for keeping everybody focused! Regarding to my
> > > > 
> > > > ticket
> > > > > > > > > > > IGNITE-5935.
> > > > > > > > > > > It is in final stage now. Tests look good. I believe
> > 
> > that it
> > > > > 
> > > > > will
> > > > > > > be
> > > > > >

Re: Apache Ignite 2.7. Last Mile

2018-10-19 Thread Nikolay Izhikov
Hello, Igniters.

We have 6 tickets for 2.7

Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
Igor Seliverstov - IGNITE-9911
Ivan Pavlukhin - IGNITE-5935, IGNITE-9944

В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> I have got one more potential 2.7 blocker [1] with straightforward fix. I
> beleive it will not break any production use case, but it leads to test
> suite hang, thus affecting other urgent issues.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-9932
> 
> чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> 
> > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but quite
> > important.
> > 
> > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > 
> > > JFYI
> > > 
> > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn 
> > > wrote:
> > > > 
> > > > Thank you. Fix has been merged to master and cherry-picked to
> > 
> > ignite-2.7.
> > > > 
> > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov 
> > > 
> > > wrote:
> > > > 
> > > > > Pavel.
> > > > > 
> > > > > Ok, I agree to include this ticket into 2.7
> > > > > Let's do it.
> > > > > 
> > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > Nikolay,
> > > > > > 
> > > > > > It completely breaks a major feature under certain conditions. I
> > > 
> > > would
> > > > > > consider it a blocker.
> > > > > > 
> > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > 
> > nizhi...@apache.org
> > > > 
> > > > > wrote:
> > > > > > 
> > > > > > > Hello, Pavel.
> > > > > > > 
> > > > > > > Is it a blocker?
> > > > > > > 
> > > > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > > > Hi Igniters,
> > > > > > > > 
> > > > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > > > The fix is ready, I'm waiting for TC run.
> > > > > > > > 
> > > > > > > > Pavel
> > > > > > > > 
> > > > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> > > 
> > > vololo...@gmail.com>
> > > > > > > 
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi NIkolay,
> > > > > > > > > 
> > > > > > > > > Thank you for keeping everybody focused! Regarding to my
> > 
> > ticket
> > > > > > > > > IGNITE-5935.
> > > > > > > > > It is in final stage now. Tests look good. I believe that it
> > > 
> > > will
> > > > > be
> > > > > > > 
> > > > > > > merged
> > > > > > > > > in couple of days (at most).
> > > > > > > > > 
> > > > > > > > > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org
> > > > > > :
> > > > > > > > > 
> > > > > > > > > > Hello, Igniters.
> > > > > > > > > > 
> > > > > > > > > > 9 tickets to go!
> > > > > > > > > > 
> > > > > > > > > > Alexey Goncharuk - IGNITE-9784
> > > > > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > > > > Taras Ledkov - IGNITE-9882
> > > > > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > > > > 
> > > > > > > > > > В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > >

Re: Apache Ignite 2.7. Last Mile

2018-10-17 Thread Nikolay Izhikov
Pavel.

Ok, I agree to include this ticket into 2.7
Let's do it.

В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> Nikolay,
> 
> It completely breaks a major feature under certain conditions. I would
> consider it a blocker.
> 
> On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov  wrote:
> 
> > Hello, Pavel.
> > 
> > Is it a blocker?
> > 
> > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > Hi Igniters,
> > > 
> > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > The fix is ready, I'm waiting for TC run.
> > > 
> > > Pavel
> > > 
> > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван 
> > 
> > wrote:
> > > 
> > > > Hi NIkolay,
> > > > 
> > > > Thank you for keeping everybody focused! Regarding to my ticket
> > > > IGNITE-5935.
> > > > It is in final stage now. Tests look good. I believe that it will be
> > 
> > merged
> > > > in couple of days (at most).
> > > > 
> > > > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov :
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > 9 tickets to go!
> > > > > 
> > > > > Alexey Goncharuk - IGNITE-9784
> > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > Taras Ledkov - IGNITE-9882
> > > > > Petr Ivanov - IGNITE-9852
> > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > Roman Kondakov - IGNITE-9663
> > > > > Alexey Stelmak - IGNITE-9776
> > > > > 
> > > > > В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> > > > > > Hi,
> > > > > > 
> > > > > > I've found that IGNITE-9723 was resolved but didn't cherry picked
> > 
> > to
> > > > > > ignite-2.7 branch. So I'll do it.
> > > > > > On Tue, Oct 16, 2018 at 2:30 PM Nikolay Izhikov <
> > 
> > nizhi...@apache.org>
> > > > > 
> > > > > wrote:
> > > > > > > 
> > > > > > > Hello, Igniters.
> > > > > > > 
> > > > > > > We have 13 tickets mapped to 2.7.
> > > > > > > All tickets assigned to some contributor.
> > > > > > > 
> > > > > > > Alexey Gonchruk - IGNITE-9784, IGNITE-9895
> > > > > > > Vladimir Ozerov - IGNITE-9887
> > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > Taras Ledkov - IGNITE-9864
> > > > > > > Igor Seliverstov   - IGNITE-9292
> > > > > > > 
> > > > > > > В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura пишет:
> > > > > > > > Nikolay,
> > > > > > > > 
> > > > > > > > I'm looking at IGNITE-9737 and IGNITE-9710 which are critical
> > > > 
> > > > issues
> > > > > > > > from my point of view.
> > > > > > > > I need some time for review, possible fixes and merge.
> > > > > > > > I will keep you informed.
> > > > > > > > On Mon, Oct 15, 2018 at 1:46 PM Igor Sapego <
> > 
> > isap...@apache.org>
> > > > > 
> > > > > wrote:
> > > > > > > > > 
> > > > > > > > > Guys, Python client is in the master and ignite-2.7 already.
> > > > > > > > > 
> > > > > > > > > Best Regards,
> > > > > > > > > Igor
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > On Mon, Oct 15, 2018 at 11:33 AM Vladimir Ozerov <
> > > > > 
> > > > > voze...@gridgain.com>
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Nikolay,
> > > > > > > > > > 
> > > > > > > > > > AI 2.7 will include Python thin clie

Re: Apache Ignite 2.7. Last Mile

2018-10-17 Thread Nikolay Izhikov
Hello, Pavel.

Is it a blocker?

В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> Hi Igniters,
> 
> I'd like to include IGNITE-9877 in 2.7, can we do that?
> The fix is ready, I'm waiting for TC run.
> 
> Pavel
> 
> On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван  wrote:
> 
> > Hi NIkolay,
> > 
> > Thank you for keeping everybody focused! Regarding to my ticket
> > IGNITE-5935.
> > It is in final stage now. Tests look good. I believe that it will be merged
> > in couple of days (at most).
> > 
> > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov :
> > 
> > > Hello, Igniters.
> > > 
> > > 9 tickets to go!
> > > 
> > > Alexey Goncharuk - IGNITE-9784
> > > Dmitriy Govorukhin - IGNITE-9898
> > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > Taras Ledkov - IGNITE-9882
> > > Petr Ivanov - IGNITE-9852
> > > Ivan Pavlukhin - IGNITE-5935
> > > Roman Kondakov - IGNITE-9663
> > > Alexey Stelmak - IGNITE-9776
> > > 
> > > В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> > > > Hi,
> > > > 
> > > > I've found that IGNITE-9723 was resolved but didn't cherry picked to
> > > > ignite-2.7 branch. So I'll do it.
> > > > On Tue, Oct 16, 2018 at 2:30 PM Nikolay Izhikov 
> > > 
> > > wrote:
> > > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > We have 13 tickets mapped to 2.7.
> > > > > All tickets assigned to some contributor.
> > > > > 
> > > > > Alexey Gonchruk - IGNITE-9784, IGNITE-9895
> > > > > Vladimir Ozerov - IGNITE-9887
> > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > Petr Ivanov - IGNITE-9852
> > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > Alexey Stelmak - IGNITE-9776
> > > > > Roman Kondakov - IGNITE-9663
> > > > > Taras Ledkov - IGNITE-9864
> > > > > Igor Seliverstov   - IGNITE-9292
> > > > > 
> > > > > В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura пишет:
> > > > > > Nikolay,
> > > > > > 
> > > > > > I'm looking at IGNITE-9737 and IGNITE-9710 which are critical
> > 
> > issues
> > > > > > from my point of view.
> > > > > > I need some time for review, possible fixes and merge.
> > > > > > I will keep you informed.
> > > > > > On Mon, Oct 15, 2018 at 1:46 PM Igor Sapego 
> > > 
> > > wrote:
> > > > > > > 
> > > > > > > Guys, Python client is in the master and ignite-2.7 already.
> > > > > > > 
> > > > > > > Best Regards,
> > > > > > > Igor
> > > > > > > 
> > > > > > > 
> > > > > > > On Mon, Oct 15, 2018 at 11:33 AM Vladimir Ozerov <
> > > 
> > > voze...@gridgain.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Nikolay,
> > > > > > > > 
> > > > > > > > AI 2.7 will include Python thin client. TC suite is crucial
> > 
> > part
> > > of this
> > > > > > > > feature, so we should keep the ticket in AI 2.7 scope.
> > > > > > > > 
> > > > > > > > On Mon, Oct 15, 2018 at 10:57 AM Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org>
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hello, Igniters.
> > > > > > > > > 
> > > > > > > > > There is no progress till Friday.
> > > > > > > > > We have 14 tickets mapped to 2.7.
> > > > > > > > > 
> > > > > > > > > В Пт, 12/10/2018 в 08:29 +0300, Nikolay Izhikov пишет:
> > > > > > > > > > Hello, Igniters.
> > > > > > > > > > 
> > > > > > > > > > We made some progress yesterday.
> > > > > > > > > > 
> > > > > > > > > > Please, note, we have 1 new ticket mapped to 2.7
> > > 
> > > "IGNITE-9852: Create
> > > > > > > > > 
> >

Re: Apache Ignite 2.7. Last Mile

2018-10-17 Thread Nikolay Izhikov
Hello, Igniters.

9 tickets to go!

Alexey Goncharuk - IGNITE-9784
Dmitriy Govorukhin - IGNITE-9898
Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
Taras Ledkov - IGNITE-9882
Petr Ivanov - IGNITE-9852
Ivan Pavlukhin - IGNITE-5935
Roman Kondakov - IGNITE-9663
Alexey Stelmak - IGNITE-9776

В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> Hi,
> 
> I've found that IGNITE-9723 was resolved but didn't cherry picked to
> ignite-2.7 branch. So I'll do it.
> On Tue, Oct 16, 2018 at 2:30 PM Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > We have 13 tickets mapped to 2.7.
> > All tickets assigned to some contributor.
> > 
> > Alexey Gonchruk - IGNITE-9784, IGNITE-9895
> > Vladimir Ozerov - IGNITE-9887
> > Dmitriy Govorukhin - IGNITE-9898
> > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > Petr Ivanov - IGNITE-9852
> > Ivan Pavlukhin - IGNITE-5935
> > Alexey Stelmak - IGNITE-9776
> > Roman Kondakov - IGNITE-9663
> > Taras Ledkov - IGNITE-9864
> > Igor Seliverstov   - IGNITE-9292
> > 
> > В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura пишет:
> > > Nikolay,
> > > 
> > > I'm looking at IGNITE-9737 and IGNITE-9710 which are critical issues
> > > from my point of view.
> > > I need some time for review, possible fixes and merge.
> > > I will keep you informed.
> > > On Mon, Oct 15, 2018 at 1:46 PM Igor Sapego  wrote:
> > > > 
> > > > Guys, Python client is in the master and ignite-2.7 already.
> > > > 
> > > > Best Regards,
> > > > Igor
> > > > 
> > > > 
> > > > On Mon, Oct 15, 2018 at 11:33 AM Vladimir Ozerov 
> > > > wrote:
> > > > 
> > > > > Nikolay,
> > > > > 
> > > > > AI 2.7 will include Python thin client. TC suite is crucial part of 
> > > > > this
> > > > > feature, so we should keep the ticket in AI 2.7 scope.
> > > > > 
> > > > > On Mon, Oct 15, 2018 at 10:57 AM Nikolay Izhikov 
> > > > > wrote:
> > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > There is no progress till Friday.
> > > > > > We have 14 tickets mapped to 2.7.
> > > > > > 
> > > > > > В Пт, 12/10/2018 в 08:29 +0300, Nikolay Izhikov пишет:
> > > > > > > Hello, Igniters.
> > > > > > > 
> > > > > > > We made some progress yesterday.
> > > > > > > 
> > > > > > > Please, note, we have 1 new ticket mapped to 2.7 "IGNITE-9852: 
> > > > > > > Create
> > > > > > 
> > > > > > TeamCity suite for Python thin client"
> > > > > > > 
> > > > > > > This ticket doesn't sound like a blocker to me. Let's exclude it 
> > > > > > > from
> > > > > > 
> > > > > > release scope.
> > > > > > > Thoughts?
> > > > > > > 
> > > > > > > Here is the list of remaining tickets(14) mapped to 2.7.
> > > > > > > 
> > > > > > > Peter Ivanov - IGNITE-9852, IGNITE-9685,
> > > > > > > Alexey Goncharuk   - IGNITE-9784
> > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> > > > > > > Dmitry Melnichuk   - IGNITE-7782
> > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > Yury Babak - IGNITE-8670
> > > > > > > Roman Kondakov - IGNITE-7953
> > > > > > > Igor Sapego  - IGNITE-9620
> > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > 
> > > > > > > Unassigned:
> > > > > > > 
> > > > > > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > > > > > 
> > > > > > > 
> > > > > > > В Чт, 11/10/2018 в 13:50 +0300, Nikolay Izhikov пишет:
> > > > > > > > Alexey, we all agreed to merge in 2.7 blockers only.
> > > > > > > > 
> > > > > > > > Is this a blocker?
> > > > > > > > 
> > > > > > > > Anyway, you are more experienced Igniter that I am.
> > > > > > > > If you th

Re: Apache Ignite 2.7. Last Mile

2018-10-16 Thread Nikolay Izhikov
Hello, Igniters.

We have 13 tickets mapped to 2.7.
All tickets assigned to some contributor.

Alexey Gonchruk - IGNITE-9784, IGNITE-9895
Vladimir Ozerov - IGNITE-9887
Dmitriy Govorukhin - IGNITE-9898
Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
Petr Ivanov - IGNITE-9852
Ivan Pavlukhin - IGNITE-5935
Alexey Stelmak - IGNITE-9776
Roman Kondakov - IGNITE-9663
Taras Ledkov - IGNITE-9864
Igor Seliverstov   - IGNITE-9292

В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura пишет:
> Nikolay,
> 
> I'm looking at IGNITE-9737 and IGNITE-9710 which are critical issues
> from my point of view.
> I need some time for review, possible fixes and merge.
> I will keep you informed.
> On Mon, Oct 15, 2018 at 1:46 PM Igor Sapego  wrote:
> > 
> > Guys, Python client is in the master and ignite-2.7 already.
> > 
> > Best Regards,
> > Igor
> > 
> > 
> > On Mon, Oct 15, 2018 at 11:33 AM Vladimir Ozerov 
> > wrote:
> > 
> > > Nikolay,
> > > 
> > > AI 2.7 will include Python thin client. TC suite is crucial part of this
> > > feature, so we should keep the ticket in AI 2.7 scope.
> > > 
> > > On Mon, Oct 15, 2018 at 10:57 AM Nikolay Izhikov 
> > > wrote:
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > There is no progress till Friday.
> > > > We have 14 tickets mapped to 2.7.
> > > > 
> > > > В Пт, 12/10/2018 в 08:29 +0300, Nikolay Izhikov пишет:
> > > > > Hello, Igniters.
> > > > > 
> > > > > We made some progress yesterday.
> > > > > 
> > > > > Please, note, we have 1 new ticket mapped to 2.7 "IGNITE-9852: Create
> > > > 
> > > > TeamCity suite for Python thin client"
> > > > > 
> > > > > This ticket doesn't sound like a blocker to me. Let's exclude it from
> > > > 
> > > > release scope.
> > > > > Thoughts?
> > > > > 
> > > > > Here is the list of remaining tickets(14) mapped to 2.7.
> > > > > 
> > > > > Peter Ivanov - IGNITE-9852, IGNITE-9685,
> > > > > Alexey Goncharuk   - IGNITE-9784
> > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> > > > > Dmitry Melnichuk   - IGNITE-7782
> > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > Yury Babak - IGNITE-8670
> > > > > Roman Kondakov - IGNITE-7953
> > > > > Igor Sapego  - IGNITE-9620
> > > > > Alexey Stelmak - IGNITE-9776
> > > > > 
> > > > > Unassigned:
> > > > > 
> > > > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > > > 
> > > > > 
> > > > > В Чт, 11/10/2018 в 13:50 +0300, Nikolay Izhikov пишет:
> > > > > > Alexey, we all agreed to merge in 2.7 blockers only.
> > > > > > 
> > > > > > Is this a blocker?
> > > > > > 
> > > > > > Anyway, you are more experienced Igniter that I am.
> > > > > > If you think we should include this ticket to 2.7 - please, do it.
> > > > > > 
> > > > > > В Чт, 11/10/2018 в 13:08 +0300, Alexey Goncharuk пишет:
> > > > > > > Nikolay,
> > > > > > > 
> > > > > > > I am waiting for final benchmark results for 9784, after that I
> > > 
> > > will
> > > > merge
> > > > > > > the change.
> > > > > > > 
> > > > > > > On the subject of Ignite 2.7 scope, our fellow Igniter Alexey
> > > > 
> > > > Platonov have
> > > > > > > found another case when a failure handler is incorrectly called on
> > > > 
> > > > node
> > > > > > > stop: https://issues.apache.org/jira/browse/IGNITE-9834. The case
> > > > 
> > > > is rare,
> > > > > > > but it is quite an unpleasant UX. Should we include it to 2.7 as
> > > > 
> > > > well?
> > > > > > > 
> > > > > > > чт, 11 окт. 2018 г. в 11:22, Nikolay Izhikov  > > > 
> > > > :
> > > > > > > 
> > > > > > > > Hello, Igniters.
> > > > > > > > 
> > > > > > > > We made a good progress yesterday.
> > > > > > &g

Re: Apache Ignite 2.7. Last Mile

2018-10-15 Thread Nikolay Izhikov
Hello, Igniters.

There is no progress till Friday.
We have 14 tickets mapped to 2.7.

В Пт, 12/10/2018 в 08:29 +0300, Nikolay Izhikov пишет:
> Hello, Igniters.
> 
> We made some progress yesterday.
> 
> Please, note, we have 1 new ticket mapped to 2.7 "IGNITE-9852: Create 
> TeamCity suite for Python thin client"
> 
> This ticket doesn't sound like a blocker to me. Let's exclude it from release 
> scope.
> Thoughts?
> 
> Here is the list of remaining tickets(14) mapped to 2.7.
> 
> Peter Ivanov - IGNITE-9852, IGNITE-9685, 
> Alexey Goncharuk   - IGNITE-9784
> Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> Dmitry Melnichuk   - IGNITE-7782
> Ivan Pavlukhin - IGNITE-5935
> Yury Babak - IGNITE-8670
> Roman Kondakov - IGNITE-7953
> Igor Sapego  - IGNITE-9620
> Alexey Stelmak - IGNITE-9776
> 
> Unassigned:
> 
> IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> 
> 
> В Чт, 11/10/2018 в 13:50 +0300, Nikolay Izhikov пишет:
> > Alexey, we all agreed to merge in 2.7 blockers only.
> > 
> > Is this a blocker?
> > 
> > Anyway, you are more experienced Igniter that I am.
> > If you think we should include this ticket to 2.7 - please, do it.
> > 
> > В Чт, 11/10/2018 в 13:08 +0300, Alexey Goncharuk пишет:
> > > Nikolay,
> > > 
> > > I am waiting for final benchmark results for 9784, after that I will merge
> > > the change.
> > > 
> > > On the subject of Ignite 2.7 scope, our fellow Igniter Alexey Platonov 
> > > have
> > > found another case when a failure handler is incorrectly called on node
> > > stop: https://issues.apache.org/jira/browse/IGNITE-9834. The case is rare,
> > > but it is quite an unpleasant UX. Should we include it to 2.7 as well?
> > > 
> > > чт, 11 окт. 2018 г. в 11:22, Nikolay Izhikov :
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > We made a good progress yesterday.
> > > > 
> > > > Here is the list of remaining tickets(17) mapped to 2.7:
> > > > 
> > > > Alexey Goncharuk   - IGNITE-9784
> > > > Taras Ledkov   - IGNITE-9171
> > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > Peter Ivanov   - IGNITE-9583, IGNITE-9823, IGNITE-9685
> > > > Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> > > > Dmitry Melnichuk   - IGNITE-7782
> > > > Ivan Pavlukhin - IGNITE-5935
> > > > Yury Babak - IGNITE-8670
> > > > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > > > Alexey Stelmak - IGNITE-9776
> > > > 
> > > > Unassigned:
> > > > 
> > > > IGNITE-9620 - MVCC: select throwing `Transaction is already completed`> 
> > > > >
> > > > > exception after mvcc missmatch
> > > > 
> > > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > > 
> > > > 
> > > > В Чт, 11/10/2018 в 10:40 +0300, Vladimir Ozerov пишет:
> > > > > What kind of help is needed?
> > > > > 
> > > > > On Wed, Oct 10, 2018 at 11:51 PM Dmitriy Setrakyan <
> > > > 
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > > 
> > > > > > Vladimir Ozerov,
> > > > > > 
> > > > > > Can you help with the unassigned MVCC tickets?
> > > > > > 
> > > > > > D.
> > > > > > 
> > > > > > On Wed, Oct 10, 2018 at 3:32 AM Nikolay Izhikov 
> > > > > > 
> > > > > > wrote:
> > > > > > 
> > > > > > > Hello, Igniters.
> > > > > > > 
> > > > > > > I list all contributors that assigned to the 2.7 tickets.
> > > > > > > If you can help them to finish that tickets - please, do.
> > > > > > > Assigners, if you need any help - please, respond to this thread.
> > > > > > > 
> > > > > > > NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on 
> > > > > > > it!
> > > > > > > 
> > > > > > > Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685,
> > > > 
> > > > IGNITE-9823
> > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > Taras Ledkov   - IGNITE-9171
> > > > 

Re: Apache Ignite 2.7. Last Mile

2018-10-11 Thread Nikolay Izhikov
Hello, Igniters.

We made some progress yesterday.

Please, note, we have 1 new ticket mapped to 2.7 "IGNITE-9852: Create TeamCity 
suite for Python thin client"

This ticket doesn't sound like a blocker to me. Let's exclude it from release 
scope.
Thoughts?

Here is the list of remaining tickets(14) mapped to 2.7.

Peter Ivanov   - IGNITE-9852, IGNITE-9685, 
Alexey Goncharuk   - IGNITE-9784
Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
Igor Seliverstov   - IGNITE-9749, IGNITE-9292
Dmitry Melnichuk   - IGNITE-7782
Ivan Pavlukhin - IGNITE-5935
Yury Babak - IGNITE-8670
Roman Kondakov - IGNITE-7953
Igor Sapego- IGNITE-9620
Alexey Stelmak - IGNITE-9776

Unassigned:

IGNITE-9663 - MVCC: Data node failure can cause TX hanging.


В Чт, 11/10/2018 в 13:50 +0300, Nikolay Izhikov пишет:
> Alexey, we all agreed to merge in 2.7 blockers only.
> 
> Is this a blocker?
> 
> Anyway, you are more experienced Igniter that I am.
> If you think we should include this ticket to 2.7 - please, do it.
> 
> В Чт, 11/10/2018 в 13:08 +0300, Alexey Goncharuk пишет:
> > Nikolay,
> > 
> > I am waiting for final benchmark results for 9784, after that I will merge
> > the change.
> > 
> > On the subject of Ignite 2.7 scope, our fellow Igniter Alexey Platonov have
> > found another case when a failure handler is incorrectly called on node
> > stop: https://issues.apache.org/jira/browse/IGNITE-9834. The case is rare,
> > but it is quite an unpleasant UX. Should we include it to 2.7 as well?
> > 
> > чт, 11 окт. 2018 г. в 11:22, Nikolay Izhikov :
> > 
> > > Hello, Igniters.
> > > 
> > > We made a good progress yesterday.
> > > 
> > > Here is the list of remaining tickets(17) mapped to 2.7:
> > > 
> > > Alexey Goncharuk   - IGNITE-9784
> > > Taras Ledkov   - IGNITE-9171
> > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > Peter Ivanov   - IGNITE-9583, IGNITE-9823, IGNITE-9685
> > > Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> > > Dmitry Melnichuk   - IGNITE-7782
> > > Ivan Pavlukhin - IGNITE-5935
> > > Yury Babak - IGNITE-8670
> > > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > > Alexey Stelmak - IGNITE-9776
> > > 
> > > Unassigned:
> > > 
> > > IGNITE-9620 - MVCC: select throwing `Transaction is already completed`> >
> > > > exception after mvcc missmatch
> > > 
> > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > 
> > > 
> > > В Чт, 11/10/2018 в 10:40 +0300, Vladimir Ozerov пишет:
> > > > What kind of help is needed?
> > > > 
> > > > On Wed, Oct 10, 2018 at 11:51 PM Dmitriy Setrakyan <
> > > 
> > > dsetrak...@apache.org>
> > > > wrote:
> > > > 
> > > > > Vladimir Ozerov,
> > > > > 
> > > > > Can you help with the unassigned MVCC tickets?
> > > > > 
> > > > > D.
> > > > > 
> > > > > On Wed, Oct 10, 2018 at 3:32 AM Nikolay Izhikov 
> > > > > wrote:
> > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > I list all contributors that assigned to the 2.7 tickets.
> > > > > > If you can help them to finish that tickets - please, do.
> > > > > > Assigners, if you need any help - please, respond to this thread.
> > > > > > 
> > > > > > NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on it!
> > > > > > 
> > > > > > Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685,
> > > 
> > > IGNITE-9823
> > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > Taras Ledkov   - IGNITE-9171
> > > > > > Alexey Goncharuk   - IGNITE-9784
> > > > > > Dmitriy Govorukhin - IGNITE-9550
> > > > > > Igor Seliverstov   - IGNITE-9749
> > > > > > Dmitry Melnichuk   - IGNITE-7782
> > > > > > Alexey Platonov- IGNITE-9726
> > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > Yury Babak - IGNITE-8670
> > > > > > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > > > > > Maxim Pudov- IGNITE-9126
> > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > Alexey Kuznetsov   - IGNITE-7926
> > > > > > 
> > > > > > Unassigned tickets:
> > > > > > 
> > > > > > IGNITE-9781 - JDK11: SSL handshake is failed
> > > > > > IGNITE-9620 - MVCC: select throwing `Transaction is already
> > > 
> > > completed`
> > > > > > exception after mvcc missmatch
> > > > > > IGNITE-9292 - MVCC SQL: Unexpected state exception when updating
> > > 
> > > backup
> > > > > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > > > > IGNITE-9724 - MVCC SQL: Test
> > > > > > 
> > > 
> > > CacheMvccSelectForUpdateQueryAbstractTest.testSelectForUpdateDistributed
> > > > > > hangs sporadically.
> > > > > > IGNITE-9133 - MVCC: Proper empty DHT transactions handling.
> > > > > > 


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7. Last Mile

2018-10-11 Thread Nikolay Izhikov
Alexey, we all agreed to merge in 2.7 blockers only.

Is this a blocker?

Anyway, you are more experienced Igniter that I am.
If you think we should include this ticket to 2.7 - please, do it.

В Чт, 11/10/2018 в 13:08 +0300, Alexey Goncharuk пишет:
> Nikolay,
> 
> I am waiting for final benchmark results for 9784, after that I will merge
> the change.
> 
> On the subject of Ignite 2.7 scope, our fellow Igniter Alexey Platonov have
> found another case when a failure handler is incorrectly called on node
> stop: https://issues.apache.org/jira/browse/IGNITE-9834. The case is rare,
> but it is quite an unpleasant UX. Should we include it to 2.7 as well?
> 
> чт, 11 окт. 2018 г. в 11:22, Nikolay Izhikov :
> 
> > Hello, Igniters.
> > 
> > We made a good progress yesterday.
> > 
> > Here is the list of remaining tickets(17) mapped to 2.7:
> > 
> > Alexey Goncharuk   - IGNITE-9784
> > Taras Ledkov   - IGNITE-9171
> > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > Peter Ivanov   - IGNITE-9583, IGNITE-9823, IGNITE-9685
> > Igor Seliverstov   - IGNITE-9749, IGNITE-9292
> > Dmitry Melnichuk   - IGNITE-7782
> > Ivan Pavlukhin - IGNITE-5935
> > Yury Babak - IGNITE-8670
> > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > Alexey Stelmak - IGNITE-9776
> > 
> > Unassigned:
> > 
> > IGNITE-9620 - MVCC: select throwing `Transaction is already completed`> >
> > > exception after mvcc missmatch
> > 
> > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > 
> > 
> > В Чт, 11/10/2018 в 10:40 +0300, Vladimir Ozerov пишет:
> > > What kind of help is needed?
> > > 
> > > On Wed, Oct 10, 2018 at 11:51 PM Dmitriy Setrakyan <
> > 
> > dsetrak...@apache.org>
> > > wrote:
> > > 
> > > > Vladimir Ozerov,
> > > > 
> > > > Can you help with the unassigned MVCC tickets?
> > > > 
> > > > D.
> > > > 
> > > > On Wed, Oct 10, 2018 at 3:32 AM Nikolay Izhikov 
> > > > wrote:
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > I list all contributors that assigned to the 2.7 tickets.
> > > > > If you can help them to finish that tickets - please, do.
> > > > > Assigners, if you need any help - please, respond to this thread.
> > > > > 
> > > > > NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on it!
> > > > > 
> > > > > Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685,
> > 
> > IGNITE-9823
> > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > Taras Ledkov   - IGNITE-9171
> > > > > Alexey Goncharuk   - IGNITE-9784
> > > > > Dmitriy Govorukhin - IGNITE-9550
> > > > > Igor Seliverstov   - IGNITE-9749
> > > > > Dmitry Melnichuk   - IGNITE-7782
> > > > > Alexey Platonov- IGNITE-9726
> > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > Yury Babak - IGNITE-8670
> > > > > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > > > > Maxim Pudov- IGNITE-9126
> > > > > Alexey Stelmak - IGNITE-9776
> > > > > Alexey Kuznetsov   - IGNITE-7926
> > > > > 
> > > > > Unassigned tickets:
> > > > > 
> > > > > IGNITE-9781 - JDK11: SSL handshake is failed
> > > > > IGNITE-9620 - MVCC: select throwing `Transaction is already
> > 
> > completed`
> > > > > exception after mvcc missmatch
> > > > > IGNITE-9292 - MVCC SQL: Unexpected state exception when updating
> > 
> > backup
> > > > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > > > IGNITE-9724 - MVCC SQL: Test
> > > > > 
> > 
> > CacheMvccSelectForUpdateQueryAbstractTest.testSelectForUpdateDistributed
> > > > > hangs sporadically.
> > > > > IGNITE-9133 - MVCC: Proper empty DHT transactions handling.
> > > > > 


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7. Last Mile

2018-10-11 Thread Nikolay Izhikov
Hello, Igniters.

We made a good progress yesterday.

Here is the list of remaining tickets(17) mapped to 2.7:

Alexey Goncharuk   - IGNITE-9784
Taras Ledkov   - IGNITE-9171
Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
Peter Ivanov   - IGNITE-9583, IGNITE-9823, IGNITE-9685
Igor Seliverstov   - IGNITE-9749, IGNITE-9292
Dmitry Melnichuk   - IGNITE-7782
Ivan Pavlukhin - IGNITE-5935
Yury Babak - IGNITE-8670
Roman Kondakov - IGNITE-7953, IGNITE-9446
Alexey Stelmak - IGNITE-9776

Unassigned:

IGNITE-9620 - MVCC: select throwing `Transaction is already completed`> > > 
exception after mvcc missmatch
IGNITE-9663 - MVCC: Data node failure can cause TX hanging.


В Чт, 11/10/2018 в 10:40 +0300, Vladimir Ozerov пишет:
> What kind of help is needed?
> 
> On Wed, Oct 10, 2018 at 11:51 PM Dmitriy Setrakyan 
> wrote:
> 
> > Vladimir Ozerov,
> > 
> > Can you help with the unassigned MVCC tickets?
> > 
> > D.
> > 
> > On Wed, Oct 10, 2018 at 3:32 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > I list all contributors that assigned to the 2.7 tickets.
> > > If you can help them to finish that tickets - please, do.
> > > Assigners, if you need any help - please, respond to this thread.
> > > 
> > > NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on it!
> > > 
> > > Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685, IGNITE-9823
> > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > Taras Ledkov   - IGNITE-9171
> > > Alexey Goncharuk   - IGNITE-9784
> > > Dmitriy Govorukhin - IGNITE-9550
> > > Igor Seliverstov   - IGNITE-9749
> > > Dmitry Melnichuk   - IGNITE-7782
> > > Alexey Platonov- IGNITE-9726
> > > Ivan Pavlukhin - IGNITE-5935
> > > Yury Babak - IGNITE-8670
> > > Roman Kondakov - IGNITE-7953, IGNITE-9446
> > > Maxim Pudov- IGNITE-9126
> > > Alexey Stelmak - IGNITE-9776
> > > Alexey Kuznetsov   - IGNITE-7926
> > > 
> > > Unassigned tickets:
> > > 
> > > IGNITE-9781 - JDK11: SSL handshake is failed
> > > IGNITE-9620 - MVCC: select throwing `Transaction is already completed`
> > > exception after mvcc missmatch
> > > IGNITE-9292 - MVCC SQL: Unexpected state exception when updating backup
> > > IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
> > > IGNITE-9724 - MVCC SQL: Test
> > > CacheMvccSelectForUpdateQueryAbstractTest.testSelectForUpdateDistributed
> > > hangs sporadically.
> > > IGNITE-9133 - MVCC: Proper empty DHT transactions handling.
> > > 


signature.asc
Description: This is a digitally signed message part


Apache Ignite 2.7. Last Mile

2018-10-10 Thread Nikolay Izhikov
Hello, Igniters.

I list all contributors that assigned to the 2.7 tickets.
If you can help them to finish that tickets - please, do.
Assigners, if you need any help - please, respond to this thread.

NOTE: We have 6 Unassigned tickets for 2.7. Let's start work on it!

Peter Ivanov   - IGNITE-9559, IGNITE-9583, IGNITE-9685, IGNITE-9823
Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
Taras Ledkov   - IGNITE-9171
Alexey Goncharuk   - IGNITE-9784
Dmitriy Govorukhin - IGNITE-9550
Igor Seliverstov   - IGNITE-9749
Dmitry Melnichuk   - IGNITE-7782
Alexey Platonov- IGNITE-9726
Ivan Pavlukhin - IGNITE-5935
Yury Babak - IGNITE-8670
Roman Kondakov - IGNITE-7953, IGNITE-9446
Maxim Pudov- IGNITE-9126
Alexey Stelmak - IGNITE-9776
Alexey Kuznetsov   - IGNITE-7926

Unassigned tickets: 

IGNITE-9781 - JDK11: SSL handshake is failed
IGNITE-9620 - MVCC: select throwing `Transaction is already completed` 
exception after mvcc missmatch
IGNITE-9292 - MVCC SQL: Unexpected state exception when updating backup
IGNITE-9663 - MVCC: Data node failure can cause TX hanging.
IGNITE-9724 - MVCC SQL: Test 
CacheMvccSelectForUpdateQueryAbstractTest.testSelectForUpdateDistributed hangs 
sporadically.
IGNITE-9133 - MVCC: Proper empty DHT transactions handling.


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-10-10 Thread Nikolay Izhikov
Hello, Igniters.

Today is a code freeze date but we still have 29 not merged tickets mapped to 
2.7 release [1]

It means:

1. We should continue work on 2.7 release and merge remaining tickets
2. Anyone who can help with #1, please, do it.
3. RC1 and further release builds will be created after all tickets will be 
merged.

I will inform you about the progress on a daily basis.


[1] 
https://issues.apache.org/jira/browse/IGNITE-8803?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20not%20in%20(documentation)))%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20


В Ср, 03/10/2018 в 14:02 +0300, Nikolay Izhikov пишет:
> Alexey.
> 
> Sorry, I lost link to IGNITE-9760 in this thread :)
> 
> Thanks, for a clarification.
> 
> 
> В Ср, 03/10/2018 в 13:58 +0300, Alexey Goncharuk пишет:
> > Nikolay, both commits fixed a regression compared to ignite-2.6. First one 
> > was mentioned by Anton Kalashnikov before (java-level deadlock during WAL 
> > flush), another - by Andrey Kuznetsov (NPE during a concurrent WAL flush).
> > 
> > --AG
> > 
> > ср, 3 окт. 2018 г. в 13:38, Nikolay Izhikov :
> > > Hello, Igniters.
> > > 
> > > Release scope is frozen.
> > > Please, if you include some new issues in release - discuss it in this 
> > > thread.
> > > 
> > > Alexey, can you, please, comment on including fix for IGNITE-9760, 
> > > IGNITE-9761 in 2.7 branch.
> > > 
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3355201f3e8cafd23b2250aaf3b91b8b8ed1
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=9d6e6ff394c05ddf7ef31a9d9ed1b492d9eeba69
> > > 
> > > В Ср, 03/10/2018 в 13:24 +0300, Vladimir Ozerov пишет:
> > > > Nobody vetos anything, let's stop use this term unless some really
> > > > important problem is discussed.
> > > > 
> > > > At this point we are in situation when new tickets are still included 
> > > > into
> > > > the scope. All want to ask is to stop including new tickets without
> > > > explaining on why they should be in AI 2.7. Regression between is AI 2.6
> > > > and AI 2.7 is enough. But "I found new NPE" is not.
> > > > 
> > > > On Wed, Oct 3, 2018 at 11:10 AM Dmitriy Pavlov 
> > > > wrote:
> > > > 
> > > > > Nikolay,
> > > > > 
> > > > > this has nothing about scaring someone. Let me explain about Apache 
> > > > > Way.
> > > > > 
> > > > > Voting -1 to release does not mean blocking it, release can't be 
> > > > > vetoed.
> > > > > Approving release is done by policy: majority approval. 3+1 binding 
> > > > > and
> > > > > more +1 than -1. Consensus approval is better but not mandatory.
> > > > > 
> > > > > Instead, if PMC says -1 to code modification it means veto and can't 
> > > > > be
> > > > > bypassed to anyone. This is a very strong statement, which should be
> > > > > applied reasonably and with technical justification. Lack of
> > > > > understanding is not a justification.
> > > > > 
> > > > > So my point instead of vetoing bugfix let's veto commits where the 
> > > > > bugs
> > > > > were introduced. I feel a number of bugs reported recently are all
> > > > > connected to WalManager, and these bugs may come from just a couple of
> > > > > fixes. PDS tests were quite stable last time, so I think it is 
> > > > > possible to
> > > > > find out why WAL crashes and hangs.
> > > > > 
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > > 
> > > > > ср, 3 окт. 2018 г. в 10:05, Andrey Kuznetsov :
> > > > > 
> > > > > > Vladimir, Nikolay,
> > > > > > 
> > > > > > For sure, I'm not an experienced Ignite contributor, so I'm sorry 
> > > > > > for
> > > > > > intervening. I've just run the reproducer from [1] against 
> > > > > > ignite-2.6
> > > > > > branch and it has passed. So, it's not an legacy bug, we've brought 
> > > > > > it
> > > > > 
> > > > > with
> > > > > > some change of 2.7 scope. Is it still ok to 

TDE. Phase-1 Merged to master

2018-10-09 Thread Nikolay Izhikov
Hello, Igniters.

I merged to master TDE.Phase 1 that has been actively discussed on dev-list.

I want to thank all guys that helped me with the implementation:

1. Dmitriy Ryabov   - initial design.
2. Anton Vinogradov - an initial review. wisdom sharing.
3. Vladimir Ozerov  - final review. priceless advices and knowledge of Ignite 
codebase and trends.
4. Dmitriy Pavlov   - security and process help.
5. Petr Ivanov  - Team City configuration.

Guys, this improvement wouldn't be done without your help.
Thank you!


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-10-09 Thread Nikolay Izhikov
+1 from me.

В Вт, 09/10/2018 в 19:16 +0300, Alexey Goncharuk пишет:
> Igniters, Nikolay,
> 
> I've recently discovered an issue [1] which was causing test suite to quit
> on TC. The root cause of the issue was an incorrect handling of WAL
> archiver stop, which causes a failure propagated to the failure processor
> and lead to a JVM halt on each node stop. This is a regression compared
> from 2.6.
> I think it is worth including this fix to 2.7 given that the fix is ready
> and verified on TC.
> 
> Please let me know if you have any objections.
> 
> вт, 9 окт. 2018 г. в 19:00, Andrey Kuznetsov :
> 
> > Igniters,
> > 
> > Recently, I have filed an issue [1] that deals with possible hanging of WAL
> > logging. I will appreciate your thoughts on its severity. To make logging
> > hang two conditions should be satisfied: WAL mode is {{FSYNC}}, and WAL
> > archiving is disabled. Should we investigate and fix this immediately or is
> > it possible to postpone till 2.8?
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-9776
> > 
> > вт, 9 окт. 2018 г. в 11:17, Andrey Kuznetsov :
> > 
> > > Ignite committers,
> > > 
> > > I have prepared a PR for 2.7 blocker [1]. Could anybody merge it to 2.7
> > > and master?
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9737
> > > 
> > > 
> > > ср, 3 окт. 2018 г. в 14:02, Nikolay Izhikov :
> > > 
> > > > Alexey.
> > > > 
> > > > Sorry, I lost link to IGNITE-9760 in this thread :)
> > > > 
> > > > Thanks, for a clarification.
> > > > 
> > > > 
> > > > В Ср, 03/10/2018 в 13:58 +0300, Alexey Goncharuk пишет:
> > > > > Nikolay, both commits fixed a regression compared to ignite-2.6. First
> > > > 
> > > > one was mentioned by Anton Kalashnikov before (java-level deadlock
> > 
> > during
> > > > WAL flush), another - by Andrey Kuznetsov (NPE during a concurrent WAL
> > > > flush).
> > > > > 
> > > > > --AG
> > > > > 
> > > > > ср, 3 окт. 2018 г. в 13:38, Nikolay Izhikov :
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > Release scope is frozen.
> > > > > > Please, if you include some new issues in release - discuss it in
> > > > 
> > > > this thread.
> > > > > > 
> > > > > > Alexey, can you, please, comment on including fix for IGNITE-9760,
> > > > 
> > > > IGNITE-9761 in 2.7 branch.
> > > > > > 
> > > > > > 
> > 
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3355201f3e8cafd23b2250aaf3b91b8b8ed1
> > > > > > 
> > 
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=9d6e6ff394c05ddf7ef31a9d9ed1b492d9eeba69
> > > > > > 
> > > > > > В Ср, 03/10/2018 в 13:24 +0300, Vladimir Ozerov пишет:
> > > > > > > Nobody vetos anything, let's stop use this term unless some really
> > > > > > > important problem is discussed.
> > > > > > > 
> > > > > > > At this point we are in situation when new tickets are still
> > > > 
> > > > included into
> > > > > > > the scope. All want to ask is to stop including new tickets
> > 
> > without
> > > > > > > explaining on why they should be in AI 2.7. Regression between is
> > > > 
> > > > AI 2.6
> > > > > > > and AI 2.7 is enough. But "I found new NPE" is not.
> > > > > > > 
> > > > > > > On Wed, Oct 3, 2018 at 11:10 AM Dmitriy Pavlov <
> > > > 
> > > > dpavlov@gmail.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Nikolay,
> > > > > > > > 
> > > > > > > > this has nothing about scaring someone. Let me explain about
> > > > 
> > > > Apache Way.
> > > > > > > > 
> > > > > > > > Voting -1 to release does not mean blocking it, release can't be
> > > > 
> > > > vetoed.
> > > > > > > > Approving release is done by policy: majority approval. 3+1
> > > > 
> > > > binding and
> > > > > > > > more +1 than -1. Consensus approval is bett

[jira] [Created] (IGNITE-9817) Update documentation and examples for Spark SQL Table Specification

2018-10-08 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9817:
---

 Summary: Update documentation and examples for Spark SQL Table 
Specification
 Key: IGNITE-9817
 URL: https://issues.apache.org/jira/browse/IGNITE-9817
 Project: Ignite
  Issue Type: Sub-task
Reporter: Nikolay Izhikov


We should update documentation and examples according to the results of 
IGNITE-9228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (IGNITE-9815) [TC Bot] Muted tests shouldn't considered as blocker

2018-10-08 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9815:
---

 Summary: [TC Bot] Muted tests shouldn't considered as blocker
 Key: IGNITE-9815
 URL: https://issues.apache.org/jira/browse/IGNITE-9815
 Project: Ignite
  Issue Type: Bug
Reporter: Nikolay Izhikov


Currently, TC bot doesn't analyze stack trace of test fail [1].
If some test failed with the link to Ignite ticket it shouldn't be considered 
as a blocker.

Link with examples of such behaviour.


https://issues.apache.org/jira/browse/IGNITE-9272?focusedCommentId=16641617=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16641617



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Apache Ignite 2.7 release

2018-10-04 Thread Nikolay Izhikov
Helo, Dmitriy.

I looked at patch.
Seems it local for a ML module.

Is it's true I'm +1 to include it to 2.7.


В Чт, 04/10/2018 в 08:33 -0500, dmitrievanthony пишет:
> Hi, Yury, Nikolay.
> 
> This issue reproduces in "TensorFlow on Apache Ignite" use cases. When user
> prepares training script (like official MNIST model
> https://github.com/tensorflow/models/tree/master/official/mnist), runs it in
> distributed standalone client mode (see this documentation
> https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute#standalone-client-mode)
> using IgniteDataset he can see that workers throw exceptions says that
> IgniteDataset operation is not loaded.
> 
> For more information about distributed training please see
> https://docs.google.com/document/d/1PtYmyaT4dPBNp1mWkCczOdCZHt2xaqB183honfMup3g/edit
> documentation.
> 
> 
> 
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-10-04 Thread Nikolay Izhikov
Hello, Yuriy

What is consequences of this issue?
How user can reproduce it?


В Чт, 04/10/2018 в 15:02 +0300, Yuriy Babak пишет:
> Igniters,
> 
> We have new ticket related with TensorFlow integration:
> https://issues.apache.org/jira/browse/IGNITE-9788
> 
> From my point of view this fix is important for release and I want to
> include it to 2.7.
> 
> Any objections?
> 
> пн, 20 авг. 2018 г. в 21:22, Nikolay Izhikov :
> 
> > Hello, Igniters.
> > 
> > I'm release manager of Apache Ignite 2.7.
> > 
> > It's time to start discussion of release. [1]
> > 
> > Current code freeze date is September, 30.
> > If you have any objections - please, responsd to this thread.
> > 
> > [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7


signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-10-03 Thread Nikolay Izhikov
Alexey.

Sorry, I lost link to IGNITE-9760 in this thread :)

Thanks, for a clarification.


В Ср, 03/10/2018 в 13:58 +0300, Alexey Goncharuk пишет:
> Nikolay, both commits fixed a regression compared to ignite-2.6. First one 
> was mentioned by Anton Kalashnikov before (java-level deadlock during WAL 
> flush), another - by Andrey Kuznetsov (NPE during a concurrent WAL flush).
> 
> --AG
> 
> ср, 3 окт. 2018 г. в 13:38, Nikolay Izhikov :
> > Hello, Igniters.
> > 
> > Release scope is frozen.
> > Please, if you include some new issues in release - discuss it in this 
> > thread.
> > 
> > Alexey, can you, please, comment on including fix for IGNITE-9760, 
> > IGNITE-9761 in 2.7 branch.
> > 
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3355201f3e8cafd23b2250aaf3b91b8b8ed1
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=9d6e6ff394c05ddf7ef31a9d9ed1b492d9eeba69
> > 
> > В Ср, 03/10/2018 в 13:24 +0300, Vladimir Ozerov пишет:
> > > Nobody vetos anything, let's stop use this term unless some really
> > > important problem is discussed.
> > > 
> > > At this point we are in situation when new tickets are still included into
> > > the scope. All want to ask is to stop including new tickets without
> > > explaining on why they should be in AI 2.7. Regression between is AI 2.6
> > > and AI 2.7 is enough. But "I found new NPE" is not.
> > > 
> > > On Wed, Oct 3, 2018 at 11:10 AM Dmitriy Pavlov 
> > > wrote:
> > > 
> > > > Nikolay,
> > > > 
> > > > this has nothing about scaring someone. Let me explain about Apache Way.
> > > > 
> > > > Voting -1 to release does not mean blocking it, release can't be vetoed.
> > > > Approving release is done by policy: majority approval. 3+1 binding and
> > > > more +1 than -1. Consensus approval is better but not mandatory.
> > > > 
> > > > Instead, if PMC says -1 to code modification it means veto and can't be
> > > > bypassed to anyone. This is a very strong statement, which should be
> > > > applied reasonably and with technical justification. Lack of
> > > > understanding is not a justification.
> > > > 
> > > > So my point instead of vetoing bugfix let's veto commits where the bugs
> > > > were introduced. I feel a number of bugs reported recently are all
> > > > connected to WalManager, and these bugs may come from just a couple of
> > > > fixes. PDS tests were quite stable last time, so I think it is possible 
> > > > to
> > > > find out why WAL crashes and hangs.
> > > > 
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > > 
> > > > ср, 3 окт. 2018 г. в 10:05, Andrey Kuznetsov :
> > > > 
> > > > > Vladimir, Nikolay,
> > > > > 
> > > > > For sure, I'm not an experienced Ignite contributor, so I'm sorry for
> > > > > intervening. I've just run the reproducer from [1] against ignite-2.6
> > > > > branch and it has passed. So, it's not an legacy bug, we've brought it
> > > > 
> > > > with
> > > > > some change of 2.7 scope. Is it still ok to ignore the bug?
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9776
> > > > > 
> > > > > ср, 3 окт. 2018 г. в 2:07, Nikolay Izhikov :
> > > > > 
> > > > > > Hello, Dmitriy.
> > > > > > 
> > > > > > I'm sorry, but I don't understand your concern.
> > > > > > 
> > > > > > Vladimir just asks experienced Ignite contributor to *explain 
> > > > > > impact*
> > > > 
> > > > of
> > > > > a
> > > > > > bug.
> > > > > > 
> > > > > > Why are you scaring us with your "-1"?
> > > > > > Is it Apache Way to do so?
> > > > > > What should be done for you to return to a constructive discussion?
> > > > > > 
> > > > > > В Ср, 03/10/2018 в 00:23 +0300, Dmitriy Pavlov пишет:
> > > > > > > Hi Igniters, Vladimir,
> > > > > > > 
> > > > > > > NPEs or hangs in WAL is a completely non-functional grid (if
> > > > > 
> > > > > persistence
> > > > > > > enabled).
> > > > > > > 
> > > > > >

Re: Apache Ignite 2.7 release

2018-10-03 Thread Nikolay Izhikov
Hello, Igniters.

Release scope is frozen.
Please, if you include some new issues in release - discuss it in this thread.

Alexey, can you, please, comment on including fix for IGNITE-9760, IGNITE-9761 
in 2.7 branch.

https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3355201f3e8cafd23b2250aaf3b91b8b8ed1
https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=9d6e6ff394c05ddf7ef31a9d9ed1b492d9eeba69

В Ср, 03/10/2018 в 13:24 +0300, Vladimir Ozerov пишет:
> Nobody vetos anything, let's stop use this term unless some really
> important problem is discussed.
> 
> At this point we are in situation when new tickets are still included into
> the scope. All want to ask is to stop including new tickets without
> explaining on why they should be in AI 2.7. Regression between is AI 2.6
> and AI 2.7 is enough. But "I found new NPE" is not.
> 
> On Wed, Oct 3, 2018 at 11:10 AM Dmitriy Pavlov 
> wrote:
> 
> > Nikolay,
> > 
> > this has nothing about scaring someone. Let me explain about Apache Way.
> > 
> > Voting -1 to release does not mean blocking it, release can't be vetoed.
> > Approving release is done by policy: majority approval. 3+1 binding and
> > more +1 than -1. Consensus approval is better but not mandatory.
> > 
> > Instead, if PMC says -1 to code modification it means veto and can't be
> > bypassed to anyone. This is a very strong statement, which should be
> > applied reasonably and with technical justification. Lack of
> > understanding is not a justification.
> > 
> > So my point instead of vetoing bugfix let's veto commits where the bugs
> > were introduced. I feel a number of bugs reported recently are all
> > connected to WalManager, and these bugs may come from just a couple of
> > fixes. PDS tests were quite stable last time, so I think it is possible to
> > find out why WAL crashes and hangs.
> > 
> > Sincerely,
> > Dmitriy Pavlov
> > 
> > ср, 3 окт. 2018 г. в 10:05, Andrey Kuznetsov :
> > 
> > > Vladimir, Nikolay,
> > > 
> > > For sure, I'm not an experienced Ignite contributor, so I'm sorry for
> > > intervening. I've just run the reproducer from [1] against ignite-2.6
> > > branch and it has passed. So, it's not an legacy bug, we've brought it
> > 
> > with
> > > some change of 2.7 scope. Is it still ok to ignore the bug?
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9776
> > > 
> > > ср, 3 окт. 2018 г. в 2:07, Nikolay Izhikov :
> > > 
> > > > Hello, Dmitriy.
> > > > 
> > > > I'm sorry, but I don't understand your concern.
> > > > 
> > > > Vladimir just asks experienced Ignite contributor to *explain impact*
> > 
> > of
> > > a
> > > > bug.
> > > > 
> > > > Why are you scaring us with your "-1"?
> > > > Is it Apache Way to do so?
> > > > What should be done for you to return to a constructive discussion?
> > > > 
> > > > В Ср, 03/10/2018 в 00:23 +0300, Dmitriy Pavlov пишет:
> > > > > Hi Igniters, Vladimir,
> > > > > 
> > > > > NPEs or hangs in WAL is a completely non-functional grid (if
> > > 
> > > persistence
> > > > > enabled).
> > > > > 
> > > > > I see no reasons to release 2.7 with such symptoms until we're sure
> > 
> > it
> > > is
> > > > > too rare/impossible to reproduce. But it seems it is not the case. I
> > > 
> > > will
> > > > > definitely vote -1 for the release if I'm aware of such problems
> > 
> > exist
> > > > and
> > > > > were not researched. Community guarantees the quality and usability
> > 
> > of
> > > > the
> > > > > product.
> > > > > 
> > > > > We should ask and answer other questions:
> > > > > 1) why there are a lot of NPEs and hangs reported recently in the
> > 
> > same
> > > > area
> > > > > 2) and why we signed-off commit(s).
> > > > > 
> > > > > Probably we can identify and revert these commit(s) from 2.7 and
> > > 
> > > research
> > > > > these failures in master (with no rush).
> > > > > 
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > > 
> > > > > вт, 2 окт. 2018 г. в 23:54, Vladimir Ozerov :
> > > > > 
> > > > > &

Re: Apache Ignite 2.7 release

2018-10-02 Thread Nikolay Izhikov
 > > > >  > part is infrastructure (adopting new modules, Java 9+ support,
> > > > > 
> > > > > etc.). So
> > > > > > >  > despite big absolute number, most of these tickets are grouped
> > > > 
> > > > around
> > > > > > >  > several big areas, and overall progress over this week should
> > 
> > be
> > > > very
> > > > > > >  good.
> > > > > > >  >
> > > > > > >  > On Mon, Oct 1, 2018 at 9:50 PM Dmitriy Setrakyan <
> > > > > 
> > > > > dsetrak...@apache.org>
> > > > > > >  > wrote:
> > > > > > >  >
> > > > > > >  > > If this filter is for 2.7 release, then I do not believe all
> > > > 
> > > > these
> > > > > > >  > tickets
> > > > > > >  > > will be closed. It would be nice to leave only "must-have"
> > > > 
> > > > tickets
> > > > > in
> > > > > > >  2.7
> > > > > > >  > > and move the rest to 2.8.
> > > > > > >  > >
> > > > > > >  > > D.
> > > > > > >  > >
> > > > > > >  > > On Mon, Oct 1, 2018 at 11:02 AM Vladimir Ozerov <
> > > > > 
> > > > > voze...@gridgain.com>
> > > > > > >  > > wrote:
> > > > > > >  > >
> > > > > > >  > > > Igniters,
> > > > > > >  > > >
> > > > > > >  > > > Please use this filter, as it properly handles tickets
> > > 
> > > without
> > > > > > >  > > components:
> > > > > > >  > > >
> > > > > > >  > > >
> > > > > > >  > > >
> > > > > > >  > >
> > > > > > >  >
> > > > > > > 
> > 
> > https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20not%20in%20(documentation)))%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> > > > > > >  > > >
> > > > > > >  > > > On Mon, Oct 1, 2018 at 6:18 PM Nikolay Izhikov <
> > > > > 
> > > > > nizhi...@apache.org>
> > > > > > >  > > > wrote:
> > > > > > >  > > >
> > > > > > >  > > > > Hello, Igniters.
> > > > > > >  > > > >
> > > > > > >  > > > > I announce scope freeze for an Apache Ignite 2.7 release.
> > > > > > >  > > > >
> > > > > > >  > > > > It means:
> > > > > > >  > > > >
> > > > > > >  > > > > 1. We add to 2.7 only critical bugs.
> > > > > > >  > > > > 2. We merge to 2.7 branch only previously announces
> > > 
> > > features.
> > > > > > >  > > > > 3. I expect we should exclude or *MERGE ALL TASKS FOR 2.7
> > > 
> > > DUE
> > > > > TO
> > > > > > >  > > OCTOBER
> > > > > > >  > > > > 10*.
> > > > > > >  > > > > So the *October 10 is DEADLINE* for new features.
> > > > > > >  > > > >
> > > > > > >  > > > > Thoughts?
> > > > > > >  > > > >
> > > > > > >  > > > > For now we have 34 In Progress tickets planned to 2.7
> > > 
> > > version
> > > > > [1].
> > > > > > >  > > > > So is you assigned to any of this ticker friendly
> > 
> > reminder
> > > > #1,
> > > > > *the
> > > > > > >  > > > > deadline is near :)*.
> > > > > > >  > > > >
> > > > > > >  > > > > [1]
> > > > > > >  > > > >
> > > > > > >  > > > >
> > > > > > >  > > >
> > > > > > >  > >
> > > > > > >  >
> > > > > > > 
> > 
> > http

Re: TDE Implementation details.

2018-10-02 Thread Nikolay Izhikov
Hello, Dmitriy.

Thank you for feedback!

> 1) I suggest changing encrypt() and decrypt() method would take the same type 
> of parameters. 
> This uniformity will avoid implementors questions about why encrypt takes 
> byte[] but decrypt takes ByteBuffer.

This is done by a reason.
encrypt and decrypt methods are on hot code path, so it important to avoid 
unnecessary type conversion.
We use ByteBuffer in places where we already operate with ByteBuffer and byte[] 
where we have it.

Anyway, ByteBuffer can be easily converted to byte[] and vice versa.

> 2) I suggest the renaming of method create()  to createKey() to make it> easy 
> to understand what method creates.> 

We already discussed new SPI and approved it with reviewers.
Anyway it's pretty easy to rename method, but I don't think I should do it at 
the moment.
All other reviewers are OK with simple `create`.

В Вт, 02/10/2018 в 17:34 +0300, Dmitriy Pavlov пишет:
> Hi Igniters,
> 
> I'm sorry I a little bit late to the party, but I found a couple of hours
> to take a look to TDE-1 implementation
> https://issues.apache.org/jira/browse/IGNITE-8485 . Nikolay kindly agreed
> to let me take a look.
> 
> Most of the change was done well and I totally agree with it. Current
> implementation seems to be extendable.
> 
> I've got several minor comments, I hope it will be fixed.
> https://github.com/apache/ignite/pull/4167
> Related to tests: let's make sure we won't fail new tests and compilation
> pass for all modules. I will be happy to help with failures analysis if it
> is required.
> 
> Proposals related to API
> 1) I suggest changing encrypt() and decrypt() method would take the same
> type of parameters. This uniformity will avoid implementors questions about
> why encrypt takes byte[] but decrypt takes ByteBuffer.
> 2) I suggest the renaming of method create()  to createKey() to make it
> easy to understand what method creates.
> 
> 
> Sincerely,
> Dmitriy Pavlov
> 
> ср, 12 сент. 2018 г. в 20:34, Denis Magda :
> 
> > Hello Nikolay,
> > 
> > Excellent progress, look forward to seeing the TDE released in 2.7!
> > 
> > --
> > Denis
> > 
> > On Wed, Sep 12, 2018 at 2:47 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Denis.
> > > 
> > > > Could you please list the limitations of Phase 1?
> > > > I'm curious what won't be supported in 2.7.
> > > 
> > > 1. We will have ability to encrypt data stored on the disk for specific
> > > cache.
> > > 
> > > - There is no limitation on API usage or something for an
> > > encrypted cache.
> > > - If some cache in a cache group is encrypted, all other caches
> > 
> > in
> > > this group must be encrypted.
> > > - Setting up master key(standard jdk key storage) is prerequisite
> > > and should be done by an administrator.
> > > - `encryptionEnabled` flag setting up on cache creation and can't
> > > be changed in runtime.
> > > 
> > > 2. We won't be able to change encryption keys for existing cache(key
> > > rotation procedure).
> > > This will be implemented in Phase 2.
> > > 
> > > В Вт, 11/09/2018 в 23:41 -0400, Denis Magda пишет:
> > > > Nikolay,
> > > > 
> > > > Could you please list the limitations of Phase 1? I'm curious what
> > 
> > won't
> > > be
> > > > supported in 2.7.
> > > > 
> > > > --
> > > > Denis
> > > > 
> > > > On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov 
> > > 
> > > wrote:
> > > > 
> > > > > > As I understand the plan is to get TDE Phase 1 released in 2.7,
> > > 
> > > right?
> > > > > 
> > > > > Yes. We will release TDE in 2.7
> > > > > 
> > > > > > Could you also list what needs to be done at Phase 2 and how much
> > > 
> > > time
> > > > > 
> > > > > it might take.
> > > > > 
> > > > > Yes, I think Dmitry Ryabov will send Phase 2 design
> > > > > 
> > > > > 
> > > > > В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > > > > > Hello, Denis.
> > > > > > 
> > > > > > Yes, Vladimir made 2 rounds of review.
> > > > > > I planning to fix all issues with implementation in a few days.
> > > > > > 
> > > > > > 
> > > > > > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda

Re: Release process documentation in wiki

2018-10-02 Thread Nikolay Izhikov
Dmitriy, thank you.

I also making some notes about release.
Feedback are welcome.

https://cwiki.apache.org/confluence/display/IGNITE/Release+manager+Notes

В Вт, 02/10/2018 в 14:42 +0300, Dmitriy Pavlov пишет:
> Hi Igniters,
> 
> We had several hot discussions related to release, and one from reasons we
> need time to come to consensus was my poor understanding of the process.
> 
> I decided to contribute currently found facts as a scheme of release
> process here
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
> 
> Please find a minute and take a look. Feedback, as always, is more than
> welcome.
> 
> Sincerely,
> Dmitriy Pavlov


signature.asc
Description: This is a digitally signed message part


Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

2018-10-01 Thread Nikolay Izhikov
Hello, Andrey.

Yes, I know it.
I've looked at the PR befor mailing :)

Do you think we can include this improvement to the 2.7 release?
Do you have time to assist Saikat with TC setup and other tasks?


В Пн, 01/10/2018 в 19:54 +0300, Andrey Mashenkov пишет:
> Dmitry, Nikolay,
> 
> Ignite-3303 is a new Ignite module and there is no changes related to core or 
> other existed modules. 
> So, PR should not affected existed functional ity and can be safely merged.
> 
> Thanks.
> 
> 
> пн, 1 окт. 2018 г., 16:04 Dmitriy Pavlov :
> > Hi Saikat,
> > 
> > I don't mind merging to master, but I have concern if it will go to 2.7. In
> > the separate discussion, we agreed on code freeze should happen during last
> > weekend, September, 30.
> > 
> > So it is now up to community and release manager to decide if fix should go
> > to the upcoming release. Usually, after the freeze, only bug/test fixes can
> > be merged to release branch.
> > 
> > Hi Nikolay,
> > 
> > could you please announce that code freeze happened?
> > 
> > Sincerely,
> > Dmitriy Pavlov
> > 
> > пн, 1 окт. 2018 г. в 3:58, Saikat Maitra :
> > 
> > > Hi Alex, Nicolay
> > >
> > > As discussed with Andrew the changes looks good. Would it be ok to merge
> > > this change to master considering the 2.7 release plan?
> > >
> > > Regards,
> > > Saikat
> > >
> > > On Fri, Sep 28, 2018 at 7:15 PM Saikat Maitra 
> > > wrote:
> > >
> > > > Thank you Andrew
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > > On Fri, Sep 28, 2018 at 7:00 PM Andrey Mashenkov <
> > > > andrey.mashen...@gmail.com> wrote:
> > > >
> > > >> Hi Saikat,
> > > >>
> > > >> Sorry for late answer. I've checked changes a day ago. Now, looks good.
> > > >> Hope, it will be merged soon.
> > > >>
> > > >> Alex, would you please merge PR to master.
> > > >>
> > > >> сб, 29 сент. 2018 г., 2:29 Saikat Maitra :
> > > >>
> > > >> > Hi Andrew,
> > > >> >
> > > >> > I have updated the changes.
> > > >> >
> > > >> > Can you please review and share feedback.
> > > >> >
> > > >> > Regards
> > > >> > Saikat
> > > >> >
> > > >> > On Sat, Sep 22, 2018 at 2:23 PM Saikat Maitra <
> > > saikat.mai...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Andrew
> > > >> > >
> > > >> > >
> > > >> > > I have updated the changes.
> > > >> > >
> > > >> > >
> > > >> > > Can you please review and share feedback.
> > > >> > >
> > > >> > >
> > > >> > > Regards
> > > >> > > Saikat
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Sep 19, 2018 at 8:11 PM, Saikat Maitra <
> > > >> saikat.mai...@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi Andrew,
> > > >> > >>
> > > >> > >> I have updated the tests and also added java docs.
> > > >> > >>
> > > >> > >> Can you please review and share feedback.
> > > >> > >>
> > > >> > >>
> > > >> > >> Regards
> > > >> > >> Saikat
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> On Sun, Sep 16, 2018 at 11:53 AM, Saikat Maitra <
> > > >> > saikat.mai...@gmail.com>
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >>> Hi Andrew,
> > > >> > >>>
> > > >> > >>> I have updated the tests and also added java docs.
> > > >> > >>>
> > > >> > >>> Please review and share feedback.
> > > >> > >>>
> > > >> > >>> Regards
> > > >> > >>> Saikat
> > > >> > >>>
> > > >> > >>>
> > > >> > >>> On Sat, Sep 8, 2018 at 2:09 PM, Saikat Maitra <
> > > >> saikat.mai...@gmail.com
> > > >> > >
> > > >> > >>> wrote:
> > > >> > >>>
> > > >> >  Hi Andrew, Alexey
> > > >> > 
> > > >> >  I have incorporated the review changes.
> > > >> > 
> > > >> >  I have also refactored the CacheEventSerializer class and moved
> > > it
> > > >> to
> > > >> >  test folder because it is used only in the
> > > >> > FlinkIgniteSourceSelfExample and
> > > >> >  not required for IgniteSource.
> > > >> > 
> > > >> >  Build links
> > > >> > https://ci.ignite.apache.org/viewLog.html?buildId=1821778;;
> > > >> > 
> > > >> >  https://ci.ignite.apache.org/viewLog.html?buildId=1821774;;
> > > >> > 
> > > >> >  Please review and share feedback.
> > > >> > 
> > > >> >  Regards
> > > >> >  Saikat
> > > >> > 
> > > >> >  On Tue, Sep 4, 2018 at 9:57 PM, Saikat Maitra <
> > > >> > saikat.mai...@gmail.com>
> > > >> >  wrote:
> > > >> > 
> > > >> > > Hi Alexey,
> > > >> > >
> > > >> > > Thank you for reviewing the changes and sharing feedback, I am
> > > >> > > updating the PR. I will share the changes shortly.
> > > >> > >
> > > >> > > Regards,
> > > >> > > Saikat
> > > >> > >
> > > >> > > On Tue, Sep 4, 2018 at 10:59 AM, Alexey Goncharuk <
> > > >> > > alexey.goncha...@gmail.com> wrote:
> > > >> > >
> > > >> > >> Hello Saikat,
> > > >> > >>
> > > >> > >> I see a few fellow Igniters added some comments to your PR
> > > >> > (including
> > > >> > >> me).
> > > >> > >> I believe the PR can be merged after you address them.
> > > >> > >>
> > > >> > >> Thanks,
> > > >> > >> 

Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

2018-10-01 Thread Nikolay Izhikov
Alex, Saikat.

As I can see there is work to be done to merge this feature.
So I think it's better to exclude it from 2.7.

Thoughts?

В Пн, 01/10/2018 в 18:35 +0300, Alexey Goncharuk пишет:
> Hello Saikat,
> 
> I am ok with the prod code changes, but I am a bit confused with the
> example being added to the tests folder. I think it should be either added
> to the examples (not sure about the dependency though), or should not be
> added at all. Also, I see that you added a new suite, has it been added to
> a TC configuration?
> 
> 
> 
> пн, 1 окт. 2018 г. в 16:36, Nikolay Izhikov :
> 
> > Hello, Saikat.
> > 
> > I have no objections to include this integration to 2.7 release.
> > But, we should ask for a final review from Alex Goncharyuk.
> > 
> > Alex, can you comment on this?
> > Is this patch ready to be merged?
> > Do you see any risks to include it to 2.7 release?
> > 
> > 
> > В Вс, 30/09/2018 в 19:57 -0500, Saikat Maitra пишет:
> > > Hi Alex, Nicolay
> > > 
> > > As discussed with Andrew the changes looks good. Would it be ok to merge
> > > this change to master considering the 2.7 release plan?
> > > 
> > > Regards,
> > > Saikat
> > > 
> > > On Fri, Sep 28, 2018 at 7:15 PM Saikat Maitra 
> > > wrote:
> > > 
> > > > Thank you Andrew
> > > > 
> > > > Regards,
> > > > Saikat
> > > > 
> > > > On Fri, Sep 28, 2018 at 7:00 PM Andrey Mashenkov <
> > > > andrey.mashen...@gmail.com> wrote:
> > > > 
> > > > > Hi Saikat,
> > > > > 
> > > > > Sorry for late answer. I've checked changes a day ago. Now, looks
> > 
> > good.
> > > > > Hope, it will be merged soon.
> > > > > 
> > > > > Alex, would you please merge PR to master.
> > > > > 
> > > > > сб, 29 сент. 2018 г., 2:29 Saikat Maitra :
> > > > > 
> > > > > > Hi Andrew,
> > > > > > 
> > > > > > I have updated the changes.
> > > > > > 
> > > > > > Can you please review and share feedback.
> > > > > > 
> > > > > > Regards
> > > > > > Saikat
> > > > > > 
> > > > > > On Sat, Sep 22, 2018 at 2:23 PM Saikat Maitra <
> > 
> > saikat.mai...@gmail.com>
> > > > > > wrote:
> > > > > > 
> > > > > > > Hi Andrew
> > > > > > > 
> > > > > > > 
> > > > > > > I have updated the changes.
> > > > > > > 
> > > > > > > 
> > > > > > > Can you please review and share feedback.
> > > > > > > 
> > > > > > > 
> > > > > > > Regards
> > > > > > > Saikat
> > > > > > > 
> > > > > > > 
> > > > > > > On Wed, Sep 19, 2018 at 8:11 PM, Saikat Maitra <
> > > > > 
> > > > > saikat.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi Andrew,
> > > > > > > > 
> > > > > > > > I have updated the tests and also added java docs.
> > > > > > > > 
> > > > > > > > Can you please review and share feedback.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Regards
> > > > > > > > Saikat
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > On Sun, Sep 16, 2018 at 11:53 AM, Saikat Maitra <
> > > > > > 
> > > > > > saikat.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi Andrew,
> > > > > > > > > 
> > > > > > > > > I have updated the tests and also added java docs.
> > > > > > > > > 
> > > > > > > > > Please review and share feedback.
> > > > > > > > > 
> > > > > > > > > Regards
> > > > > > > > > Saikat
> > > > > > > > > 
>

Re: Apache Ignite 2.7 release

2018-10-01 Thread Nikolay Izhikov
Hello, Igniters.

I announce scope freeze for an Apache Ignite 2.7 release.

It means:

1. We add to 2.7 only critical bugs.
2. We merge to 2.7 branch only previously announces features.
3. I expect we should exclude or *MERGE ALL TASKS FOR 2.7 DUE TO OCTOBER 10*.
So the *October 10 is DEADLINE* for new features.

Thoughts?

For now we have 34 In Progress tickets planned to 2.7 version [1].
So is you assigned to any of this ticker friendly reminder #1, *the deadline is 
near :)*.

[1] 
https://issues.apache.org/jira/browse/IGNITE-8803?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20component%20!%3D%20documentation%20)%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20



В Пн, 01/10/2018 в 16:13 +0300, Andrey Gura пишет:
> Agree with Andrey.
> 
> IGNITE-9723 will be merged to ignite-2.7 branch soon.
> On Mon, Oct 1, 2018 at 3:56 PM Andrey Kuznetsov  wrote:
> > 
> > Igniters,
> > 
> > There is an inaccuracy in critical worker termination detection, and I'm
> > working on a fix right now, see [1]. Also, we have trivial yet important
> > fix [2], this one is ready to merge.
> > 
> > I deem both should get to 2.7. Any objections?
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-9744
> > [2] https://issues.apache.org/jira/browse/IGNITE-9723
> > 
> > пн, 1 окт. 2018 г. в 13:13, Dmitriy Pavlov :
> > 
> > > Folks,
> > > 
> > > We can consider this thread as an almost healthy discussion about SG
> > > inclusion to 2.7.
> > > 
> > > For the future, I encourage all Igniters to discuss things before 
> > > agreement
> > > on dev.list instead of trying to log 'some pre-build agreements' here. BTW
> > > such logged 'agreements' are not always valid.
> > > 
> > > Dev list is not a log of community actions; it is the principal place 
> > > where
> > > discussion runs. I sincerely hope all similar decisions will be building 
> > > on
> > > the dev list.
> > > 
> > > Sincerely,
> > > Dmitriy Pavlov
> > > 
> > > вс, 30 сент. 2018 г. в 10:49, Nikolay Izhikov :
> > > 
> > > > Hello, guys.
> > > > 
> > > > I moved SG tasks to 2.8.
> > > > 
> > > > В Сб, 29/09/2018 в 17:11 +0300, Vladimir Ozerov пишет:
> > > > > I fully support Dmitry's concerns. I do not see a single chance for
> > > > 
> > > > Service
> > > > > Grid to be included into 2.7 scope, but we are still keeping hope
> > > 
> > > alive.
> > > > > What for? Our experience shows, that any feature involving consensus
> > > > > algorithms typically take weeks to be finalized after initial public
> > > > > review. Yes, it is possible that at some day in October we will see it
> > > > > completed, but what about testing? What about regressions with other
> > > > > components? Let me reming you that from Monday a lot of efforts will 
> > > > > be
> > > > 
> > > > put
> > > > > into project stabilization. This is the sole purpose of that
> > > 
> > > pre-release
> > > > > phase. Accepting a patch with deep rework of Ignite internals in the
> > > > 
> > > > middle
> > > > > of that process, means that our effrots will be lost. This is simply a
> > > > > matter of respect to contributor's time.
> > > > > 
> > > > > Another problem is that this false hope puts us at rush. Rush during
> > > > > design, development, review, testing. Result is known - bad features,
> > > > 
> > > > which
> > > > > makes damage to the project.
> > > > > 
> > > > > So my question is - why don't we just want to move it to AI 2.8 right
> > > > 
> > > > now?
> > > > > Feature is big, feature is very far from being ready. This simple
> > > 
> > > action
> > > > > immediately shifts the focus from dates to quality of the product, and
> > > > > remove any risks that potential merge will defeat stabilization 
> > > > > effrots
> > > > 
> > > > of
> > > > > other contributors.
> > > > > 
> > > > > 
> > > > > 
> > > > > On Sat, Sep 29, 2018 at 8:32 AM Vyacheslav Daradur <
> > > 
> > > daradu...@gmail.com>
&g

Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

2018-10-01 Thread Nikolay Izhikov
Hello, Saikat.

I have no objections to include this integration to 2.7 release.
But, we should ask for a final review from Alex Goncharyuk.

Alex, can you comment on this?
Is this patch ready to be merged?
Do you see any risks to include it to 2.7 release?


В Вс, 30/09/2018 в 19:57 -0500, Saikat Maitra пишет:
> Hi Alex, Nicolay
> 
> As discussed with Andrew the changes looks good. Would it be ok to merge
> this change to master considering the 2.7 release plan?
> 
> Regards,
> Saikat
> 
> On Fri, Sep 28, 2018 at 7:15 PM Saikat Maitra 
> wrote:
> 
> > Thank you Andrew
> > 
> > Regards,
> > Saikat
> > 
> > On Fri, Sep 28, 2018 at 7:00 PM Andrey Mashenkov <
> > andrey.mashen...@gmail.com> wrote:
> > 
> > > Hi Saikat,
> > > 
> > > Sorry for late answer. I've checked changes a day ago. Now, looks good.
> > > Hope, it will be merged soon.
> > > 
> > > Alex, would you please merge PR to master.
> > > 
> > > сб, 29 сент. 2018 г., 2:29 Saikat Maitra :
> > > 
> > > > Hi Andrew,
> > > > 
> > > > I have updated the changes.
> > > > 
> > > > Can you please review and share feedback.
> > > > 
> > > > Regards
> > > > Saikat
> > > > 
> > > > On Sat, Sep 22, 2018 at 2:23 PM Saikat Maitra 
> > > > wrote:
> > > > 
> > > > > Hi Andrew
> > > > > 
> > > > > 
> > > > > I have updated the changes.
> > > > > 
> > > > > 
> > > > > Can you please review and share feedback.
> > > > > 
> > > > > 
> > > > > Regards
> > > > > Saikat
> > > > > 
> > > > > 
> > > > > On Wed, Sep 19, 2018 at 8:11 PM, Saikat Maitra <
> > > 
> > > saikat.mai...@gmail.com>
> > > > > wrote:
> > > > > 
> > > > > > Hi Andrew,
> > > > > > 
> > > > > > I have updated the tests and also added java docs.
> > > > > > 
> > > > > > Can you please review and share feedback.
> > > > > > 
> > > > > > 
> > > > > > Regards
> > > > > > Saikat
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > On Sun, Sep 16, 2018 at 11:53 AM, Saikat Maitra <
> > > > 
> > > > saikat.mai...@gmail.com>
> > > > > > wrote:
> > > > > > 
> > > > > > > Hi Andrew,
> > > > > > > 
> > > > > > > I have updated the tests and also added java docs.
> > > > > > > 
> > > > > > > Please review and share feedback.
> > > > > > > 
> > > > > > > Regards
> > > > > > > Saikat
> > > > > > > 
> > > > > > > 
> > > > > > > On Sat, Sep 8, 2018 at 2:09 PM, Saikat Maitra <
> > > 
> > > saikat.mai...@gmail.com
> > > > > 
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi Andrew, Alexey
> > > > > > > > 
> > > > > > > > I have incorporated the review changes.
> > > > > > > > 
> > > > > > > > I have also refactored the CacheEventSerializer class and moved 
> > > > > > > > it
> > > 
> > > to
> > > > > > > > test folder because it is used only in the
> > > > 
> > > > FlinkIgniteSourceSelfExample and
> > > > > > > > not required for IgniteSource.
> > > > > > > > 
> > > > > > > > Build links
> > > > 
> > > > https://ci.ignite.apache.org/viewLog.html?buildId=1821778;;
> > > > > > > > 
> > > > > > > > https://ci.ignite.apache.org/viewLog.html?buildId=1821774;;
> > > > > > > > 
> > > > > > > > Please review and share feedback.
> > > > > > > > 
> > > > > > > > Regards
> > > > > > > > Saikat
> > > > > > > > 
> > > > > > > > On Tue, Sep 4, 2018 at 9:57 PM, Saikat Maitra <
> > > > 
> > > > saikat.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi Alexey,
> > > > > > > > > 
> > > > > > > > > Thank you for reviewing the changes and sharing feedback, I am
> > > > > > > > > updating the PR. I will share the changes shortly.
> > > > > > > > > 
> > > > > > > > > Regards,
> > > > > > > > > Saikat
> > > > > > > > > 
> > > > > > > > > On Tue, Sep 4, 2018 at 10:59 AM, Alexey Goncharuk <
> > > > > > > > > alexey.goncha...@gmail.com> wrote:
> > > > > > > > > 
> > > > > > > > > > Hello Saikat,
> > > > > > > > > > 
> > > > > > > > > > I see a few fellow Igniters added some comments to your PR
> > > > 
> > > > (including
> > > > > > > > > > me).
> > > > > > > > > > I believe the PR can be merged after you address them.
> > > > > > > > > > 
> > > > > > > > > > Thanks,
> > > > > > > > > > AG
> > > > > > > > > > 
> > > > > > > > > > пт, 31 авг. 2018 г. в 3:11, Saikat Maitra <
> > > 
> > > saikat.mai...@gmail.com
> > > > > :
> > > > > > > > > > 
> > > > > > > > > > > Thank you, Denis
> > > > > > > > > > > 
> > > > > > > > > > > Regards,
> > > > > > > > > > > Saikat
> > > > > > > > > > > 
> > > > > > > > > > > On Thu, Aug 30, 2018 at 7:01 PM, Denis Magda <
> > > 
> > > dma...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hello Saikat,
> > > > > > > > > > > > 
> > > > > > > > > > > > Hopefully, someone from the community will review the
> > > 
> > > changes in
> > > > > > > > > > the
> > > > > > > > > > > > nearest time.
> > > > > > > > > > > > 
> > > > > > > > > > > > --
> > > > > > > > > > > > Denis
> > > > > > > > > > > > 
> > > > > > > > > > > > On Thu, Aug 30, 2018 at 4:37 PM Saikat Maitra <
> > > > > > > > > > 
> > > > > > > > > > saikat.mai...@gmail.com>
> > > > > 

Re: Apache Ignite 2.7 release

2018-09-30 Thread Nikolay Izhikov
Hello, guys.

I moved SG tasks to 2.8.

В Сб, 29/09/2018 в 17:11 +0300, Vladimir Ozerov пишет:
> I fully support Dmitry's concerns. I do not see a single chance for Service
> Grid to be included into 2.7 scope, but we are still keeping hope alive.
> What for? Our experience shows, that any feature involving consensus
> algorithms typically take weeks to be finalized after initial public
> review. Yes, it is possible that at some day in October we will see it
> completed, but what about testing? What about regressions with other
> components? Let me reming you that from Monday a lot of efforts will be put
> into project stabilization. This is the sole purpose of that pre-release
> phase. Accepting a patch with deep rework of Ignite internals in the middle
> of that process, means that our effrots will be lost. This is simply a
> matter of respect to contributor's time.
> 
> Another problem is that this false hope puts us at rush. Rush during
> design, development, review, testing. Result is known - bad features, which
> makes damage to the project.
> 
> So my question is - why don't we just want to move it to AI 2.8 right now?
> Feature is big, feature is very far from being ready. This simple action
> immediately shifts the focus from dates to quality of the product, and
> remove any risks that potential merge will defeat stabilization effrots of
> other contributors.
> 
> 
> 
> On Sat, Sep 29, 2018 at 8:32 AM Vyacheslav Daradur 
> wrote:
> 
> > Dmitriy,
> > 
> > Hot redeployment and versioning will not be implemented in phase 1,
> > but it is scheduled once it is finished.
> > 
> > Here is an umbrella ticket [1] to track phase 1 scope.
> > 
> > It includes very few new features, but we completely rework component
> > to improve guarantees to be more reliable and we build the base for
> > new features.
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > On Fri, Sep 28, 2018 at 9:38 PM Dmitriy Setrakyan 
> > wrote:
> > > 
> > > I am not sure I can agree. SG redesign includes:
> > > - hot redeployment
> > > - versioning
> > > 
> > > In my experience, features like these take about 1 month to test properly
> > > and fix all the bugs, including redeployment tests and restart tests on
> > > larger topologies, together with overnight runs. If this type of testing
> > > has not been performed, I think it would be unreasonable to expect this
> > > feature making it into the release.
> > > 
> > > Can someone comment on the testing?
> > > 
> > > D.
> > > 
> > > 
> > > On Fri, Sep 28, 2018 at 10:38 AM Dmitriy Pavlov 
> > > wrote:
> > > 
> > > > Nikolay, because I think you're a do'er, but not a commenter, like me,
> > 
> > for
> > > > example, I can trust your opinion.
> > > > 
> > > > I will join review if I have spare cycles.
> > > > 
> > > > пт, 28 сент. 2018 г. в 20:34, Denis Magda :
> > > > 
> > > > > Nikolay,
> > > > > 
> > > > > Thanks for stepping in and giving more context. In general, I'm
> > 
> > fully for
> > > > > your proposal below:
> > > > > 
> > > > > My vote goes to option *a*.
> > > > > > I think we should release 2.7 with the bunch of new cool features.
> > > > > > *AND* we should plan 2.8 release at the end of the year with SG
> > > > 
> > > > redesign
> > > > > > and MVCC stabilization tasks.
> > > > > 
> > > > > 
> > > > > --
> > > > > Denis
> > > > > 
> > > > > On Fri, Sep 28, 2018 at 10:30 AM Nikolay Izhikov <
> > 
> > nizhi...@apache.org>
> > > > > wrote:
> > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > I think we shouldn't put so many emotions in discussion of any
> > > > > > contribution.
> > > > > > Even so big and important as SG redesign.
> > > > > > 
> > > > > > The crucial point we all agreed about: Service Grid redesign a big
> > > > > 
> > > > > feature
> > > > > > that can significally improve Ignite.
> > > > > > We all want to have it in the product.
> > > > > > 
> > > > > > Let me write my vision of the current task state:
> > > > > > 
> > > > > > 1

Re: Apache Ignite 2.7 release

2018-09-28 Thread Nikolay Izhikov
t; > > 
> > > > > > Exactly. So correct statement is “it is up to *community* to decide
> > > > > 
> > > > > whether
> > > > > > something goes to 2.7”.
> > > > > > 
> > > > > > пт, 28 сент. 2018 г. в 11:11, Dmitriy Pavlov <
> > 
> > dpavlov@gmail.com
> > > > :
> > > > > > 
> > > > > > > No, it is up to the community to discuss after their review
> > > 
> > > results.
> > > > > > > 
> > > > > > > пт, 28 сент. 2018 г. в 11:09, Vladimir Ozerov <
> > > 
> > > voze...@gridgain.com
> > > > > :
> > > > > > > 
> > > > > > > > Dmitriy,
> > > > > > > > 
> > > > > > > > Did I read your words correctly that it is up to implementor
> > 
> > of a
> > > > > > single
> > > > > > > > feature to decide whether release of all other features and
> > 
> > fixes
> > > > to
> > > > > be
> > > > > > > > delayed?
> > > > > > > > 
> > > > > > > > пт, 28 сент. 2018 г. в 11:00, Dmitriy Pavlov <
> > > > 
> > > > dpavlov@gmail.com
> > > > > > :
> > > > > > > > 
> > > > > > > > > My point we can wait a bit for services because
> > > > > > > > > 1  we are open-minded and we don't have outside pressure to
> > 
> > do
> > > > > > release
> > > > > > > in
> > > > > > > > > October
> > > > > > > > > 2  and services it is not some new feature, which suddenly
> > > > 
> > > > appeared
> > > > > > in
> > > > > > > > > autumn, it is a well known and important feature.
> > > > > > > > > 
> > > > > > > > > So it is up to Vyacheslav, Anton and Nikolay to decide.
> > > > > > > > > 
> > > > > > > > > Decisions can be services are not ready/ready to merge only
> > 
> > to
> > > > > > > > master/ready
> > > > > > > > > to merge to master and to 2.7.
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > пт, 28 сент. 2018 г. в 10:46, Vladimir Ozerov <
> > > > > 
> > > > > voze...@gridgain.com
> > > > > > > :
> > > > > > > > > 
> > > > > > > > > > Dmitry,
> > > > > > > > > > 
> > > > > > > > > > Community agreement was to perform the release in October.
> > 
> > Of
> > > > > > course
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > wait a bit for services. Then we wait a bit for other cool
> > > > > 
> > > > > features
> > > > > > > > ready
> > > > > > > > > > by that time, then again and again, and release will never
> > > > > 
> > > > > happen.
> > > > > > > And
> > > > > > > > > > while we are waiting for new features to come, already
> > > > 
> > > > completerd
> > > > > > > > > features
> > > > > > > > > > cannot be used by anyone.
> > > > > > > > > > 
> > > > > > > > > > This is why we have an agreement that if feature is not
> > > 
> > > ready,
> > > > it
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > moved to future release, instead of shifting release. The
> > > 
> > > sole
> > > > > > reason
> > > > > > > > to
> > > > > > > > > > have strict dates when decisions are made is to let release
> > > > > 
> > > > > happen.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > On Fri, Sep 28, 2018 at 2:22 AM Dmitriy Pavlov <
&

Re: Critical worker threads liveness checking drawbacks

2018-09-28 Thread Nikolay Izhikov
Ticket created - https://issues.apache.org/jira/browse/IGNITE-9737

Fixed version is 2.7.

В Пт, 28/09/2018 в 11:41 +0300, Alexey Goncharuk пишет:
> Nikolay, I agree, a user should be able to disable both thread liveness
> check and checkpoint read lock timeout check from config and a system
> property.
> 
> пт, 28 сент. 2018 г. в 11:30, Nikolay Izhikov :
> 
> > Hello, Igniters.
> > 
> > I found that this feature can't be disabled from config.
> > The only way to disable it is from JMX bean.
> > 
> > I think it very dangerous: If we have some corner case or a bug in this
> > Watch Dog it can make Ignite unusable.
> > I propose to implement possibility to disable this feature both - from
> > config and from JVM options.
> > 
> > What do you think?
> > 
> > В Чт, 27/09/2018 в 16:14 +0300, Andrey Kuznetsov пишет:
> > > Maxim,
> > > 
> > > Thanks for being attentive! It's definitely a typo. Could you please
> > 
> > create
> > > an issue?
> > > 
> > > чт, 27 сент. 2018 г. в 16:00, Maxim Muzafarov :
> > > 
> > > > Folks,
> > > > 
> > > > I've found in `GridCachePartitionExchangeManager:2684` [1] (master
> > 
> > branch)
> > > > exchange future wrapped
> > > > with double `blockingSectionEnd` method. Is it correct? I just want to
> > > > understand this change and
> > > > how should I use this in the future.
> > > > 
> > > > Should I file a new issue to fix this? I think here
> > 
> > `blockingSectionBegin`
> > > > method should be used.
> > > > 
> > > > -
> > > > blockingSectionEnd();
> > > > 
> > > > try {
> > > > resVer = exchFut.get(exchTimeout, TimeUnit.MILLISECONDS);
> > > > } finally {
> > > > blockingSectionEnd();
> > > > }
> > > > 
> > > > 
> > > > [1]
> > > > 
> > > > 
> > 
> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java#L2684
> > > > 
> > > > On Wed, 26 Sep 2018 at 22:47 Vyacheslav Daradur 
> > > > wrote:
> > > > 
> > > > > Andrey Gura, thank you for the answer!
> > > > > 
> > > > > I agree that wrapping of 'init' method reduces the profit of watchdog
> > > > > service in case of PME worker, but in other cases, we should wrap all
> > > > > possible long sections on GridDhtPartitionExchangeFuture. For example
> > > > > 'onCacheChangeRequest' method or
> > > > > 'cctx.affinity().onCacheChangeRequest' inside because it may take
> > > > > significant time (reproducer attached).
> > > > > 
> > > > > I only want to point out a possible issue which may allow to end-user
> > > > > halt the Ignite cluster accidentally.
> > > > > 
> > > > > I'm sure that PME experts know how to fix this issue properly.
> > > > > On Wed, Sep 26, 2018 at 10:28 PM Andrey Gura 
> > 
> > wrote:
> > > > > > 
> > > > > > Vyacheslav,
> > > > > > 
> > > > > > Exchange worker is strongly tied with
> > > > > > GridDhtPartitionExchangeFuture#init and it is ok. Exchange worker
> > 
> > also
> > > > > > shouldn't be blocked for long time but in reality it happens.It
> > 
> > also
> > > > > > means that your change doesn't make sense.
> > > > > > 
> > > > > > What actually make sense it is identification of places which
> > > > > > intentionally blocking. May be some places/actions should be
> > 
> > braced by
> > > > > > blocking guards.
> > > > > > 
> > > > > > If you have failing tests please make sure that your
> > 
> > failureHandler is
> > > > > > NoOpFailureHandler or any other handler with ignoreFailureTypes =
> > > > > > [CRITICAL_WORKER_BLOCKED].
> > > > > > 
> > > > > > 
> > > > > > On Wed, Sep 26, 2018 at 9:43 PM Vyacheslav Daradur <
> > > > 
> > > > daradu...@gmail.com>
> > > > > wrote:
> > > > > > > 
> > > > > > > Hi Igniters!
> > > > > > > 
> > > > > > >

[jira] [Created] (IGNITE-9737) Ignite WatchDog service should be configurable

2018-09-28 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9737:
---

 Summary: Ignite WatchDog service should be configurable
 Key: IGNITE-9737
 URL: https://issues.apache.org/jira/browse/IGNITE-9737
 Project: Ignite
  Issue Type: Bug
Reporter: Nikolay Izhikov
 Fix For: 2.7


At the moment, there is no way to disable Ignite WatchDog service from config 
or JVM option.
In any corner case or bug in that feature Ignite can become fully unusable due 
to unpredictable shutdown.

We should provide a way to enable/disable this feature from config or from JVM 
option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Critical worker threads liveness checking drawbacks

2018-09-28 Thread Nikolay Izhikov
Hello, Igniters.

I found that this feature can't be disabled from config.
The only way to disable it is from JMX bean.

I think it very dangerous: If we have some corner case or a bug in this Watch 
Dog it can make Ignite unusable.
I propose to implement possibility to disable this feature both - from config 
and from JVM options.

What do you think?

В Чт, 27/09/2018 в 16:14 +0300, Andrey Kuznetsov пишет:
> Maxim,
> 
> Thanks for being attentive! It's definitely a typo. Could you please create
> an issue?
> 
> чт, 27 сент. 2018 г. в 16:00, Maxim Muzafarov :
> 
> > Folks,
> > 
> > I've found in `GridCachePartitionExchangeManager:2684` [1] (master branch)
> > exchange future wrapped
> > with double `blockingSectionEnd` method. Is it correct? I just want to
> > understand this change and
> > how should I use this in the future.
> > 
> > Should I file a new issue to fix this? I think here `blockingSectionBegin`
> > method should be used.
> > 
> > -
> > blockingSectionEnd();
> > 
> > try {
> > resVer = exchFut.get(exchTimeout, TimeUnit.MILLISECONDS);
> > } finally {
> > blockingSectionEnd();
> > }
> > 
> > 
> > [1]
> > 
> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java#L2684
> > 
> > On Wed, 26 Sep 2018 at 22:47 Vyacheslav Daradur 
> > wrote:
> > 
> > > Andrey Gura, thank you for the answer!
> > > 
> > > I agree that wrapping of 'init' method reduces the profit of watchdog
> > > service in case of PME worker, but in other cases, we should wrap all
> > > possible long sections on GridDhtPartitionExchangeFuture. For example
> > > 'onCacheChangeRequest' method or
> > > 'cctx.affinity().onCacheChangeRequest' inside because it may take
> > > significant time (reproducer attached).
> > > 
> > > I only want to point out a possible issue which may allow to end-user
> > > halt the Ignite cluster accidentally.
> > > 
> > > I'm sure that PME experts know how to fix this issue properly.
> > > On Wed, Sep 26, 2018 at 10:28 PM Andrey Gura  wrote:
> > > > 
> > > > Vyacheslav,
> > > > 
> > > > Exchange worker is strongly tied with
> > > > GridDhtPartitionExchangeFuture#init and it is ok. Exchange worker also
> > > > shouldn't be blocked for long time but in reality it happens.It also
> > > > means that your change doesn't make sense.
> > > > 
> > > > What actually make sense it is identification of places which
> > > > intentionally blocking. May be some places/actions should be braced by
> > > > blocking guards.
> > > > 
> > > > If you have failing tests please make sure that your failureHandler is
> > > > NoOpFailureHandler or any other handler with ignoreFailureTypes =
> > > > [CRITICAL_WORKER_BLOCKED].
> > > > 
> > > > 
> > > > On Wed, Sep 26, 2018 at 9:43 PM Vyacheslav Daradur <
> > 
> > daradu...@gmail.com>
> > > wrote:
> > > > > 
> > > > > Hi Igniters!
> > > > > 
> > > > > Thank you for this important improvement!
> > > > > 
> > > > > I've looked through implementation and noticed that
> > > > > GridDhtPartitionsExchangeFuture#init has not been wrapped in blocked
> > > > > section. This means it easy to halt the node in case of longrunning
> > > > > actions during PME, for example when we create a cache with
> > > > > StoreFactrory which connect to 3rd party DB.
> > > > > 
> > > > > I'm not sure that it is the right behavior.
> > > > > 
> > > > > I filled the issue [1] and prepared the PR [2] with reproducer and
> > > 
> > > possible fix.
> > > > > 
> > > > > Andrey, could you please look at and confirm that it makes sense?
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9710
> > > > > [2] https://github.com/apache/ignite/pull/4845
> > > > > On Mon, Sep 24, 2018 at 9:46 PM Andrey Kuznetsov 
> > > 
> > > wrote:
> > > > > > 
> > > > > > Denis,
> > > > > > 
> > > > > > I've created the ticket [1] with short description of the
> > > 
> > > functionality.
> > > > > > 
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9679
> > > > > > 
> > > > > > 
> > > > > > пн, 24 сент. 2018 г. в 17:46, Denis Magda :
> > > > > > 
> > > > > > > Andrey K. and G.,
> > > > > > > 
> > > > > > > Thanks, do we have a documentation ticket created? Prachi
> > 
> > (copied)
> > > can help
> > > > > > > with the documentation.
> > > > > > > 
> > > > > > > --
> > > > > > > Denis
> > > > > > > 
> > > > > > > On Mon, Sep 24, 2018 at 5:51 AM Andrey Gura 
> > > 
> > > wrote:
> > > > > > > 
> > > > > > > > Andrey,
> > > > > > > > 
> > > > > > > > finally your change is merged to master branch. Congratulations
> > > 
> > > and
> > > > > > > > thank you very much! :)
> > > > > > > > 
> > > > > > > > I think that the next step is feature that will allow signal
> > > 
> > > about
> > > > > > > > blocked threads to the monitoring tools via MXBean.
> > > > > > > > 
> > > > > > > > I hope you will continue development of this feature and
> > 
> > provide
> > > your
> > > > > > > > vision in new JIRA 

Re: Apache Ignite 2.7 release

2018-09-27 Thread Nikolay Izhikov
Hello, Vova.

Thank you for clear release status.
I'm +1 for your proposal.

чт, 27 сент. 2018 г., 18:25 Alexey Kuznetsov :

> Vova,
>
> Huge +1 to do a stabilization.
>
>
> --
> Alexey Kuznetsov
>


Re: [TC Bot] Proposal of improvement

2018-09-25 Thread Nikolay Izhikov
Hello, Dmitriy

> What about the case when committer creates ignite-9679 branch and tests it> 
> without PR?

It means, committer is experienced enough to run tests via Team City interface 
:).

> So scanning seems to be possible only in JIRA

I don't understand you here.
You can retrieve comments filtered by *date*.
You don't have to scan all 1000 PR's one by one.
Anyway 1000 PR doesn't sound like big issue for me.

My vote goes strong to GiHub user interface.
I think we should have closer integration with GitHub, not Jira.

Jira is about tickets and project management.
GitHub is about code, commits and patches.
We test patch, not ticket.


В Вт, 25/09/2018 в 00:06 +0300, Dmitriy Pavlov пишет:
> Hi Nikolay,
> 
> What about the case when committer creates ignite-9679 branch and tests it
> without PR?
> 
> We have 1100+ open PRs and less than 100 open tickets. So scanning seems to
> be possible only in JIRA. Mention probably will work for GitHub, but it
> needs to be researched.
> 
> Two open PRs is not a valid situation in the majority of cases and How To
> Contribute asks to avoid it. The bot can ignore closed PRs and the bot can
> expect there is only one open PR per ticket.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> пн, 24 сент. 2018 г. в 23:41, Nikolay Izhikov :
> 
> > Hello, Dmitriy.
> > 
> > > But it could be a lot of work to handle mentions (probably from the
> > 
> > email> account and inbox).
> > 
> > Actually, it can be done via GitHub REST API [1].
> > It has 'since' param, so getting new GitHub comments is a very basic task.
> > 
> > > Patch available ticket
> > 
> > I think we shouldn't take a ticket as an entity that should be tested.
> > For me, it's a PR.
> > 
> > Moreover, it's a common case when we have several PR in a ticket.
> > And it's a common case when both of them has to be tested.
> > 
> > My vote goes to the closer integration with GitHub.
> > 
> > [1]
> > https://developer.github.com/v3/pulls/comments/#list-comments-in-a-repository
> > 
> > В Пн, 24/09/2018 в 22:36 +0300, Dmitriy Pavlov пишет:
> > > Hi Nikolay,
> > > 
> > > The idea makes perfect sense for me, and we should definitely take the
> > 
> > best
> > > practices from other big Apache projects.
> > > 
> > > But it could be a lot of work to handle mentions (probably from the email
> > > account and inbox).
> > > 
> > > I would like to suggest the following algorithm:
> > > 
> > > Patch available ticket, which was never checked by the bot will be
> > > processed in the following steps:
> > > 1. check existing run all (by PR or by branch name), if found go to the
> > > step 3
> > > 2. run-all to be triggered by PR
> > > 3. results should be analyzed for the presence of possible blockers. If
> > > there is no blockers go to step 5.
> > > 4. re-run of particular suites containing possible blockers should be
> > > applied to try to get success for very rare flaky failures (<1%). Go to 3
> > > (this go to should be done only once).
> > > 5. comment should be added to JIRA ticket containing information about
> > > results.
> > > 
> > > If a ticket was processed by bot early (probably author added some fixes)
> > > but still in PA state, the bot will check comments list and find possible
> > > new mentions (made after the previous build complete date). If it finds
> > > such comments it goes to step 1 (trying to find only new builds
> > 
> > available).
> > > 
> > > What do you think?
> > > 
> > > Sincerely,
> > > Dmitriy Pavlov
> > > 
> > > пн, 24 сент. 2018 г. в 21:43, Nikolay Izhikov :
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > I propose to implement following behaviour:
> > > > 
> > > > 1. Execute "Run all" suite for specific PR when the author of PR makes
> > 
> > a
> > > > comment
> > > > "@mtcga.bot Run Tests!" in GitHub comments.
> > > > 
> > > > 2. Send a comment with "Run All" results both: to a Jira ticket and
> > 
> > GitHub
> > > > comment.
> > > > 
> > > > 3. Label PR based on "Run All" results like it done in Apache Kafka [1]
> > > > 
> > > > I've create ticket for this proposal [2]
> > > > 
> > > > Thoughts?
> > > > 
> > > > [1] https://github.com/apache/kafka/pulls
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-9678

signature.asc
Description: This is a digitally signed message part


[jira] [Created] (IGNITE-9681) Wrong GIT config in Team City release archive

2018-09-25 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9681:
---

 Summary: Wrong GIT config in Team City release archive
 Key: IGNITE-9681
 URL: https://issues.apache.org/jira/browse/IGNITE-9681
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Nikolay Izhikov
 Fix For: 2.7


Release archive created by "[Prepare Vote #1] Java &.Net & C++ (Complete 
assembly)" [1] contains wrong .git/config. It includes local Team City path and 
doesn't work properly on release manager local environment.

Example of config file(lfs section is Team City specifi):

{noformat}
[core]
repositoryformatversion = 0
filemode = false
bare = false
logallrefupdates = true
sparseCheckout = true
[remote "origin"]
url = https://git-wip-us.apache.org/repos/asf/ignite
fetch = +refs/heads/*:refs/remotes/origin/*
[lfs]
storage = /data/teamcity/system/git/git-E4D58B67.git/lfs
[branch "master"]
remote = origin
merge = refs/heads/master

{noformat}


[1] 
https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: [TC Bot] Proposal of improvement

2018-09-24 Thread Nikolay Izhikov
Hello, Dmitriy.

> But it could be a lot of work to handle mentions (probably from the email> 
> account and inbox).

Actually, it can be done via GitHub REST API [1].
It has 'since' param, so getting new GitHub comments is a very basic task.

> Patch available ticket

I think we shouldn't take a ticket as an entity that should be tested.
For me, it's a PR.

Moreover, it's a common case when we have several PR in a ticket.
And it's a common case when both of them has to be tested.

My vote goes to the closer integration with GitHub.

[1] 
https://developer.github.com/v3/pulls/comments/#list-comments-in-a-repository

В Пн, 24/09/2018 в 22:36 +0300, Dmitriy Pavlov пишет:
> Hi Nikolay,
> 
> The idea makes perfect sense for me, and we should definitely take the best
> practices from other big Apache projects.
> 
> But it could be a lot of work to handle mentions (probably from the email
> account and inbox).
> 
> I would like to suggest the following algorithm:
> 
> Patch available ticket, which was never checked by the bot will be
> processed in the following steps:
> 1. check existing run all (by PR or by branch name), if found go to the
> step 3
> 2. run-all to be triggered by PR
> 3. results should be analyzed for the presence of possible blockers. If
> there is no blockers go to step 5.
> 4. re-run of particular suites containing possible blockers should be
> applied to try to get success for very rare flaky failures (<1%). Go to 3
> (this go to should be done only once).
> 5. comment should be added to JIRA ticket containing information about
> results.
> 
> If a ticket was processed by bot early (probably author added some fixes)
> but still in PA state, the bot will check comments list and find possible
> new mentions (made after the previous build complete date). If it finds
> such comments it goes to step 1 (trying to find only new builds available).
> 
> What do you think?
> 
> Sincerely,
> Dmitriy Pavlov
> 
> пн, 24 сент. 2018 г. в 21:43, Nikolay Izhikov :
> 
> > Hello, Igniters.
> > 
> > I propose to implement following behaviour:
> > 
> > 1. Execute "Run all" suite for specific PR when the author of PR makes a
> > comment
> > "@mtcga.bot Run Tests!" in GitHub comments.
> > 
> > 2. Send a comment with "Run All" results both: to a Jira ticket and GitHub
> > comment.
> > 
> > 3. Label PR based on "Run All" results like it done in Apache Kafka [1]
> > 
> > I've create ticket for this proposal [2]
> > 
> > Thoughts?
> > 
> > [1] https://github.com/apache/kafka/pulls
> > [2] https://issues.apache.org/jira/browse/IGNITE-9678

signature.asc
Description: This is a digitally signed message part


[TC Bot] Proposal of improvement

2018-09-24 Thread Nikolay Izhikov
Hello, Igniters.

I propose to implement following behaviour:

1. Execute "Run all" suite for specific PR when the author of PR makes a comment
"@mtcga.bot Run Tests!" in GitHub comments.

2. Send a comment with "Run All" results both: to a Jira ticket and GitHub 
comment.

3. Label PR based on "Run All" results like it done in Apache Kafka [1]

I've create ticket for this proposal [2]

Thoughts?

[1] https://github.com/apache/kafka/pulls
[2] https://issues.apache.org/jira/browse/IGNITE-9678

signature.asc
Description: This is a digitally signed message part


[jira] [Created] (IGNITE-9678) [TC Bot] Trigger Run All for github comment

2018-09-24 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9678:
---

 Summary: [TC Bot] Trigger Run All for github comment
 Key: IGNITE-9678
 URL: https://issues.apache.org/jira/browse/IGNITE-9678
 Project: Ignite
  Issue Type: Bug
Reporter: Nikolay Izhikov


Team City bot should work in the following way:
It should be able to:

1. Execute "Run all" suite for specific PR when the author of PR makes a 
comment "@mtcga.bot Run Tests!" in GitHub comments.
2. Send a comment with "Run All" results both: to a Jira ticket and GitHub 
comment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Sign Keys for release artifacts

2018-09-24 Thread Nikolay Izhikov
Hello, Igniters.

Cos, I have a question regarding Apache Ignite release.

I have committer priveledges in Apache Ignite.
I don't have PMC role. 

Since I don't have PMC role, I can't add my GPG key to KEYS [1] file.

Is it OK if someone with PMC role adds my GPG key to KEYS file?

[1] https://dist.apache.org/repos/dist/release/ignite/KEYS

signature.asc
Description: This is a digitally signed message part


Ignite-2.7 branch created

2018-09-24 Thread Nikolay Izhikov
Hello, Igniters.

I've created branch for an Apache Ignite 2.7 release [1]
Please, cherry-pick tickets that should be in 2.7 to that branch.

[1] 
https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=shortlog;h=refs/heads/ignite-2.7

signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 - Release Procedure issues

2018-09-24 Thread Nikolay Izhikov
Hello, Petr.

My suggestion is to migrate to a newer version of GPG and throw an error 
message if one use old version.

В Пн, 24/09/2018 в 14:53 +0300, Petr Ivanov пишет:
> I’ve checked the changes and they are good both on old and latest versions of 
> Ubuntu.
> 
> 
> However, I’ve stumbled upon another problem — GPG: current release scripts do 
> not honour latest GPG versions.
> I can introduce corresponding changes, but question is — should release 
> script check for GPG version and have 2 version of signing commands or just 
> warn user about old version of GPG and exit?
> 
> 
> > On 21 Sep 2018, at 19:46, Nikolay Izhikov  wrote:
> > 
> > Hello, Petr.
> > 
> > Seems that rpm build script doesn't work on a lates Ubuntu Linux.
> > I've created a ticket [1] and found a fix for it [2]
> > 
> > With one line fix rpm build is working under my environment.
> > Can you check fix on your environment?
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-9665
> > 
> > [2] https://github.com/apache/ignite/pull/4808
> > 
> > В Пт, 21/09/2018 в 16:22 +0300, Petr Ivanov пишет:
> > > Hi, Nikolay
> > > 
> > > 
> > > I’ve tested vote_3_step_1 and vote_3_step_2 scripts from [1] and they are 
> > > OK.
> > > My configuration:
> > > - generated gnupg key (~/.gnupg)
> > > - Ubuntu 16.04 (with latest updates)
> > > - packages: subversion git unzip alien rpm fakeroot gcc dpkg-sig 
> > > gnupg-agent
> > > 
> > > Please double check you environment for release procedure
> > > 
> > > 
> > > [1] 
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1914618=ApacheIgniteReleaseJava8_PrepareVote=artifacts#!hkm8d5gqy4ii
> > > 
> > > 
> > > > On 20 Sep 2018, at 17:39, Nikolay Izhikov  wrote:
> > > > 
> > > > Hello, Igniters.
> > > > 
> > > > I've started to write Wiki article for a future Release Managers.
> > > > Since release process doesn't described anywhere public I do it while 
> > > > releasing 2.7:
> > > > 
> > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+manager+Notes
> > > > 
> > > > Any feedback is strongly appreciated.
> > > > 
> > > > I've tried to walk through vote steps in release procedure and found 
> > > > some issues:
> > > > 
> > > > Some of them we already fixed with Petr Ivanod and Anton Vinogradov.
> > > > Thank you, guys!
> > > > 
> > > > For now, I stuck on the following issue:
> > > > 
> > > > `vote_3_step_1\[packages\]build.sh` is broken.
> > > > I got following output:
> > > > 
> > > > ```
> > > > RPM build errors:
> > > >   bogus date in %changelog: Fri Jun 17 2018 Peter Ivanov 
> > > >  - 2.6.0-1
> > > >   File not found: 
> > > > /tmp/tmp.EezfJVTwLm/BUILDROOT/apache-ignite-2.6.0-1.x86_64/usr/share/doc/apache-ignite-2.6.0/MIGRATION_GUIDE.txt
> > > > + processTrap
> > > > + echo 'Removing temporary work directories:  /tmp/tmp.EezfJVTwLm'
> > > > Removing temporary work directories:  /tmp/tmp.EezfJVTwLm
> > > > + rm -rf /tmp/tmp.EezfJVTwLm
> > > > ```
> > > > 
> > > > 1. Script uses version number 2.6 somehow.
> > > > 2. It fails because MIGRATION_GUIDE file doesn't exists.
> > > > 
> > > > Is there anybody who can help with this issue?
> > > > 
> > > > 
> > > > 
> 
> 

signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 - Release Procedure issues

2018-09-21 Thread Nikolay Izhikov
Hello, Petr.

Seems that rpm build script doesn't work on a lates Ubuntu Linux.
I've created a ticket [1] and found a fix for it [2]

With one line fix rpm build is working under my environment.
Can you check fix on your environment?

[1] https://issues.apache.org/jira/browse/IGNITE-9665

[2] https://github.com/apache/ignite/pull/4808

В Пт, 21/09/2018 в 16:22 +0300, Petr Ivanov пишет:
> Hi, Nikolay
> 
> 
> I’ve tested vote_3_step_1 and vote_3_step_2 scripts from [1] and they are OK.
> My configuration:
>  - generated gnupg key (~/.gnupg)
>  - Ubuntu 16.04 (with latest updates)
>  - packages: subversion git unzip alien rpm fakeroot gcc dpkg-sig gnupg-agent
> 
> Please double check you environment for release procedure
> 
> 
> [1] 
> https://ci.ignite.apache.org/viewLog.html?buildId=1914618=ApacheIgniteReleaseJava8_PrepareVote=artifacts#!hkm8d5gqy4ii
> 
> 
> > On 20 Sep 2018, at 17:39, Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > I've started to write Wiki article for a future Release Managers.
> > Since release process doesn't described anywhere public I do it while 
> > releasing 2.7:
> > 
> > https://cwiki.apache.org/confluence/display/IGNITE/Release+manager+Notes
> > 
> > Any feedback is strongly appreciated.
> > 
> > I've tried to walk through vote steps in release procedure and found some 
> > issues:
> > 
> > Some of them we already fixed with Petr Ivanod and Anton Vinogradov.
> > Thank you, guys!
> > 
> > For now, I stuck on the following issue:
> > 
> > `vote_3_step_1\[packages\]build.sh` is broken.
> > I got following output:
> > 
> > ```
> > RPM build errors:
> >bogus date in %changelog: Fri Jun 17 2018 Peter Ivanov 
> >  - 2.6.0-1
> >File not found: 
> > /tmp/tmp.EezfJVTwLm/BUILDROOT/apache-ignite-2.6.0-1.x86_64/usr/share/doc/apache-ignite-2.6.0/MIGRATION_GUIDE.txt
> > + processTrap
> > + echo 'Removing temporary work directories:  /tmp/tmp.EezfJVTwLm'
> > Removing temporary work directories:  /tmp/tmp.EezfJVTwLm
> > + rm -rf /tmp/tmp.EezfJVTwLm
> > ```
> > 
> > 1. Script uses version number 2.6 somehow.
> > 2. It fails because MIGRATION_GUIDE file doesn't exists.
> > 
> > Is there anybody who can help with this issue?
> > 
> > 
> > 
> 
> 

signature.asc
Description: This is a digitally signed message part


[jira] [Created] (IGNITE-9665) RPM build doesn't work on lates Ubuntu linux

2018-09-21 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9665:
---

 Summary: RPM build doesn't work on lates Ubuntu linux
 Key: IGNITE-9665
 URL: https://issues.apache.org/jira/browse/IGNITE-9665
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
 Fix For: 2.7


Release script fails to build RPM package on lates Ubuntu Linux
Error message is:

{{noformat}}
Checking for unpackaged file(s): /usr/lib/rpm/check-files 
/tmp/tmp.ROYlZNaiI4/BUILDROOT/apache-ignite-2.7.0-1.x86_64
error: Installed (but unpackaged) file(s) found:
   /usr/share/doc/apache-ignite-2.7.0/bin/ignitevisorcmd.sh
   /usr/share/doc/apache-ignite-2.7.0/examples/README.txt
   /usr/share/doc/apache-ignite-2.7.0/examples/config/example-cache.xml
   /usr/share/doc/apache-ignite-2.7.0/examples/config/example-data-regions.xml
{{noformat}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Apache Ignite 2.7 release

2018-09-21 Thread Nikolay Izhikov
Hello, Igniters.

I ping assigners in all tickets hase been planned for 2.7 and work is started.
I plan to move all tickets in 'Open' state to 2.8 on Moday, 24 September.

Please, respond, if you have any objections.


В Ср, 19/09/2018 в 16:02 +0300, Andrey Gura пишет:
> Nikolay,
> 
> since we talk about scope freeze all you need now just create
> ignite-2.7 branch. We still can have tickets targeted to 2.7 release
> in progress. So you shouldn't move tickets to 2.8 because they can be
> targeted to 2.7 intentionally and will be merged to master and
> ignite-2.7 branches.
> On Wed, Sep 19, 2018 at 12:18 PM Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > Vova, thank you for pointing this out.
> > 
> > I ask all community members to iterate over assigned tickets.
> > Please, move your tickets to 2.8 release if work is still in progress.
> > At the end of the day, I will do it by myself and tomorrow we should have
> > release scope prepared.
> > 
> > 
> > ср, 19 сент. 2018 г. в 12:08, Vladimir Ozerov :
> > 
> > > My point was not about code freeze, but about scope freeze, what means 
> > > that
> > > starting from this point AI 2.7 release should not receive any commits
> > > which doesn't relate to it.
> > > 
> > > On Wed, Sep 19, 2018 at 11:55 AM Anton Vinogradov  wrote:
> > > 
> > > > Vova,
> > > > 
> > > > AFAIK, codefreeze was not announced yet.
> > > > Correct me in case I missed this.
> > > > 
> > > > Now, Nikolay finishing checks that he's able to perform release
> > > 
> > > (everything
> > > > installed and properly configured).
> > > > So, I recommend him to mention this check here to solve any
> > > > misunderstanding in case some branches or tags related to 2.7 will be
> > > 
> > > found
> > > > before we announced codefreeze.
> > > > 
> > > > ср, 19 сент. 2018 г. в 11:45, Vladimir Ozerov :
> > > > 
> > > > > Anton,
> > > > > 
> > > > > What do you mean under "preparation to real 2.7". It is already real,
> > > 
> > > we
> > > > > reached formal code freeze phase we agreed on. At this point we need 
> > > > > to
> > > > > exclude non-2.7 commits, otherwise we will not be able to come to
> > > 
> > > stable
> > > > > branch in two weeks.
> > > > > For this reason creating a branch for 2.7 at the moment is perfectly
> > > > 
> > > > valid
> > > > > thing. This is how we made releases previously.
> > > > > 
> > > > > On Tue, Sep 18, 2018 at 7:59 PM Dmitriy Pavlov 
> > > > > wrote:
> > > > > 
> > > > > > Hi Paul.
> > > > > > 
> > > > > > There are 2 PRs linked to that ticket. Who is reviewing your 
> > > > > > changes?
> > > > > > 
> > > > > > Branch for 2.7 is still master, so if your changes are reviewed and
> > > > > > accepted soon it will be in 2.7.
> > > > > > 
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > > 
> > > > > > вт, 18 сент. 2018 г. в 16:22, Paul Anderson :
> > > > > > 
> > > > > > > Hi, may I ask for IGNITE-9298 to be included in 2.7 pls
> > > > > > > 
> > > > > > > On Tue, Sep 18, 2018 at 1:03 PM Nikolay Izhikov <
> > > 
> > > nizhi...@apache.org
> > > > > 
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hello, folks.
> > > > > > > > 
> > > > > > > > Thanks for the comments.
> > > > > > > > 
> > > > > > > > I will follow them.
> > > > > > > > 
> > > > > > > > В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> > > > > > > > > Nikolay,
> > > > > > > > > 
> > > > > > > > > 1) *Do not* create ignite-2.7 branch until we're not started
> > > > > > > 
> > > > > > > preparation
> > > > > > > > to
> > > > > > > > > real 2.7.
> > > > > > > > > Use some temporary branch for this check instead, eg.
> > > > &

Apache Ignite 2.7 - Release Procedure issues

2018-09-20 Thread Nikolay Izhikov
Hello, Igniters.

I've started to write Wiki article for a future Release Managers.
Since release process doesn't described anywhere public I do it while releasing 
2.7:

https://cwiki.apache.org/confluence/display/IGNITE/Release+manager+Notes

Any feedback is strongly appreciated.

I've tried to walk through vote steps in release procedure and found some 
issues:

Some of them we already fixed with Petr Ivanod and Anton Vinogradov.
Thank you, guys!

For now, I stuck on the following issue:

`vote_3_step_1\[packages\]build.sh` is broken.
I got following output:

```
RPM build errors:
bogus date in %changelog: Fri Jun 17 2018 Peter Ivanov 
 - 2.6.0-1
File not found: 
/tmp/tmp.EezfJVTwLm/BUILDROOT/apache-ignite-2.6.0-1.x86_64/usr/share/doc/apache-ignite-2.6.0/MIGRATION_GUIDE.txt
+ processTrap
+ echo 'Removing temporary work directories:  /tmp/tmp.EezfJVTwLm'
Removing temporary work directories:  /tmp/tmp.EezfJVTwLm
+ rm -rf /tmp/tmp.EezfJVTwLm
```

1. Script uses version number 2.6 somehow.
2. It fails because MIGRATION_GUIDE file doesn't exists.

Is there anybody who can help with this issue?





signature.asc
Description: This is a digitally signed message part


Apache Ignite 2.7 - Service Grid Redesign

2018-09-19 Thread Nikolay Izhikov
Hello, Igniters.

Currently, we are working on release scope for Apache Ignite 2.7
We had plans to include "Service Grid Redesign. Phase 1" to 2.7 release.

If I understand correctly, the plan is following - We should have PR that
is ready for review at the end of September.

This deadline is very close to the code freeze date.

Let's discuss, how we should handle this risks?
Who will be able to review this PR when it's will be ready?
Vyacheslav, can you comment on this?


Re: Apache Ignite 2.7 release

2018-09-19 Thread Nikolay Izhikov
Hello, Igniters.

Vova, thank you for pointing this out.

I ask all community members to iterate over assigned tickets.
Please, move your tickets to 2.8 release if work is still in progress.
At the end of the day, I will do it by myself and tomorrow we should have
release scope prepared.


ср, 19 сент. 2018 г. в 12:08, Vladimir Ozerov :

> My point was not about code freeze, but about scope freeze, what means that
> starting from this point AI 2.7 release should not receive any commits
> which doesn't relate to it.
>
> On Wed, Sep 19, 2018 at 11:55 AM Anton Vinogradov  wrote:
>
> > Vova,
> >
> > AFAIK, codefreeze was not announced yet.
> > Correct me in case I missed this.
> >
> > Now, Nikolay finishing checks that he's able to perform release
> (everything
> > installed and properly configured).
> > So, I recommend him to mention this check here to solve any
> > misunderstanding in case some branches or tags related to 2.7 will be
> found
> > before we announced codefreeze.
> >
> > ср, 19 сент. 2018 г. в 11:45, Vladimir Ozerov :
> >
> > > Anton,
> > >
> > > What do you mean under "preparation to real 2.7". It is already real,
> we
> > > reached formal code freeze phase we agreed on. At this point we need to
> > > exclude non-2.7 commits, otherwise we will not be able to come to
> stable
> > > branch in two weeks.
> > > For this reason creating a branch for 2.7 at the moment is perfectly
> > valid
> > > thing. This is how we made releases previously.
> > >
> > > On Tue, Sep 18, 2018 at 7:59 PM Dmitriy Pavlov 
> > > wrote:
> > >
> > > > Hi Paul.
> > > >
> > > > There are 2 PRs linked to that ticket. Who is reviewing your changes?
> > > >
> > > > Branch for 2.7 is still master, so if your changes are reviewed and
> > > > accepted soon it will be in 2.7.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 18 сент. 2018 г. в 16:22, Paul Anderson :
> > > >
> > > > > Hi, may I ask for IGNITE-9298 to be included in 2.7 pls
> > > > >
> > > > > On Tue, Sep 18, 2018 at 1:03 PM Nikolay Izhikov <
> nizhi...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello, folks.
> > > > > >
> > > > > > Thanks for the comments.
> > > > > >
> > > > > > I will follow them.
> > > > > >
> > > > > > В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> > > > > > > Nikolay,
> > > > > > >
> > > > > > > 1) *Do not* create ignite-2.7 branch until we're not started
> > > > > preparation
> > > > > > to
> > > > > > > real 2.7.
> > > > > > > Use some temporary branch for this check instead, eg.
> > > > > > > ignite-2.7-release-test
> > > > > > >
> > > > > > > 2) Please make sure you'll not cause real release actions
> (maven
> > > > > release
> > > > > > > and so on).
> > > > > > > Perform only vote_* steps.
> > > > > > >
> > > > > > > 3) Make sure you'll remove all tags, branches, and other RC
> > > artifacts
> > > > > > after
> > > > > > > check.
> > > > > > >
> > > > > > > 4) Mark this release as RC0 to make sure it will be clear to
> > > > everybody
> > > > > > that
> > > > > > > it's a check.
> > > > > > >
> > > > > > >
> > > > > > > вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >:
> > > > > > >
> > > > > > > > If it is an Ignite release, then it has to pass through the
> > vote.
> > > > If
> > > > > > not,
> > > > > > > > then you can do the test without publishing or uploading the
> > > > release.
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov <
> > mr.wei...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >

Re: Apache Ignite 2.7 release

2018-09-18 Thread Nikolay Izhikov
Hello, folks.

Thanks for the comments.

I will follow them.

В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> Nikolay,
> 
> 1) *Do not* create ignite-2.7 branch until we're not started preparation to
> real 2.7.
> Use some temporary branch for this check instead, eg.
> ignite-2.7-release-test
> 
> 2) Please make sure you'll not cause real release actions (maven release
> and so on).
> Perform only vote_* steps.
> 
> 3) Make sure you'll remove all tags, branches, and other RC artifacts after
> check.
> 
> 4) Mark this release as RC0 to make sure it will be clear to everybody that
> it's a check.
> 
> 
> вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan :
> 
> > If it is an Ignite release, then it has to pass through the vote. If not,
> > then you can do the test without publishing or uploading the release.
> > 
> > D.
> > 
> > On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov  wrote:
> > 
> > > Ok.
> > > 
> > > In case of TC questions — ask me.
> > > 
> > > 
> > > 
> > > > On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> > > > 
> > > > Hello, Petr.
> > > > 
> > > > I want to make ignite-2.7 branch today.
> > > > And execute release procedure based on this branch.
> > > > 
> > > > However, ignite-2.7 branch will be copy of master until code freeze
> > 
> > date.
> > > > 
> > > > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> > > > > Will it be just a test or there is already ignite-2.7 branch?
> > > > > 
> > > > > Fabric removal related TC modifications are not ready yet, and code is
> > > 
> > > not in master.
> > > > > 
> > > > > 
> > > > > 
> > > > > > On 18 Sep 2018, at 13:07, Nikolay Izhikov 
> > 
> > wrote:
> > > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > I want to start and release procedures and make an RC1 build.
> > > > > > 
> > > > > > It has a 2 intention:
> > > > > > 
> > > > > > 1. I want to walk through all release steps to make sure they all
> > > 
> > > works for me.
> > > > > > So I will be fully ready on release date.
> > > > > > 
> > > > > > 2. We have updated some dependencies in 2.7 and we need to make sure
> > > 
> > > binary build is still workable.
> > > > > > 
> > > > > > Any objections?
> > > > > > 
> > > > > > 
> > > > > > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > > > > > We already have all the mechanics in place to work with 
> > > > > > > properties -
> > > 
> > > we use
> > > > > > > ignite.build and ignite.revision from ignite.properties which are
> > > 
> > > adjusted
> > > > > > > during the build in the binary package.
> > > > > > > 
> > > > > > > Should I create the ticket if there are no objections?
> > > > > > > 
> > > > > > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> > > 
> > > ilya.kasnach...@gmail.com>:
> > > > > > > 
> > > > > > > > Hello!
> > > > > > > > 
> > > > > > > > So now there's an issue that this script makes source change 
> > > > > > > > after
> > > 
> > > every
> > > > > > > > build, show up in git status.
> > > > > > > > 
> > > > > > > > What we could do to it:
> > > > > > > > - Commit the changes after the build, once. In hopes that it 
> > > > > > > > won't
> > > 
> > > change
> > > > > > > > very often. With benefit that we could do that right now, before
> > 
> > the
> > > code
> > > > > > > > freeze.
> > > > > > > > - Move these values to a properties file from both pom.xml and
> > > > > > > > IgniteProvider.java. Any problems with this approach? We'll just
> > > 
> > > read them
> > > > > > > > from classpath properties file.
> > > > > > > > - Update the links in the file once and r

Re: Apache Ignite 2.7 release

2018-09-18 Thread Nikolay Izhikov
Hello, Petr.

I want to make ignite-2.7 branch today.
And execute release procedure based on this branch.

However, ignite-2.7 branch will be copy of master until code freeze date.

В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> Will it be just a test or there is already ignite-2.7 branch?
> 
> Fabric removal related TC modifications are not ready yet, and code is not in 
> master.
> 
> 
> 
> > On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > I want to start and release procedures and make an RC1 build.
> > 
> > It has a 2 intention:
> > 
> > 1. I want to walk through all release steps to make sure they all works for 
> > me.
> > So I will be fully ready on release date.
> > 
> > 2. We have updated some dependencies in 2.7 and we need to make sure binary 
> > build is still workable.
> > 
> > Any objections?
> > 
> > 
> > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > We already have all the mechanics in place to work with properties - we 
> > > use
> > > ignite.build and ignite.revision from ignite.properties which are adjusted
> > > during the build in the binary package.
> > > 
> > > Should I create the ticket if there are no objections?
> > > 
> > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev :
> > > 
> > > > Hello!
> > > > 
> > > > So now there's an issue that this script makes source change after every
> > > > build, show up in git status.
> > > > 
> > > > What we could do to it:
> > > > - Commit the changes after the build, once. In hopes that it won't 
> > > > change
> > > > very often. With benefit that we could do that right now, before the 
> > > > code
> > > > freeze.
> > > > - Move these values to a properties file from both pom.xml and
> > > > IgniteProvider.java. Any problems with this approach? We'll just read 
> > > > them
> > > > from classpath properties file.
> > > > - Update the links in the file once and remove them from build process. 
> > > > Why
> > > > were they added to build process in the first place - to make them
> > > > configurable during build?
> > > > 
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > > 
> > > > 
> > > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> > > > 
> > > > > Ilya,
> > > > > 
> > > > > The "latest" version is the default, and resolved by
> > > > > https://ignite.apache.org/latest which is used by our web site when a
> > > > > user download the latest Ignite version. And I think this is the
> > > > 
> > > > authority
> > > > > to judge of the latest official release (pom.xml you suggest can have
> > > > > SNAPSHOTs etc.).
> > > > > Also, as I explained during our review sessions, ignite-mesos-2.6.0 
> > > > > is a
> > > > > driver and doesn't mean you need to have Ignite 2.6.0. User can run 
> > > > > any
> > > > > version of Ignite he/she specifies. By default, it's "latest" but a 
> > > > > user
> > > > > can specify any version needed, even from a non-archive URL.
> > > > > 
> > > > > In short, what we have now
> > > > > 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by 
> > > > > default
> > > > > -> it will try to resolve the latest officially releases version of
> > > > 
> > > > Apache
> > > > > Ignite, find the closest mirror and download Ignite in a minute. If 
> > > > > the
> > > > > version resolution fails, we fall back to the slow apache archive (as 
> > > > > you
> > > > > suggest; in my opinion we better fail-fast instead of waiting for 
> > > > > hours
> > > > 
> > > > to
> > > > > download, so the user can choose another download option (3))
> > > > > 2. If the user specifies the version explicitly, it goes to the slow
> > > > > apache archive.
> > > > > 3. The user can put ignite zip file on his/her http server and provide
> > > > 
> > > > the
> > > > > URL as a parameter to the driver, if options 1 and 2 don't work.
> > > > > 

Re: Apache Ignite 2.7 release

2018-09-18 Thread Nikolay Izhikov
already do with URLs. So if you could not resolve "latest" you will
> > > download 2.7.0.
> > > 
> > > Buuut, maybe it's not necessary, maybe we should just *discourage
> > > "latest"*, which is in my opinion almost always a bad idea.
> > > 
> > > WDYT?
> > > 
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > > 
> > > 
> > > вс, 9 сент. 2018 г. в 5:47, Roman Shtykh :
> > > 
> > > Hi Ilya,
> > > 
> > > Sorry, missed that.
> > > Added now.
> > > 
> > > --
> > > Roman Shtykh
> > > 
> > > 
> > > On Thursday, September 6, 2018, 6:16:58 p.m. GMT+9, Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com> wrote:
> > > 
> > > 
> > > Hello!
> > > 
> > > The last of my requests still standing is that we should fall-back to
> > > single URL download in case of error with 'latest' version. Everything
> > 
> > else
> > > looks good to me.
> > > 
> > > Can we do that? I'm really worried that Apache API will go sour.
> > > 
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > > 
> > > 
> > > чт, 6 сент. 2018 г. в 8:56, Roman Shtykh :
> > > 
> > > Hi Ilya,
> > > 
> > > Thanks again.
> > > 
> > > 1) Done.
> > > 2) Used catch() for latest version.
> > > 
> > > Please see my comments on github.
> > > --
> > > Roman Shtykh
> > > 
> > > 
> > > On Wednesday, September 5, 2018, 11:30:10 p.m. GMT+9, Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com> wrote:
> > > 
> > > 
> > > Hello!
> > > 
> > > I've left a new wave of replies.
> > > 
> > > Basically, 1) let's keep DOWNLOAD_URL_PATTERN string value inlined so
> > > that it will work even if build process is broken (would be useful for
> > 
> > e.g.
> > > developing out of IDE)
> > > And also I urge you to catch() around new fragile Apache JSON API
> > > resolving, and download the 'current' version instead, as defined by
> > > ignite-mesos version.
> > > 
> > > This is because this module is not under continuouos scrutiny so extra
> > > care should be applied.
> > > --
> > > Ilya Kasnacheev
> > > 
> > > 
> > > вт, 4 сент. 2018 г. в 13:42, Roman Shtykh :
> > > 
> > > Thanks, Ilya!
> > > I will check your comments, and discuss it at JIRA.
> > > 
> > > --
> > > Roman Shtykh
> > > 
> > > 
> > > On Tuesday, September 4, 2018, 7:17:53 p.m. GMT+9, Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com> wrote:
> > > 
> > > 
> > > Hello!
> > > 
> > > IGNITE-9408 <https://issues.apache.org/jira/browse/IGNITE-9408> looks
> > > good to me and may be merged right away.
> > > 
> > > IGNITE-9388 <https://issues.apache.org/jira/browse/IGNITE-9388> needs
> > > more work in my opinion, I have commented the PR. I also advice having
> > 
> > test
> > > for this functionality.
> > > 
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > > 
> > > 
> > > вт, 4 сент. 2018 г. в 6:52, Roman Shtykh :
> > > 
> > > Igniters,
> > > I would like Mesos integration update be included in the upcoming
> > > release.Can anyone review prs for the following issues?
> > > IGNITE-9388: mesos IgniteProvider tries to access obsolete ignite.run or
> > > download from slow archiveIGNITE-9408: Update mesos version
> > > 
> > > Roman Shtykh
> > > 
> > > On Thursday, August 30, 2018, 9:25:43 p.m. GMT+9, Vyacheslav Daradur
> > 
> > <
> > > daradu...@gmail.com> wrote:
> > > 
> > >  Hi Igniters!
> > > 
> > > I'm working on the following Service Grid tasks:
> > > - IGNITE-8361 Use discovery messages for service deployment
> > > - IGNITE-8362 Collect service deployment results asynchronously on
> > > coordinator
> > > - IGNITE-8363 Handle topology changes during service deployment
> > > - IGNITE-8364 Propagate deployed services to joining nodes
> > > - IGNITE-8365 Introduce service failure events
> > > - IGNITE-3392 Propagate service deployment results from assigned nodes
> > > to initiator
> > > 
> > > Let's call them *phase 1* because the should be implemented together
> > > (atomically).
> > > 
> > > I do my best to finish phase 1 for including to 2.7 release.
> > > 
> > > But I'm not sure that the solution will be fully completed till the
> > > beginning of October.
> > > 
> > > On Wed, Aug 29, 2018 at 7:18 PM Nikolay Izhikov 
> > > wrote:
> > > > 
> > > > Hell, Yakov
> > > > 
> > > > I'm ok with your proposal.
> > > > 
> > > >* Scope freeze - September 17 - We should have a full list of
> > > 
> > > tickets for 2.7 here.
> > > >* Code freeze - October 01 - We should merge all 2.7 tickets to
> > > 
> > > master here.
> > > >* Vote on RC1 - October 11.
> > > >* Vote on release - October 15.
> > > > 
> > > > В Ср, 29/08/2018 в 12:39 +0300, Yakov Zhdanov пишет:
> > > > > Nikolay,
> > > > > 
> > > > > I think we should have 2 weeks after code freeze which by the way may
> > > > > include RC1 voting stage. This way I would like us to agree that
> > > 
> > > release
> > > > > candidate should be sent to vote on Oct, 11th and we can release on
> > > 
> > > Oct,
> > > > > 15th.
> > > > > 
> > > > > What do you think?
> > > > > 
> > > > > --Yakov
> > > 
> > > 
> > > 
> > > --
> > > Best Regards, Vyacheslav D.
> > > 
> > > 

signature.asc
Description: This is a digitally signed message part


Re: TeamCity Helper shows new failures in PRs

2018-09-14 Thread Nikolay Izhikov
Hello, Dmitriy.

Webinar is a great idea, let's have it.

В Пт, 14/09/2018 в 09:39 +0300, Dmitriy Pavlov пишет:
> Hi Folks,
> 
> There is no description of this feature yet. But I think we should come
> back to the idea of running a webinar.
> 
> Please share your vision. If Igniters will attend I send an invitation to
> the beginning of next week.
> 
> I hope in the nearest time we will have a few more features to speak about.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> чт, 13 сент. 2018 г. в 22:58, Nikolay Izhikov :
> 
> > Hello, Dmitrii.
> > 
> > Can you give a link to detailed description of this feature?
> > 
> > Actually, I wonder if someone except bit developers know how to use it
> > 
> > чт, 13 сент. 2018 г., 22:46 Dmitrii Ryabov :
> > 
> > > Hello, Igniters!
> > > 
> > > We've improved tests analysis in the TeamCity Helper.
> > > 
> > > Now, when you view PR analysis, you can see a table with possible
> > 
> > blockers
> > > - failed tests and suites, that most likely appeared in the pull request.
> > > 

signature.asc
Description: This is a digitally signed message part


Re: Ignite Spark Bugs

2018-09-13 Thread Nikolay Izhikov
Hello, Denis.

I've created a tests for Ignite codebase and attached it to the ticket [1].

[1] https://issues.apache.org/jira/browse/IGNITE-9593

В Вт, 11/09/2018 в 13:34 +0300, Nikolay Izhikov пишет:
> Hello, Denis.
> 
> Thanks for providing reproducers for bugs!
> 
> I taked a look at your project and be able to reproduce some of errors.
> I will create a ticket and start investigation in a next few days.
> 
> В Вт, 11/09/2018 в 12:28 +0300, Dmitriy Pavlov пишет:
> > Hi Denis,
> > 
> > Thank you for bringing this here and for your efforts to reproduce it.
> > Would you like to create an issue and contribute these test into Ignite
> > code base?
> > 
> > If you would like to proceed with the patch submission process, please
> > sign-in to Apache JIRA and share your JIRA username, I will add you as the
> > contributor.
> > 
> > You can count on my support.
> > 
> > Sincerely,
> > Dmitriy Pavlov
> > 
> > вт, 11 сент. 2018 г. в 10:51, Денис Костин :
> > 
> > > Hello everybody!
> > > 
> > > I found a few bugs in the actual version of Apache Ignite Spark (2.6.0)
> > > and described in my GitHub: https://github.com/x-x-z/IgniteSparkBugs
> > > 
> > > I wrote a test for each case:
> > > https://github.com/x-x-z/IgniteSparkBugs/blob/master/src/test/java/org/xxz/ignite_spark_bugs/IgniteSparkBugsApplicationTests.java
> > > 
> > > Check, please.
> > > 
> > > 
> > > Best regards,
> > > Denis Kostin

signature.asc
Description: This is a digitally signed message part


[jira] [Created] (IGNITE-9593) Spart Optimization fails to optimize statements

2018-09-13 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9593:
---

 Summary: Spart Optimization fails to optimize statements
 Key: IGNITE-9593
 URL: https://issues.apache.org/jira/browse/IGNITE-9593
 Project: Ignite
  Issue Type: Bug
  Components: spark
Affects Versions: 2.6
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
 Fix For: 2.7
 Attachments: Spark_optimization_bugs_reproducer.patch

In some cases, {{IgniteOptimization}} fails to optimize spark query. Reproducer 
attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: TeamCity Helper shows new failures in PRs

2018-09-13 Thread Nikolay Izhikov
Hello, Dmitrii.

Can you give a link to detailed description of this feature?

Actually, I wonder if someone except bit developers know how to use it

чт, 13 сент. 2018 г., 22:46 Dmitrii Ryabov :

> Hello, Igniters!
>
> We've improved tests analysis in the TeamCity Helper.
>
> Now, when you view PR analysis, you can see a table with possible blockers
> - failed tests and suites, that most likely appeared in the pull request.
>


Re: Table Names in Spark Catalog

2018-09-13 Thread Nikolay Izhikov
Hello, Stuart.

Do you need any assistance with this task from me or other community member?

В Вт, 04/09/2018 в 19:03 +0300, Nikolay Izhikov пишет:
> Hello, Stuart.
> 
> Sorry for the silence.
> 
> I was swamped the last couple of days.
> 
> I think you can go forward and implement suggested solution.
> I'm -0 with it.
> So no block from my side, but I'm still no happy with abstractions :).
> 
> В Пн, 03/09/2018 в 09:35 +0100, Stuart Macdonald пишет:
> > Nikolay, Val, it would be good if we could reach agreement here so that I
> > can make the necessary modifications before the 2.7 cutoff.
> > 
> > Nikolay - would you be comfortable if I went ahead and made database=schema?
> > 
> > Stuart.
> > 
> > On Mon, Aug 27, 2018 at 10:22 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> > 
> > > Hi Nikolay,
> > > 
> > > I think it's actually pretty unfortunate that Spark uses term "database"
> > > here, as it essentially refers to a schema in my view. Usually, database 
> > > is
> > > something you create a physical connection to, and connection is bind to
> > > that database. To connect to another database you need to create a new
> > > connection. In Spark, however, you can switch between "databases" within a
> > > single session, which looks really weird to me because it's usually a
> > > characteristic of a schema. Having said that, I understand your concern,
> > > but I don't think there is an ideal solution.
> > > 
> > > As for your approach, I still don't understand how it will allow to fully
> > > support schemas in catalog.
> > > - How will you get a list of tables within a particular schema? In other
> > > words, what would listTables() method return?
> > > - How will you switch between the schemas?
> > > - Etc.
> > > 
> > > I still think assuming database=schema is the best we can do here, but I
> > > would be happy to hear another opinions from other community members.
> > > 
> > > OPTION_SCHEMA should definitely be introduced though (I thought we already
> > > did, no?). CREATE TABLE will be supported with this ticket:
> > > https://issues.apache.org/jira/browse/IGNITE-5780. For now we will have to
> > > throw an exception if custom schema name is provided when creating a Spark
> > > session, but table does not exist yet.
> > > 
> > > -Val
> > > 
> > > On Sun, Aug 26, 2018 at 7:56 AM Nikolay Izhikov 
> > > wrote:
> > > 
> > > > Igniters,
> > > > 
> > > > Personally, I don't like the solution with database == schema name.
> > > > 
> > > > 1. I think we should try to use the right abstractions.
> > > > schema == database doesn't sound right for me.
> > > > 
> > > > Do you want to answer to all of our users something like that:
> > > > 
> > > > - "How I can change Ignite SQL schema?"
> > > > - "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".
> > > > 
> > > > 2. I think we restrict whole solution with that decision.
> > > > If Ignite will support multiple databases in the future we just don't
> > > 
> > > have
> > > > a place for it.
> > > > 
> > > > I think we should do the following:
> > > > 
> > > > 1. IgniteExternalCatalog should be able to return *ALL* tables
> > > > within Ignite instance.
> > > > We shouldn't restrict tables list by schema by default.
> > > > We should return tables with schema name - `schema.table`
> > > > 
> > > > 2. We should introduce `OPTION_SCHEMA` for a dataframe to 
> > > > specify
> > > > a schema.
> > > > 
> > > > There is an issue with the second step: We can't use schema name
> > > > in `CREATE TABLE` clause.
> > > > This is restriction of current Ignite SQL.
> > > > 
> > > > I propose to make the following:
> > > > 
> > > > 1. For all write modes that requires the creation of table we
> > > > should disallow usage of table outside of `SQL_PUBLIC`
> > > > or usage of `OPTION_SCHEMA`. We should throw proper exception 
> > > > for
> > > > this case.
> > > > 
> > > > 2. Create a ticket to support `CRE

Re: TDE Implementation details.

2018-09-12 Thread Nikolay Izhikov
Hello, Denis.

> Could you please list the limitations of Phase 1?
> I'm curious what won't be supported in 2.7.

1. We will have ability to encrypt data stored on the disk for specific cache.

- There is no limitation on API usage or something for an encrypted 
cache.
- If some cache in a cache group is encrypted, all other caches in this 
group must be encrypted.
- Setting up master key(standard jdk key storage) is prerequisite and 
should be done by an administrator.
- `encryptionEnabled` flag setting up on cache creation and can't be 
changed in runtime.

2. We won't be able to change encryption keys for existing cache(key rotation 
procedure).
This will be implemented in Phase 2.

В Вт, 11/09/2018 в 23:41 -0400, Denis Magda пишет:
> Nikolay,
> 
> Could you please list the limitations of Phase 1? I'm curious what won't be
> supported in 2.7.
> 
> --
> Denis
> 
> On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov  wrote:
> 
> > > As I understand the plan is to get TDE Phase 1 released in 2.7, right?
> > 
> > Yes. We will release TDE in 2.7
> > 
> > > Could you also list what needs to be done at Phase 2 and how much time
> > 
> > it might take.
> > 
> > Yes, I think Dmitry Ryabov will send Phase 2 design
> > 
> > 
> > В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > > Hello, Denis.
> > > 
> > > Yes, Vladimir made 2 rounds of review.
> > > I planning to fix all issues with implementation in a few days.
> > > 
> > > 
> > > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > > > Hi Nikolay,
> > > > 
> > > > Has anybody started looking into your request? As I understand the
> > 
> > plan is
> > > > to get TDE Phase 1 released in 2.7, right?
> > > > 
> > 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > > > 
> > > > Could you also list what needs to be done at Phase 2 and how much time
> > 
> > it
> > > > might take.
> > > > 
> > > > --
> > > > Denis
> > > > 
> > > > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov 
> > 
> > wrote:
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > I want to share with you TDE implementation details.
> > > > > I think it can simplify review and acception of TDE implementation.
> > > > > This mail is copy of wiki page [1].
> > > > > 
> > > > > Please, share your thoughts.
> > > > > 
> > > > > TDE is ready for review [2].
> > > > > Please, let me know, who is able to make final review.
> > > > > 
> > > > > This document describes some internal details of TDE.Phase 1
> > > > > implementation [3].
> > > > > I suggest that reader familiar with the general design described in
> > 
> > IEP-18
> > > > > [4].
> > > > > 
> > > > > 
> > > > > Cache group key management and node join enhancements:
> > > > > 
> > > > > 1. GridEncryptionManager encapsulates all logic related to
> > 
> > key
> > > > > management:
> > > > > a. All group encryption keys are stored in MetaStore.
> > > > > 
> > > > > b. Joining node sends to cluster:
> > > > > * Master key digest.
> > > > > * All group keys saved locally (Map > > > > byte[]>). Keys send over a network in encrypted form.
> > > > > 
> > > > > c. Coordinator on new node join:
> > > > > * Compares new node master key digest with
> > 
> > the
> > > > > local master key digest.
> > > > > If differs then new node join is rejected.
> > > > > 
> > > > > * Compares local and received group keys.
> > > > > If some key differs then new node join is
> > > > > rejected.
> > > > > 
> > > > > d. All server nodes:
> > > > > * If some of received keys are new then they
> > 
> > save
> > > > > locally.
> > > > > 
> > > > > 2. Dynamic cache creation:
> > > > >  

Re: TDE Implementation details.

2018-09-11 Thread Nikolay Izhikov
> As I understand the plan is to get TDE Phase 1 released in 2.7, right?

Yes. We will release TDE in 2.7

> Could you also list what needs to be done at Phase 2 and how much time it 
> might take.

Yes, I think Dmitry Ryabov will send Phase 2 design 


В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> Hello, Denis.
> 
> Yes, Vladimir made 2 rounds of review.
> I planning to fix all issues with implementation in a few days.
> 
> 
> В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > Hi Nikolay,
> > 
> > Has anybody started looking into your request? As I understand the plan is
> > to get TDE Phase 1 released in 2.7, right?
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > 
> > Could you also list what needs to be done at Phase 2 and how much time it
> > might take.
> > 
> > --
> > Denis
> > 
> > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > I want to share with you TDE implementation details.
> > > I think it can simplify review and acception of TDE implementation.
> > > This mail is copy of wiki page [1].
> > > 
> > > Please, share your thoughts.
> > > 
> > > TDE is ready for review [2].
> > > Please, let me know, who is able to make final review.
> > > 
> > > This document describes some internal details of TDE.Phase 1
> > > implementation [3].
> > > I suggest that reader familiar with the general design described in IEP-18
> > > [4].
> > > 
> > > 
> > > Cache group key management and node join enhancements:
> > > 
> > > 1. GridEncryptionManager encapsulates all logic related to key
> > > management:
> > > a. All group encryption keys are stored in MetaStore.
> > > 
> > > b. Joining node sends to cluster:
> > > * Master key digest.
> > > * All group keys saved locally (Map > > byte[]>). Keys send over a network in encrypted form.
> > > 
> > > c. Coordinator on new node join:
> > > * Compares new node master key digest with the
> > > local master key digest.
> > > If differs then new node join is rejected.
> > > 
> > > * Compares local and received group keys.
> > > If some key differs then new node join is
> > > rejected.
> > > 
> > > d. All server nodes:
> > > * If some of received keys are new then they save
> > > locally.
> > > 
> > > 2. Dynamic cache creation:
> > > a. On server node - Encryption key is generated and added
> > > to DynamicCacheChangeRequest.
> > > 
> > > b. On client node:
> > > * Prior to creation of DynamicCacheChangeRequest
> > > we have to get new encryption key from some server node.
> > > * New request added to solve this -
> > > GenerateEncryptionKeyRequest.
> > > 
> > > 
> > > WAL Record encryption:
> > > 
> > > 
> > > 1. New type of WAL record created – EncryptedRecord.
> > > 
> > > 2. EncryptedRecord is a container that stores some
> > > WalRecordCacheGroupAware in encrypted form inside.
> > > 
> > > 3. Write:
> > > Each record for an encrypted group that implements
> > > WalRecordCacheGroupAware written to WAL in encrypted form.
> > > Instead of that record we write EncryptedRecord with plain record
> > > inside(PageSnapshot, PageDeltaRecord, etc).
> > > 
> > > 4. Read: There are 3 different cases on EncryptedRecord read:
> > > a. WAL restore – we read EncryptedRecord, decrypt internal
> > > record and return it.
> > > 
> > > b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> > > EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return 
> > > it
> > > from an iterator.
> > > 
> > > c. Meta storage restore process – On node start we restore
> > > MetaStore.
> > > When we do it no encryption keys are available, because
> > > they are stored in MetaStore.
> > > So we can’t decrypt EncryptedRecord

Re: TDE Implementation details.

2018-09-11 Thread Nikolay Izhikov
Hello, Denis.

Yes, Vladimir made 2 rounds of review.
I planning to fix all issues with implementation in a few days.


В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> Hi Nikolay,
> 
> Has anybody started looking into your request? As I understand the plan is
> to get TDE Phase 1 released in 2.7, right?
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> 
> Could you also list what needs to be done at Phase 2 and how much time it
> might take.
> 
> --
> Denis
> 
> On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:
> 
> > Hello, Igniters.
> > 
> > I want to share with you TDE implementation details.
> > I think it can simplify review and acception of TDE implementation.
> > This mail is copy of wiki page [1].
> > 
> > Please, share your thoughts.
> > 
> > TDE is ready for review [2].
> > Please, let me know, who is able to make final review.
> > 
> > This document describes some internal details of TDE.Phase 1
> > implementation [3].
> > I suggest that reader familiar with the general design described in IEP-18
> > [4].
> > 
> > 
> > Cache group key management and node join enhancements:
> > 
> > 1. GridEncryptionManager encapsulates all logic related to key
> > management:
> > a. All group encryption keys are stored in MetaStore.
> > 
> > b. Joining node sends to cluster:
> > * Master key digest.
> > * All group keys saved locally (Map > byte[]>). Keys send over a network in encrypted form.
> > 
> > c. Coordinator on new node join:
> > * Compares new node master key digest with the
> > local master key digest.
> > If differs then new node join is rejected.
> > 
> > * Compares local and received group keys.
> > If some key differs then new node join is
> > rejected.
> > 
> > d. All server nodes:
> > * If some of received keys are new then they save
> > locally.
> > 
> > 2. Dynamic cache creation:
> > a. On server node - Encryption key is generated and added
> > to DynamicCacheChangeRequest.
> > 
> > b. On client node:
> > * Prior to creation of DynamicCacheChangeRequest
> > we have to get new encryption key from some server node.
> > * New request added to solve this -
> > GenerateEncryptionKeyRequest.
> > 
> > 
> > WAL Record encryption:
> > 
> > 
> > 1. New type of WAL record created – EncryptedRecord.
> > 
> > 2. EncryptedRecord is a container that stores some
> > WalRecordCacheGroupAware in encrypted form inside.
> > 
> > 3. Write:
> > Each record for an encrypted group that implements
> > WalRecordCacheGroupAware written to WAL in encrypted form.
> > Instead of that record we write EncryptedRecord with plain record
> > inside(PageSnapshot, PageDeltaRecord, etc).
> > 
> > 4. Read: There are 3 different cases on EncryptedRecord read:
> > a. WAL restore – we read EncryptedRecord, decrypt internal
> > record and return it.
> > 
> > b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> > EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return it
> > from an iterator.
> > 
> > c. Meta storage restore process – On node start we restore
> > MetaStore.
> > When we do it no encryption keys are available, because
> > they are stored in MetaStore.
> > So we can’t decrypt EncryptedRecord and just return it
> > from an iterator.
> > We don't need decrypted record on this step to operate
> > properly.
> > 
> > 
> > Page encryption:
> > 
> > 
> > 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc)
> > for gain maximum perfrormance.
> > 
> > 2. There is a 16 byte overhead for and AES CBC mode.
> > 
> > 3. So we have to preserve 16 bytes in page in memory to get
> > encrypted page size equal to 2^n when written it to disk.
> > 
> > 4. PageIO has many methods with pageSize parameter.
> > So for encrypted cache groups page size is calculated as
> > cfg.getPageSize() - 16 byte.
> > 
> > 
> > [1]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > [2] https://github.com/apache/ignite/pull/4167
> > [3] https://issues.apache.org/jira/browse/IGNITE-8485
> > [4]
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption
> > 

signature.asc
Description: This is a digitally signed message part


Re: Ignite Spark Bugs

2018-09-11 Thread Nikolay Izhikov
Hello, Denis.

Thanks for providing reproducers for bugs!

I taked a look at your project and be able to reproduce some of errors.
I will create a ticket and start investigation in a next few days.

В Вт, 11/09/2018 в 12:28 +0300, Dmitriy Pavlov пишет:
> Hi Denis,
> 
> Thank you for bringing this here and for your efforts to reproduce it.
> Would you like to create an issue and contribute these test into Ignite
> code base?
> 
> If you would like to proceed with the patch submission process, please
> sign-in to Apache JIRA and share your JIRA username, I will add you as the
> contributor.
> 
> You can count on my support.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> вт, 11 сент. 2018 г. в 10:51, Денис Костин :
> 
> > Hello everybody!
> > 
> > I found a few bugs in the actual version of Apache Ignite Spark (2.6.0)
> > and described in my GitHub: https://github.com/x-x-z/IgniteSparkBugs
> > 
> > I wrote a test for each case:
> > https://github.com/x-x-z/IgniteSparkBugs/blob/master/src/test/java/org/xxz/ignite_spark_bugs/IgniteSparkBugsApplicationTests.java
> > 
> > Check, please.
> > 
> > 
> > Best regards,
> > Denis Kostin
> > 

signature.asc
Description: This is a digitally signed message part


Re: Table Names in Spark Catalog

2018-09-04 Thread Nikolay Izhikov
Hello, Stuart.

Sorry for the silence.

I was swamped the last couple of days.

I think you can go forward and implement suggested solution.
I'm -0 with it.
So no block from my side, but I'm still no happy with abstractions :).

В Пн, 03/09/2018 в 09:35 +0100, Stuart Macdonald пишет:
> Nikolay, Val, it would be good if we could reach agreement here so that I
> can make the necessary modifications before the 2.7 cutoff.
> 
> Nikolay - would you be comfortable if I went ahead and made database=schema?
> 
> Stuart.
> 
> On Mon, Aug 27, 2018 at 10:22 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> 
> > Hi Nikolay,
> > 
> > I think it's actually pretty unfortunate that Spark uses term "database"
> > here, as it essentially refers to a schema in my view. Usually, database is
> > something you create a physical connection to, and connection is bind to
> > that database. To connect to another database you need to create a new
> > connection. In Spark, however, you can switch between "databases" within a
> > single session, which looks really weird to me because it's usually a
> > characteristic of a schema. Having said that, I understand your concern,
> > but I don't think there is an ideal solution.
> > 
> > As for your approach, I still don't understand how it will allow to fully
> > support schemas in catalog.
> > - How will you get a list of tables within a particular schema? In other
> > words, what would listTables() method return?
> > - How will you switch between the schemas?
> > - Etc.
> > 
> > I still think assuming database=schema is the best we can do here, but I
> > would be happy to hear another opinions from other community members.
> > 
> > OPTION_SCHEMA should definitely be introduced though (I thought we already
> > did, no?). CREATE TABLE will be supported with this ticket:
> > https://issues.apache.org/jira/browse/IGNITE-5780. For now we will have to
> > throw an exception if custom schema name is provided when creating a Spark
> > session, but table does not exist yet.
> > 
> > -Val
> > 
> > On Sun, Aug 26, 2018 at 7:56 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Igniters,
> > > 
> > > Personally, I don't like the solution with database == schema name.
> > > 
> > > 1. I think we should try to use the right abstractions.
> > > schema == database doesn't sound right for me.
> > > 
> > > Do you want to answer to all of our users something like that:
> > > 
> > > - "How I can change Ignite SQL schema?"
> > > - "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".
> > > 
> > > 2. I think we restrict whole solution with that decision.
> > > If Ignite will support multiple databases in the future we just don't
> > 
> > have
> > > a place for it.
> > > 
> > > I think we should do the following:
> > > 
> > > 1. IgniteExternalCatalog should be able to return *ALL* tables
> > > within Ignite instance.
> > > We shouldn't restrict tables list by schema by default.
> > > We should return tables with schema name - `schema.table`
> > > 
> > > 2. We should introduce `OPTION_SCHEMA` for a dataframe to specify
> > > a schema.
> > > 
> > > There is an issue with the second step: We can't use schema name
> > > in `CREATE TABLE` clause.
> > > This is restriction of current Ignite SQL.
> > > 
> > > I propose to make the following:
> > > 
> > > 1. For all write modes that requires the creation of table we
> > > should disallow usage of table outside of `SQL_PUBLIC`
> > > or usage of `OPTION_SCHEMA`. We should throw proper exception for
> > > this case.
> > > 
> > > 2. Create a ticket to support `CREATE TABLE` with custom schema
> > > name.
> > > 
> > > 3. After resolving ticket from step 2 we can add full support of
> > > custom schema to Spark integration.
> > > 
> > > 4. We should throw an exception if user try to use setDatabase.
> > > 
> > > Is that makes sense for you?
> > > 
> > > В Вс, 26/08/2018 в 14:09 +0100, Stuart Macdonald пишет:
> > > > I'll go ahead and make the changes to represent the schema name as the
> > > > database name for the purposes of the Spark catalog.
> > > > 
> > > > If anyone knows of an e

Re: Possible problems with closing AsyncFileIO

2018-09-04 Thread Nikolay Izhikov
Hello, Sergey.

If some resources are not closed after usage it certainly a bug.
Can you create a ticket and attach a simplest reproducer for desribed issue?

В Вт, 04/09/2018 в 17:44 +0300, Sergey Antonov пишет:
> Hello, Igniters!
> 
>  I found bug in org.apache
> .ignite.internal.processors.cache.persistence.file.AsyncFileIO#close(): If
> one of  async futures (ChannelOpFuture) throw IgniteCheckedException
> AsynchronousFileChannel associated with AsyncFileIO instance will not be
> closed.
> It's correct behaviour or bug?
> 

signature.asc
Description: This is a digitally signed message part


Re: Python thin client

2018-09-04 Thread Nikolay Izhikov
Hello, Dmitry.

I understand that for experienced Python developer it obvious from stack trace 
I send.

But can we check python version on startup?
And print big fat error message "You are using wrong python version".

From my experience, there are some tickets in Ignite that should be implemented 
in various thin clients.
It a very trivial changes, but lack of testability makes this task harder then 
steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as clear as 
possible.

В Пн, 03/09/2018 в 17:07 +1000, Dmitry Melnichuk пишет:
> Hello, Nikolay!
> 
> Thank you for your interest in Python thin client.
> 
> The traceback suggests that you using Python 2.7. Unfortunately, my 
> client supports only Python versions 3.4 and later.
> 
> Since you are using the latest Ubuntu OS, chances are you already have a 
> recent Python 3 installed. Try `pip3`.
> 
> On 9/3/18 4:25 PM, Nikolay Izhikov wrote:
> > Hello, Dmitry.
> > 
> > I tried to build your lib locally and it failed.
> > Error message and pip version are below.
> > 
> > Seems, we have to fix developer instructions.
> > Do we need specific version of pip or something?
> > I tried to use default versions.
> > I using Ubuntu Linux.
> > 
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v 
> > -e .
> > Obtaining file:///home/dragon/src/ignite/modules/platforms/python
> >Running setup.py 
> > (path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info 
> > for package from file:///home/dragon/src/ignite/modules/platforms/python
> >  Running command python setup.py egg_info
> >  Traceback (most recent call last):
> >File "", line 1, in 
> >File "/home/dragon/src/ignite/modules/platforms/python/setup.py", 
> > line 20
> >  def is_a_requirement(line: str) -> bool:
> >   ^
> >  SyntaxError: invalid syntax
> > Cleaning up...
> > Command "python setup.py egg_info" failed with error code 1 in 
> > /home/dragon/src/ignite/modules/platforms/python/
> > Exception information:
> > Traceback (most recent call last):
> >File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in 
> > main
> >  status = self.run(options, args)
> >File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 
> > 353, in run
> >  wb.build(autobuilding=True)
> >File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
> >  self.requirement_set.prepare_files(self.finder)
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in 
> > prepare_files
> >  ignore_dependencies=self.ignore_dependencies))
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in 
> > _prepare_file
> >  abstract_dist.prep_for_dist()
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in 
> > prep_for_dist
> >  self.req_to_install.run_egg_info()
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 
> > 439, in run_egg_info
> >  command_desc='python setup.py egg_info')
> >File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, 
> > in call_subprocess
> >  % (command_desc, proc.returncode, cwd))
> > InstallationError: Command "python setup.py egg_info" failed with error 
> > code 1 in /home/dragon/src/ignite/modules/platforms/python/
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache 
> > policy python-pip
> > python-pip:
> >Установлен: 9.0.1-2.3~ubuntu1
> >Кандидат:   9.0.1-2.3~ubuntu1
> >Таблица версий:
> >   *** 9.0.1-2.3~ubuntu1 500
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe 
> > amd64 Packages
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe 
> > i386 Packages
> >  100 /var/lib/dpkg/status
> >   9.0.1-2 500
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 
> > Packages
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 
> > Packages
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a
> > Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 
> > 2018 x86_64 x86_64 x86_64 GNU/Linux
> > 

signature.asc
Description: This is a digitally signed message part


Re: Python thin client

2018-09-03 Thread Nikolay Izhikov
Hello, Dmitry.

I tried to build your lib locally and it failed.
Error message and pip version are below.

Seems, we have to fix developer instructions.
Do we need specific version of pip or something? 
I tried to use default versions.
I using Ubuntu Linux.

dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v -e .
Obtaining file:///home/dragon/src/ignite/modules/platforms/python
  Running setup.py 
(path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info for 
package from file:///home/dragon/src/ignite/modules/platforms/python
Running command python setup.py egg_info
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/dragon/src/ignite/modules/platforms/python/setup.py", line 20
def is_a_requirement(line: str) -> bool:
 ^
SyntaxError: invalid syntax
Cleaning up...
Command "python setup.py egg_info" failed with error code 1 in 
/home/dragon/src/ignite/modules/platforms/python/
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in 
run
wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in 
prepare_files
ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in 
_prepare_file
abstract_dist.prep_for_dist()
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in 
prep_for_dist
self.req_to_install.run_egg_info()
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in 
run_egg_info
command_desc='python setup.py egg_info')
  File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, in 
call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 
in /home/dragon/src/ignite/modules/platforms/python/
dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache policy 
python-pip
python-pip:
  Установлен: 9.0.1-2.3~ubuntu1
  Кандидат:   9.0.1-2.3~ubuntu1
  Таблица версий:
 *** 9.0.1-2.3~ubuntu1 500
500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 
Packages
500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe i386 
Packages
100 /var/lib/dpkg/status
 9.0.1-2 500
500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 Packages
dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a
Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 
x86_64 x86_64 x86_64 GNU/Linux

В Пн, 03/09/2018 в 16:10 +1000, Dmitry Melnichuk пишет:
> Hello, Igniters!
> 
> Please review my work on Ignite thin client library written in Python 3.
> 
> Pull request:
> https://github.com/apache/ignite/pull/4278
> 
> Jira issue with initial proposal:
> https://issues.apache.org/jira/browse/IGNITE-7782
> 
> The documentation is temporarily available at:
> https://apache-ignite-binary-protocol-client.readthedocs.io/
> 
> It covers installation, requirements, API specification, coding 
> examples, instructions on how to run tests and build the documentation 
> itself.
> 
> Dmitry

signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-08-29 Thread Nikolay Izhikov
Hell, Yakov

I'm ok with your proposal.

* Scope freeze - September 17 - We should have a full list of tickets 
for 2.7 here.
* Code freeze - October 01 - We should merge all 2.7 tickets to master 
here.
* Vote on RC1 - October 11.
* Vote on release - October 15.

В Ср, 29/08/2018 в 12:39 +0300, Yakov Zhdanov пишет:
> Nikolay,
> 
> I think we should have 2 weeks after code freeze which by the way may
> include RC1 voting stage. This way I would like us to agree that release
> candidate should be sent to vote on Oct, 11th and we can release on Oct,
> 15th.
> 
> What do you think?
> 
> --Yakov

signature.asc
Description: This is a digitally signed message part


[jira] [Created] (IGNITE-9403) TDE - Phase-1. Thin clients

2018-08-28 Thread Nikolay Izhikov (JIRA)
Nikolay Izhikov created IGNITE-9403:
---

 Summary: TDE - Phase-1. Thin clients
 Key: IGNITE-9403
 URL: https://issues.apache.org/jira/browse/IGNITE-9403
 Project: Ignite
  Issue Type: Sub-task
Reporter: Nikolay Izhikov


We should provide support for a new {{encryptionEnabled}} flag in cache 
configuration for all thin clients:

  - .Net
  - Java
  -  NodeJs
  - Python
  - PHP
  - C++(for now it doesn't have support of {{CacheConfiguration}})

Backward compatibility should be preserved.
A contributor can take 
[commit](https://github.com/apache/ignite/commit/baab0a6ddd3973fb8fca6ecb0a7d841d7d3a72be)
 as an example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Apache Ignite 2.7 release

2018-08-28 Thread Nikolay Izhikov
Hello, Igniters.

I think we should discuss the release schedule.

Current dates are following:

* Code Freeze: September 30, 2018
* Voting Date: October 1, 2018
* Release Date: October 15, 2018

We discussed it privately with Vladimir Ozerov.

Is seems better to reschedule a bit:

* Scope freeze - September 17 - We should have a full list of tickets 
for 2.7 here.
* Code freeze - October 01 - We should merge all 2.7 tickets to master 
here.
* Vote - October 08.

What do you think?


В Сб, 25/08/2018 в 00:57 +0300, Dmitriy Pavlov пишет:
> I hope Vyacheslav can comment better than me. I suppose it is, more or
> less, rectifications and clarifications of design aspects. Not overall
> redesign.
> 
> I also hope Igniters, especially Services experts, will join the discussion
> in the separate topic. Now after a couple of days there is no reaction.
> 
> сб, 25 авг. 2018 г. в 0:53, Dmitriy Setrakyan :
> 
> > On Fri, Aug 24, 2018 at 2:50 PM, Dmitriy Pavlov 
> > wrote:
> > 
> > > Hi Dmitriy, I suppose it highly depends on how fast community will come
> > 
> > to
> > > a consensus about design. So it is up to us to make this happen
> > > 
> > 
> > I am confused then. If we are still discussing design, then we are miles
> > away. Do you know if there anything in service grid that has already been
> > implemented and can be released?
> > 

signature.asc
Description: This is a digitally signed message part


Re: Apache Ignite 2.7 release

2018-08-28 Thread Nikolay Izhikov
Hello, Pavel.

IGNITE-6055 merged to master.

Please, update your code accordingly.
You can take edits in CPP, .Net, NodeJS thin clients as example.

В Пт, 24/08/2018 в 21:49 +0300, Nikolay Izhikov пишет:
> PR is fully ready.
> 
> I'm waiting final approve from Vladimir Ozerov.
> 
> Hope, This will be merged in master on Monday :).
> 
> В Пт, 24/08/2018 в 11:47 -0700, Pavel Petroshenko пишет:
> > Hi Nikolay,
> > 
> > Thank you for the heads up. The dependency should be resolved before the 
> > clients are merged to master, so the sooner you merge the PR the better.
> > 
> > p.
> > 
> > On Fri, Aug 24, 2018 at 12:47 AM, Nikolay Izhikov  
> > wrote:
> > > Hello, Pavel.
> > > 
> > > Please, be aware of IGNITE-6055 [1]
> > > 
> > > I'm edit thin protocol in that ticket.
> > > I can't support changes in Python and PHP clients, because, they are not 
> > > merged in master, yet.
> > > Write me, If you have any questions about new fields.
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6055
> > > 
> > > В Чт, 23/08/2018 в 18:02 -0700, Pavel Petroshenko пишет:
> > > > Hi Nikolay,
> > > > 
> > > > Python [1], PHP [2], and Node.js [3] thin clients will get into the 
> > > > release.
> > > > 
> > > > Thanks,
> > > > p.
> > > > 
> > > > [1] https://jira.apache.org/jira/browse/IGNITE-7782
> > > > [2] https://jira.apache.org/jira/browse/IGNITE-7783
> > > > [3] https://jira.apache.org/jira/browse/IGNITE-
> > > > 
> > > > 
> > > > On Tue, Aug 21, 2018 at 12:20 PM, Dmitriy Setrakyan 
> > > >  wrote:
> > > > > Thanks, Nikolay!
> > > > > 
> > > > > I think it is important to include the links to all important Jira 
> > > > > tickets
> > > > > in this thread, so that the community can track them.
> > > > > 
> > > > > D.
> > > > > 
> > > > > On Tue, Aug 21, 2018 at 12:06 AM, Nikolay Izhikov 
> > > > > 
> > > > > wrote:
> > > > > 
> > > > > > Hello, Dmitriy.
> > > > > > 
> > > > > > I think Transparent Data Encryption will be available in 2.7
> > > > > > 
> > > > > > В Пн, 20/08/2018 в 13:20 -0700, Dmitriy Setrakyan пишет:
> > > > > > > Hi Nikolay,
> > > > > > > 
> > > > > > > Thanks for being the release manager!
> > > > > > > 
> > > > > > > I am getting a bit lost in all these tickets. Can we specify some
> > > > > > > high-level tickets, that are not plain bug fixes, which will be
> > > > > > 
> > > > > > interesting
> > > > > > > for the community to notice?
> > > > > > > 
> > > > > > > For example, here are some significant tasks that the community 
> > > > > > > is either
> > > > > > > working on or has been working on:
> > > > > > > 
> > > > > > > - Node.JS client
> > > > > > > - Python client
> > > > > > > - Transactional SQL (MVCC)
> > > > > > > - service grid stabilization
> > > > > > > - SQL memory utilization improvements
> > > > > > > - more?
> > > > > > > 
> > > > > > > Can you please solicit status from the community for these tasks?
> > > > > > > 
> > > > > > > D.
> > > > > > > 
> > > > > > > On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov 
> > > > > > > 
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hello, Igniters.
> > > > > > > > 
> > > > > > > > I'm release manager of Apache Ignite 2.7.
> > > > > > > > 
> > > > > > > > It's time to start discussion of release. [1]
> > > > > > > > 
> > > > > > > > Current code freeze date is September, 30.
> > > > > > > > If you have any objections - please, responsd to this thread.
> > > > > > > > 
> > > > > > > > [1] https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > 
> > > > > > Apache+Ignite+2.7
> > > > > > 
> > > > 
> > > > 

signature.asc
Description: This is a digitally signed message part


Re: Table Names in Spark Catalog

2018-08-26 Thread Nikolay Izhikov
Igniters, 

Personally, I don't like the solution with database == schema name.

1. I think we should try to use the right abstractions. 
schema == database doesn't sound right for me.

Do you want to answer to all of our users something like that:

- "How I can change Ignite SQL schema?"
- "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".

2. I think we restrict whole solution with that decision.
If Ignite will support multiple databases in the future we just don't have a 
place for it.

I think we should do the following:

1. IgniteExternalCatalog should be able to return *ALL* tables within 
Ignite instance. 
We shouldn't restrict tables list by schema by default.
We should return tables with schema name - `schema.table`

2. We should introduce `OPTION_SCHEMA` for a dataframe to specify a 
schema.

There is an issue with the second step: We can't use schema name in 
`CREATE TABLE` clause.
This is restriction of current Ignite SQL.

I propose to make the following:

1. For all write modes that requires the creation of table we should 
disallow usage of table outside of `SQL_PUBLIC`
or usage of `OPTION_SCHEMA`. We should throw proper exception for this 
case.

2. Create a ticket to support `CREATE TABLE` with custom schema name.

3. After resolving ticket from step 2 we can add full support of custom 
schema to Spark integration.

4. We should throw an exception if user try to use setDatabase.

Is that makes sense for you?

В Вс, 26/08/2018 в 14:09 +0100, Stuart Macdonald пишет:
> I'll go ahead and make the changes to represent the schema name as the
> database name for the purposes of the Spark catalog.
> 
> If anyone knows of an existing way to list all available schemata within an
> Ignite instance please let me know, otherwise the first task will be
> creating that mechanism.
> 
> Stuart.
> 
> On Fri, Aug 24, 2018 at 6:23 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> 
> > Nikolay,
> > 
> > If there are multiple configuration in XML, IgniteContext will always use
> > only one of them. Looks like current approach simply doesn't work. I
> > propose to report schema name as 'database' in Spark. If there are multiple
> > clients, you would create multiple sessions and multiple catalogs.
> > 
> > Makes sense?
> > 
> > -Val
> > 
> > On Fri, Aug 24, 2018 at 12:33 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Valentin.
> > > 
> > > > catalog exist in scope of a single IgniteSparkSession> (and therefore
> > > 
> > > single IgniteContext and single Ignite instance)?
> > > 
> > > Yes.
> > > Actually, I was thinking about use case when we have several Ignite
> > > configuration in one XML file.
> > > Now I see, may be this is too rare use-case to support.
> > > 
> > > Stuart, Valentin, What is your proposal?
> > > 
> > > В Ср, 22/08/2018 в 08:56 -0700, Valentin Kulichenko пишет:
> > > > Nikolay,
> > > > 
> > > > Whatever we decide on would be right :) Basically, we need to answer
> > 
> > this
> > > > question: does the catalog exist in scope of a single
> > 
> > IgniteSparkSession
> > > > (and therefore single IgniteContext and single Ignite instance)? In
> > 
> > other
> > > > words, in case of a rare use case when a single Spark application
> > > 
> > > connects
> > > > to multiple Ignite clusters, would there be a catalog created per
> > > 
> > > cluster?
> > > > 
> > > > If the answer is yes, current logic doesn't make sense.
> > > > 
> > > > -Val
> > > > 
> > > > 
> > > > On Wed, Aug 22, 2018 at 1:44 AM Nikolay Izhikov 
> > > 
> > > wrote:
> > > > 
> > > > > Hello, Valentin.
> > > > > 
> > > > > > I believe we should get rid of this logic and use Ignite schema
> > 
> > name
> > > as
> > > > > 
> > > > > database name in Spark's catalog.
> > > > > 
> > > > > When I develop Ignite integration with Spark Data Frame I use
> > 
> > following
> > > > > abstraction described by Vladimir Ozerov:
> > > > > 
> > > > > "1) Let's consider Ignite cluster as a single database ("catalog" in
> > > 
> > > ANSI
> > > > > SQL'92 terms)." [1]
> > > > > 
> > > > > Am I was wrong? If

  1   2   3   4   >