Re: Code inspection

[Nikolay Izhikov]

Actually, I dont see anything wrong with failing *compilation* task.

I think one should use project code style for everyday coding, not only for
ready-to-merge PRs.

If we cant use code style for everyday coding, we should change the
codestyle.

What do you think?

ср, 13 февр. 2019 г., 10:11 Petr Ivanov mr.wei...@gmail.com:

> I guess that was about failing build configuration with Checkstype, not
> compilation build itself.
>
> > On 12 Feb 2019, at 18:03, Павлухин Иван  wrote:
> >
> > Folks,
> >
> > Are you going to fail job compiling Ignite sources [1] if some
> > inspection found a problem? Can we avoid it? It is quite common
> > pattern to start some feature implementation with making a sketch and
> > running tests against it. I found it convenient to skip some style
> > requirements for such sketches (e.g. well formed javadocs).
> >
> > [1]
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_BuildApacheIgnite
> >
> > пн, 11 февр. 2019 г. в 11:38, Nikolay Izhikov :
> >>
> >> Petr, we should have 1 configuration for project, may be 1 configuration
> >> per programming language.
> >>
> >> пн, 11 февр. 2019 г., 11:33 Petr Ivanov mr.wei...@gmail.com:
> >>
> >>> I was asking about how many build configuration is intended? One for
> all
> >>> and multiple per module?
> >>>
> >>> With IDEA inspections it was going to be build configuration per
> module.
> >>>
> >>>
> >>>
> >>>
>  On 11 Feb 2019, at 11:24, Nikolay Izhikov 
> wrote:
> 
>  Hello, Petr.
> 
>  Are you saying that we have not single build task? And each module
> builds
>  when it required? If yes, then I propose to create a task like
> "Licence
>  check" which will be run for every patch.
> 
>  My point is that violation of codestyle should be treated as hard as
>  compile error.
> 
>  пн, 11 февр. 2019 г., 11:16 Petr Ivanov mr.wei...@gmail.com:
> 
> > Is build configuration Inspections [Core] meant to transform into
> single
> > all-modules check build configuration (without module subdivision)?
> >
> >
> >> On 11 Feb 2019, at 11:02, Nikolay Izhikov 
> wrote:
> >>
> >> Hello, Maxim.
> >>
> >> +1 from me for migrating to checkstyle.
> >>
> >> Oleg, there is plugin for IDEA with 2mln downloads -
> >> https://plugins.jetbrains.com/plugin/1065-checkstyle-idea
> >>
> >> I propose do the following:
> >>
> >> 1. Migrate current checks to checkstyle.
> >> 2. Apply checks to all Ignite modules. Currently, only core module
> are
> >> checked.
> >> I will review and commit this patch, or do it by my own.
> >>
> >> 3. Include code style checks to "Build Apache Ignite" suite. Ignite
> has
> > to
> >> fail to build if patch violates codestyle.
> >>
> >> вс, 10 февр. 2019 г. в 07:54, Павлухин Иван :
> >>
> >>> Hi,
> >>>
> >>> I also think that some warning from IDEA that some code style rule
> is
> >>> violated is a must-have.
> >>>
> >>> вс, 10 февр. 2019 г. в 01:58, oignatenko  >:
> 
>  Hi Maxim,
> 
>  I believe that whatever style checks we establish at Teamcity, we
> > better
>  take care of making it easy for developers to find and fix
> violations
> > in
>  their typical dev environment (for Ignite this means, in IDEA). I
> >>> think
> >>> it
>  is important that developers can maintain required style with
> minimal
> >>> effort
>  on their side.
> 
>  If above is doable then I am 200% for migrating our Teamcity
> > inspections
> >>> to
>  checkstyle / maven.
> 
>  This is because I am very disappointed observing how it stays
> broken
> > for
> >>> so
>  long. And worst of all, even when (if) it is fixed, I feel we will
> >>> always be
>  at risk that it breaks again and that we will have to again wait
> for
> >>> months
>  for it to be fixed.
> 
>  This is such a stark contrast with my experience regarding
> checkstyle
> >>> based
>  inspections. These just work and you just never fear that it is
> going
> > to
>  break for some obscure reason, this is so much better than what I
> > observe
>  now.
> 
>  One suggestion in case if we pick checkstyle - I recommend keeping
> >>> its
>  config file somewhere in the project under version control. I
> used to
>  maintain such a shared style config at one of past jobs and after
> >>> some
>  experimenting it turned out most convenient to have it this way -
> so
> > that
>  developers could easily assess and discuss style settings and keep
> > track
> >>> of
>  changes in these. (note how Kafka folks from your link [5] appear
> to
> >>> be
>  doing it this way)
> 
>  regards, Oleg
> 
> 
> 

Re: [MTCGA]: new failures in builds [2979488] needs to be handled

[Dmitriy Pavlov]

Looks like a flaky issue in Spring suite, but not test failure. Is it still
needs to be researched, but it is not a regression.

вт, 12 февр. 2019 г. в 14:56, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *Recently contributed test failed in master
> internal.GridSpringBeanSerializationSelfTest
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8768893146029766837=%3Cdefault%3E=testDetails
>  No changes in the build
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 14:56:51 12-02-2019
>

Re: [MTCGA]: new failures in builds [2818501] needs to be handled

[Dmitriy Pavlov]

Please Ignore, Build 2818501 is too old. It is not present in TC history.

I've created an improvement to ignore failures in past builds.

ср, 13 февр. 2019 г. в 04:36, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New stable failure of a flaky test in master
> IgniteClusterActivateDeactivateTestWithPersistence.testActivateCachesRestore_5_Servers_WithNewCaches
>
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-787329362447508937=%3Cdefault%3E=testDetails
>  No changes in the build
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 04:36:44 13-02-2019
>

Re: Code inspection

[Petr Ivanov]

I guess that was about failing build configuration with Checkstype, not 
compilation build itself.

> On 12 Feb 2019, at 18:03, Павлухин Иван  wrote:
> 
> Folks,
> 
> Are you going to fail job compiling Ignite sources [1] if some
> inspection found a problem? Can we avoid it? It is quite common
> pattern to start some feature implementation with making a sketch and
> running tests against it. I found it convenient to skip some style
> requirements for such sketches (e.g. well formed javadocs).
> 
> [1] 
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_BuildApacheIgnite
> 
> пн, 11 февр. 2019 г. в 11:38, Nikolay Izhikov :
>> 
>> Petr, we should have 1 configuration for project, may be 1 configuration
>> per programming language.
>> 
>> пн, 11 февр. 2019 г., 11:33 Petr Ivanov mr.wei...@gmail.com:
>> 
>>> I was asking about how many build configuration is intended? One for all
>>> and multiple per module?
>>> 
>>> With IDEA inspections it was going to be build configuration per module.
>>> 
>>> 
>>> 
>>> 
 On 11 Feb 2019, at 11:24, Nikolay Izhikov  wrote:
 
 Hello, Petr.
 
 Are you saying that we have not single build task? And each module builds
 when it required? If yes, then I propose to create a task like "Licence
 check" which will be run for every patch.
 
 My point is that violation of codestyle should be treated as hard as
 compile error.
 
 пн, 11 февр. 2019 г., 11:16 Petr Ivanov mr.wei...@gmail.com:
 
> Is build configuration Inspections [Core] meant to transform into single
> all-modules check build configuration (without module subdivision)?
> 
> 
>> On 11 Feb 2019, at 11:02, Nikolay Izhikov  wrote:
>> 
>> Hello, Maxim.
>> 
>> +1 from me for migrating to checkstyle.
>> 
>> Oleg, there is plugin for IDEA with 2mln downloads -
>> https://plugins.jetbrains.com/plugin/1065-checkstyle-idea
>> 
>> I propose do the following:
>> 
>> 1. Migrate current checks to checkstyle.
>> 2. Apply checks to all Ignite modules. Currently, only core module are
>> checked.
>> I will review and commit this patch, or do it by my own.
>> 
>> 3. Include code style checks to "Build Apache Ignite" suite. Ignite has
> to
>> fail to build if patch violates codestyle.
>> 
>> вс, 10 февр. 2019 г. в 07:54, Павлухин Иван :
>> 
>>> Hi,
>>> 
>>> I also think that some warning from IDEA that some code style rule is
>>> violated is a must-have.
>>> 
>>> вс, 10 февр. 2019 г. в 01:58, oignatenko :
 
 Hi Maxim,
 
 I believe that whatever style checks we establish at Teamcity, we
> better
 take care of making it easy for developers to find and fix violations
> in
 their typical dev environment (for Ignite this means, in IDEA). I
>>> think
>>> it
 is important that developers can maintain required style with minimal
>>> effort
 on their side.
 
 If above is doable then I am 200% for migrating our Teamcity
> inspections
>>> to
 checkstyle / maven.
 
 This is because I am very disappointed observing how it stays broken
> for
>>> so
 long. And worst of all, even when (if) it is fixed, I feel we will
>>> always be
 at risk that it breaks again and that we will have to again wait for
>>> months
 for it to be fixed.
 
 This is such a stark contrast with my experience regarding checkstyle
>>> based
 inspections. These just work and you just never fear that it is going
> to
 break for some obscure reason, this is so much better than what I
> observe
 now.
 
 One suggestion in case if we pick checkstyle - I recommend keeping
>>> its
 config file somewhere in the project under version control. I used to
 maintain such a shared style config at one of past jobs and after
>>> some
 experimenting it turned out most convenient to have it this way - so
> that
 developers could easily assess and discuss style settings and keep
> track
>>> of
 changes in these. (note how Kafka folks from your link [5] appear to
>>> be
 doing it this way)
 
 regards, Oleg
 
 
 Mmuzaf wrote
> Igniters,
> 
> I've found that some of the community members have faced with
> `[Inspections] Core suite [1]` is not working well enough on TC. The
> suite has a `FAILED` status for more than 2 months due to some
>>> issues
> in TeamCity application [2]. Current suite behaviour confuses not
>>> only
> new contributors but also other community members. Moreover, this
> suite is no longer checks rules we previously configured. For
> instance, in the master branch, I've found 11 `Unused 

[jira] [Created] (IGNITE-11303) Python thin client: best effort affinity

[Dmitry Melnichuk (JIRA)]

Dmitry Melnichuk created IGNITE-11303:
-

 Summary: Python thin client: best effort affinity
 Key: IGNITE-11303
 URL: https://issues.apache.org/jira/browse/IGNITE-11303
 Project: Ignite
  Issue Type: Task
  Components: thin client
Reporter: Dmitry Melnichuk
Assignee: Dmitry Melnichuk


The goal is to implement 
[IEP-23|https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients]
 using background thread (`threading` module).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Python examples are missing in Ignite 2.7

[Dmitry Melnichuk]

We just browse through the contents of the latest Ignite bundles [1].

Source bundle [2] contains the examples all right; they are in this
folder:

/apache-ignite-2.7.0-src/modules/platforms/python/examples/

Binary bundle [3] do not contain examples. I'm not totally sure, but
the reasoning I gave earlier might apply to binary bundle too.

Can anyone who's aware of the bundling procedure/scripts chime in,
please?

[1] https://ignite.apache.org/download.cgi

[2] 
http://mirrors.standaloneinstaller.com/apache//ignite/2.7.0/apache-ignite-2.7.0-src.zip

[3] 
http://mirrors.standaloneinstaller.com/apache//ignite/2.7.0/apache-ignite-2.7.0-bin.zip

On Wed, 2019-02-13 at 12:49 +1000, Dmitry Melnichuk wrote:
> Denis,
> 
> If by “release procedure” you mean the contents of the PyPI package,
> then it is not a bug, but a deliberate decision, that was documented
> in
> README [1]: 
> 
> > Installation
> > 
> > - for end user
> > 
> > If you only want to use the pyignite module in your project, do:
> > 
> > $ pip install pyignite
> > 
> > -- for developer
> > 
> > If you want to run tests, examples or build documentation, clone
> > the
> > whole repository…
> 
> The reasoning:
> 
> 1. The examples do not have much value by themselves. They are useful
> only in conjunction with the documentation. If we do not ship the
> documentation via PyPI, then we should not ship the examples either.
> 
> 2. In production environment, the extra packaged stuff like examples
> will be just a waste of space.
> 
> 3. Most Python libraries and frameworks I know of, e.g. Django or
> Scrapy, use the same approach: they have examples and test apps in
> the
> repository and reference them through their docs, but do not ship
> them 
> via PyPI.
> 
> [1] 
> https://github.com/apache/ignite/tree/master/modules/platforms/python#installation
> 
> On Tue, 2019-02-12 at 16:23 -0800, Denis Magda wrote:
> > Igniters,
> > 
> > Seems python examples were not added to the release bundle?
> > https://apacheignite.readme.io/v2.7/docs/python-thin-client#section-running-an-example
> > 
> > There is no "examples" folder for Python. Any flaws in the release
> > procedure?
> > 
> > -
> > Denis

Re: Python examples are missing in Ignite 2.7

[Dmitry Melnichuk]

Denis,

If by “release procedure” you mean the contents of the PyPI package,
then it is not a bug, but a deliberate decision, that was documented in
README [1]: 

> Installation
> 
> - for end user
> 
> If you only want to use the pyignite module in your project, do:
> 
> $ pip install pyignite
> 
> -- for developer
> 
> If you want to run tests, examples or build documentation, clone the
> whole repository…

The reasoning:

1. The examples do not have much value by themselves. They are useful
only in conjunction with the documentation. If we do not ship the
documentation via PyPI, then we should not ship the examples either.

2. In production environment, the extra packaged stuff like examples
will be just a waste of space.

3. Most Python libraries and frameworks I know of, e.g. Django or
Scrapy, use the same approach: they have examples and test apps in the
repository and reference them through their docs, but do not ship them 
via PyPI.

[1] 
https://github.com/apache/ignite/tree/master/modules/platforms/python#installation

On Tue, 2019-02-12 at 16:23 -0800, Denis Magda wrote:
> Igniters,
> 
> Seems python examples were not added to the release bundle?
> https://apacheignite.readme.io/v2.7/docs/python-thin-client#section-running-an-example
> 
> There is no "examples" folder for Python. Any flaws in the release
> procedure?
> 
> -
> Denis

[MTCGA]: new failures in builds [2818501] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New stable failure of a flaky test in master 
IgniteClusterActivateDeactivateTestWithPersistence.testActivateCachesRestore_5_Servers_WithNewCaches
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-787329362447508937=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 04:36:44 13-02-2019 

Python examples are missing in Ignite 2.7

[Denis Magda]

Igniters,

Seems python examples were not added to the release bundle?
https://apacheignite.readme.io/v2.7/docs/python-thin-client#section-running-an-example

There is no "examples" folder for Python. Any flaws in the release
procedure?

-
Denis

Re: Binary clients: fallback to the previous versions of the protocol

[Dmitry Melnichuk]

Igor,

Thank you for your explanation. I think the matter begins to clear up
for me now.

The backward compatibility issue you described can not be applied to
Python, because Python applications, unlike Java ones, do not have to
be built. They rely on package manager (pip, conda, et c.) to run
anywhere, including production.

At the stage of deployment, the package manager collects dependencies
using a specially crafted response file, often called
`requirements.txt`.

For example, to ensure that their application will work with the
current _and_ future minor versions of pyignite, the user may include a
line in their `requirements.txt` file:

pyignite < x

where x is a next major version number. In compliance with semantic
versioning, the line is basically says: “Use the latest available
version, that is earlier than x”.

When upgrading Ignite server, system administrator or devops engineer
must also update or recreate the app's environment, or update OS-level
packages, or redeploy the app using Docker − the exact procedure may
vary, but in any case it should be completely standard − to deliver the
latest suitable dependencies.

And then the same app connects to a latest Ignite server.

Here is more about how pip understands versions:

https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers

What we really need to do for this to work seamlessly, is to establish
the clear relation between products' versions. Regretfully, I have not
done this before; just did not expect for this issue to come up. I
think it would be best for pyignite major and minor to be set according
to the Ignite binary protocol versions, i.e. pyignite 1.2.z handles
Ignite binary protocol v1.2, and so on. But that is another matter.

On Tue, 2019-02-12 at 13:39 +0300, Igor Sapego wrote:
> Hi,
> 
> Well, this approach was used for quite some time in ODBC and JDBC,
> so thin client has just inherited it, as we have the same port and
> the
> same handshake message header. Maybe that's why you could not find
> any mention of versioning in thin client context.
> 
> 3. Let's start from this point as this is the root of the issue.
> Preconditions
> are simple - the backward compatibility. We should guarantee that
> user
> can use any version of thin client with any version of server while
> they
> share the same major version. This can be important, if someone have
> user applications built with thin client and they don't want to break
> their
> user applications, when they just want to update servers. What is
> appropriate here is to give user some kind of warning that they use
> outdated client or server.
> 
> Can you explain how your approach is going to work in the example
> above? I'm not quite sure I understand you correctly.
> 
> 1. The API for this can be as simple as a set of allowed version in
> a configuration for both server and client. Maybe we should discuss
> such
> kind of API as it can be useful for example in terms of security.
> 
> 2. Complexity is the price developers have to pay for better user
> experience.
> Nothing new here. Software is not designed for developers, it's
> designed
> for a user.
> 
> Best Regards,
> Igor
> 
> 
> On Tue, Feb 12, 2019 at 11:56 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
> 
> > Hello fellow igniters!
> > 
> > I recently started reconnaissance before the
> > implementation of IEP-23 [1] in Python thin client. I think it is
> > an
> > interesting and much promising feature. As I understand, the
> > changes
> > associated with this feature will result in a new version of the
> > binary
> > protocol.
> > 
> > What bothers me is not the changes required by the
> > implementation itself, but the upcoming changes in the way that the
> > fallback to the previous versions of the protocol is going to be
> > handled.
> > 
> > I have not found a open discussion on this topic neither on
> > this mailing list, nor in Jira. If such a discussion took place,
> > please
> > inform me with a link. Based on what I have heard so far, I made
> > the
> > following conclusions:
> > 
> > 1) the client is now expected to handle multiple
> > binary protocol versions;
> > 2) since the protocol is implemented the way
> > the client first reports its version to the server on handshake,
> > and
> > not the other way, the client now must try to connect to the server
> > using its preferred version of the protocol, and in case of an
> > error,
> > reconnect with another version.
> > 
> > These changes are raised some concerns
> > for me, for example:
> > 
> > 1) the version negotiation mechanism implemented on
> > client can not be made transparent to the end user. We must give
> > the
> > user an option, so that they could choose from the entire set of
> > versions supported by the client, a subset of the versions their
> > application is designed to work with. That will complicate the user
> > API;
> > 
> > 2) the complexity of the client will also be increased, especially
> > 

[jira] [Created] (IGNITE-11302) idleConnectionTimeout TcpComm different on server and client (client default > server custom) lead to wait until client timeout on server side

[ARomantsov (JIRA)]

ARomantsov created IGNITE-11302:
---

 Summary: idleConnectionTimeout TcpComm different on server and 
client (client default > server custom) lead to wait until client timeout on 
server side
 Key: IGNITE-11302
 URL: https://issues.apache.org/jira/browse/IGNITE-11302
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: ARomantsov
 Fix For: 2.8


Server config:





Client config





Server wait until default idleConnectionTimeout (10 m) for client fail.
If both config with idleConnectionTimeout=1 s - ignite worked according to 
config




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11301) MVCC: Remove extra MVCC initialization messages from logs

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-11301:
---

 Summary: MVCC: Remove extra MVCC initialization messages from logs
 Key: IGNITE-11301
 URL: https://issues.apache.org/jira/browse/IGNITE-11301
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Affects Versions: 2.7
Reporter: Ivan Pavlukhin


Currently upon each MVCC cache start we check that MVCC facilities are started 
on a node. As a result we can see many unneeded messages like:
{{[INFO ]... Mvcc processor started.}}
{{[WARN ]... Attempting to start active vacuum.}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Code inspection

[Павлухин Иван]

Folks,

Are you going to fail job compiling Ignite sources [1] if some
inspection found a problem? Can we avoid it? It is quite common
pattern to start some feature implementation with making a sketch and
running tests against it. I found it convenient to skip some style
requirements for such sketches (e.g. well formed javadocs).

[1] 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_BuildApacheIgnite

пн, 11 февр. 2019 г. в 11:38, Nikolay Izhikov :
>
> Petr, we should have 1 configuration for project, may be 1 configuration
> per programming language.
>
> пн, 11 февр. 2019 г., 11:33 Petr Ivanov mr.wei...@gmail.com:
>
> > I was asking about how many build configuration is intended? One for all
> > and multiple per module?
> >
> > With IDEA inspections it was going to be build configuration per module.
> >
> >
> >
> >
> > > On 11 Feb 2019, at 11:24, Nikolay Izhikov  wrote:
> > >
> > > Hello, Petr.
> > >
> > > Are you saying that we have not single build task? And each module builds
> > > when it required? If yes, then I propose to create a task like "Licence
> > > check" which will be run for every patch.
> > >
> > > My point is that violation of codestyle should be treated as hard as
> > > compile error.
> > >
> > > пн, 11 февр. 2019 г., 11:16 Petr Ivanov mr.wei...@gmail.com:
> > >
> > >> Is build configuration Inspections [Core] meant to transform into single
> > >> all-modules check build configuration (without module subdivision)?
> > >>
> > >>
> > >>> On 11 Feb 2019, at 11:02, Nikolay Izhikov  wrote:
> > >>>
> > >>> Hello, Maxim.
> > >>>
> > >>> +1 from me for migrating to checkstyle.
> > >>>
> > >>> Oleg, there is plugin for IDEA with 2mln downloads -
> > >>> https://plugins.jetbrains.com/plugin/1065-checkstyle-idea
> > >>>
> > >>> I propose do the following:
> > >>>
> > >>> 1. Migrate current checks to checkstyle.
> > >>> 2. Apply checks to all Ignite modules. Currently, only core module are
> > >>> checked.
> > >>> I will review and commit this patch, or do it by my own.
> > >>>
> > >>> 3. Include code style checks to "Build Apache Ignite" suite. Ignite has
> > >> to
> > >>> fail to build if patch violates codestyle.
> > >>>
> > >>> вс, 10 февр. 2019 г. в 07:54, Павлухин Иван :
> > >>>
> >  Hi,
> > 
> >  I also think that some warning from IDEA that some code style rule is
> >  violated is a must-have.
> > 
> >  вс, 10 февр. 2019 г. в 01:58, oignatenko :
> > >
> > > Hi Maxim,
> > >
> > > I believe that whatever style checks we establish at Teamcity, we
> > >> better
> > > take care of making it easy for developers to find and fix violations
> > >> in
> > > their typical dev environment (for Ignite this means, in IDEA). I
> > think
> >  it
> > > is important that developers can maintain required style with minimal
> >  effort
> > > on their side.
> > >
> > > If above is doable then I am 200% for migrating our Teamcity
> > >> inspections
> >  to
> > > checkstyle / maven.
> > >
> > > This is because I am very disappointed observing how it stays broken
> > >> for
> >  so
> > > long. And worst of all, even when (if) it is fixed, I feel we will
> >  always be
> > > at risk that it breaks again and that we will have to again wait for
> >  months
> > > for it to be fixed.
> > >
> > > This is such a stark contrast with my experience regarding checkstyle
> >  based
> > > inspections. These just work and you just never fear that it is going
> > >> to
> > > break for some obscure reason, this is so much better than what I
> > >> observe
> > > now.
> > >
> > > One suggestion in case if we pick checkstyle - I recommend keeping
> > its
> > > config file somewhere in the project under version control. I used to
> > > maintain such a shared style config at one of past jobs and after
> > some
> > > experimenting it turned out most convenient to have it this way - so
> > >> that
> > > developers could easily assess and discuss style settings and keep
> > >> track
> >  of
> > > changes in these. (note how Kafka folks from your link [5] appear to
> > be
> > > doing it this way)
> > >
> > > regards, Oleg
> > >
> > >
> > > Mmuzaf wrote
> > >> Igniters,
> > >>
> > >> I've found that some of the community members have faced with
> > >> `[Inspections] Core suite [1]` is not working well enough on TC. The
> > >> suite has a `FAILED` status for more than 2 months due to some
> > issues
> > >> in TeamCity application [2]. Current suite behaviour confuses not
> > only
> > >> new contributors but also other community members. Moreover, this
> > >> suite is no longer checks rules we previously configured. For
> > >> instance, in the master branch, I've found 11 `Unused imports` which
> > >> should have been caught earlier (e.g. for
> > >> {{IgniteCachePutAllRestartTest} 

[jira] [Created] (IGNITE-11300) MVCC: forbid using DataStreamer with allowOverwrite=true

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-11300:
---

 Summary: MVCC: forbid using DataStreamer with allowOverwrite=true
 Key: IGNITE-11300
 URL: https://issues.apache.org/jira/browse/IGNITE-11300
 Project: Ignite
  Issue Type: Task
  Components: mvcc
Affects Versions: 2.7
Reporter: Ivan Pavlukhin
 Fix For: 2.8


Calling {{IgniteDataStreamer.allowOverwrite(true)}} configures a streamer to 
use single-key cache put/remove operations for data modification. But 
put/remove operations on MVCC caches can be aborted due to write conflicts. So, 
some development effort is needed to support that mode properly. Let's throw 
exception in such case for MVCC caches.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11299) During SSL Handshake GridNio

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-11299:


 Summary: During SSL Handshake GridNio
 Key: IGNITE-11299
 URL: https://issues.apache.org/jira/browse/IGNITE-11299
 Project: Ignite
  Issue Type: Bug
Reporter: Ilya Kasnacheev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11298) TcpCommunicationSpi does not support TLSv1.3

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-11298:


 Summary: TcpCommunicationSpi does not support TLSv1.3
 Key: IGNITE-11298
 URL: https://issues.apache.org/jira/browse/IGNITE-11298
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.7
Reporter: Ilya Kasnacheev


When started on Java 11 we cannot form a secure cluster - Discovery will 
happily use the default TLSv1.2 but Communication will fail with its custom 
SSLEngine-using code.

Need to fix that.

Until that, nodes may be salvaged by setProtocol("TLSv1.2") on 
SslContextFactory, or by system property -Djdk.tls.client.protocols="TLSv1.2"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11297) Improving read of hot variables in WAL

[Anton Kalashnikov (JIRA)]

Anton Kalashnikov created IGNITE-11297:
--

 Summary: Improving read of hot variables in WAL
 Key: IGNITE-11297
 URL: https://issues.apache.org/jira/browse/IGNITE-11297
 Project: Ignite
  Issue Type: Improvement
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov


Looks like it is not neccessery mark some variables as volatile in 
FileWriteAheadLogManager because its initialized only one time on start but its 
have a lot of read of them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[MTCGA]: new failures in builds [2979488] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Recently contributed test failed in master 
internal.GridSpringBeanSerializationSelfTest 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8768893146029766837=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 14:56:51 12-02-2019 

[jira] [Created] (IGNITE-11296) 3rd-party persistence: Backup and primary partitions data differ after a single IgniteCache.get that loaded data from the persistent store which breaks skipStore behavi

[Igor Kamyshnikov (JIRA)]

Igor Kamyshnikov created IGNITE-11296:
-

 Summary: 3rd-party persistence: Backup and primary partitions data 
differ after a single IgniteCache.get that loaded data from the persistent 
store which breaks skipStore behavior
 Key: IGNITE-11296
 URL: https://issues.apache.org/jira/browse/IGNITE-11296
 Project: Ignite
  Issue Type: Bug
  Components: cache, cassandra
Affects Versions: 2.7, 2.5
Reporter: Igor Kamyshnikov


1) run 2 ignite servers on different machines
(this is important because of 
org.apache.ignite.internal.processors.cache.GridCacheContext#selectAffinityNodeBalanced
 - it takes into account MACs)

2) run cassandra and insert some records into cassandra

3) connect to the ignite cluster as Ignite client node and invoke
IgniteCache.get(pk);

4) execute IgniteCache.withSkipStore().get(pk) several times
The values returned will be randomly NULLs or non-NULLs.

5) depending on a chance, the data loaded in 3) can appear in primary partition 
or backup partition. If they are in backup partition, then they are not visible 
to Ignite JDBC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[Discussion] [ML] Future of Pipelines

[Alexey Zinoviev]

Hi, Igniters! 

Currently, we have in Ignite ML next Pipeline API in alpha version via
PipelineMDL and Pipeline classes.

I'm going to finish Pipeline API for the next release 2.8 and need you help
in brainstorming

You could find an example in master  here

  

The snippet is added here

 PipelineMdl mdl = new Pipeline()
.addFeatureExtractor(featureExtractor)
.addLabelExtractor(lbExtractor)
.addPreprocessor(new EncoderTrainer()
.withEncoderType(EncoderType.STRING_ENCODER)
.withEncodedFeature(1)
.withEncodedFeature(6))
.addPreprocessor(new ImputerTrainer())
.addPreprocessor(new MinMaxScalerTrainer())
.addPreprocessor(new NormalizationTrainer()
.withP(1))
.addTrainer(new DecisionTreeClassificationTrainer(5, 0))
.fit(ignite, dataCache);

If you have any experience with Pipeline APi in another ML framework as
scikit-learn, Spark, .NET ML and etc, please have a look at this API and
write any suggestion about possible features and use-cases

Feel free to add any suggestions or comments.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Binary clients: fallback to the previous versions of the protocol

[Igor Sapego]

Hi,

Well, this approach was used for quite some time in ODBC and JDBC,
so thin client has just inherited it, as we have the same port and the
same handshake message header. Maybe that's why you could not find
any mention of versioning in thin client context.

3. Let's start from this point as this is the root of the issue.
Preconditions
are simple - the backward compatibility. We should guarantee that user
can use any version of thin client with any version of server while they
share the same major version. This can be important, if someone have
user applications built with thin client and they don't want to break their
user applications, when they just want to update servers. What is
appropriate here is to give user some kind of warning that they use
outdated client or server.

Can you explain how your approach is going to work in the example
above? I'm not quite sure I understand you correctly.

1. The API for this can be as simple as a set of allowed version in
a configuration for both server and client. Maybe we should discuss such
kind of API as it can be useful for example in terms of security.

2. Complexity is the price developers have to pay for better user
experience.
Nothing new here. Software is not designed for developers, it's designed
for a user.

Best Regards,
Igor


On Tue, Feb 12, 2019 at 11:56 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Hello fellow igniters!
>
> I recently started reconnaissance before the
> implementation of IEP-23 [1] in Python thin client. I think it is an
> interesting and much promising feature. As I understand, the changes
> associated with this feature will result in a new version of the binary
> protocol.
>
> What bothers me is not the changes required by the
> implementation itself, but the upcoming changes in the way that the
> fallback to the previous versions of the protocol is going to be
> handled.
>
> I have not found a open discussion on this topic neither on
> this mailing list, nor in Jira. If such a discussion took place, please
> inform me with a link. Based on what I have heard so far, I made the
> following conclusions:
>
> 1) the client is now expected to handle multiple
> binary protocol versions;
> 2) since the protocol is implemented the way
> the client first reports its version to the server on handshake, and
> not the other way, the client now must try to connect to the server
> using its preferred version of the protocol, and in case of an error,
> reconnect with another version.
>
> These changes are raised some concerns
> for me, for example:
>
> 1) the version negotiation mechanism implemented on
> client can not be made transparent to the end user. We must give the
> user an option, so that they could choose from the entire set of
> versions supported by the client, a subset of the versions their
> application is designed to work with. That will complicate the user
> API;
>
> 2) the complexity of the client will also be increased, especially
> considering unit testing. Certain tests will require a certain version
> of Ignite server, so they can no longer be implemented environment-
> agnostically, leading to a weird mix on unit and integration levels;
>
> 3)
> I do not see, or not aware of, the preconditions to such changes. I
> think the number of potential applications that can benefit from multi-
> protocol client is quite low. Usually all the hassle of version
> matching lie on the deployment level, above the app level. Please prove
> me wrong on this subject.
>
> Instead of supporting multiple binary
> protocols inside one client package, I would strongly prefer to support
> each version of the protocol separately, in the corresponding branch of
> the package repository. It would still be possible for the end user to
> use multiple protocols in one app, but with certain code-level and
> deployment-level efforts combined.
>
> Again, I most probably lack awareness
> on the subject, and therefore my conclusions may be premature, but
> anyway, let us discuss. I will appreciate any bit of knowledge from the
> community.
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 23:+Best+Effort+Affinity+for+thin+clients
>
>

[jira] [Created] (IGNITE-11295) [ML] Add readme file to SparkModelParser module

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-11295:
-

 Summary: [ML] Add readme file to SparkModelParser module
 Key: IGNITE-11295
 URL: https://issues.apache.org/jira/browse/IGNITE-11295
 Project: Ignite
  Issue Type: Sub-task
  Components: ml
Affects Versions: 2.8
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev
 Fix For: 2.8


This file should contain examples of usage and instruction how to use this 
module



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11294) [ML] Use ML logger and env variables in Spark ML Parser

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-11294:
-

 Summary: [ML] Use ML logger and env variables in Spark ML Parser
 Key: IGNITE-11294
 URL: https://issues.apache.org/jira/browse/IGNITE-11294
 Project: Ignite
  Issue Type: Sub-task
  Components: ml
Affects Versions: 2.8
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev
 Fix For: 2.8


Add logger to SparkModelParser class and environment usage



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Binary clients: fallback to the previous versions of the protocol

[Dmitry Melnichuk]

Hello fellow igniters!

I recently started reconnaissance before the
implementation of IEP-23 [1] in Python thin client. I think it is an
interesting and much promising feature. As I understand, the changes
associated with this feature will result in a new version of the binary
protocol.

What bothers me is not the changes required by the
implementation itself, but the upcoming changes in the way that the
fallback to the previous versions of the protocol is going to be
handled.

I have not found a open discussion on this topic neither on
this mailing list, nor in Jira. If such a discussion took place, please
inform me with a link. Based on what I have heard so far, I made the
following conclusions:

1) the client is now expected to handle multiple
binary protocol versions;
2) since the protocol is implemented the way
the client first reports its version to the server on handshake, and
not the other way, the client now must try to connect to the server
using its preferred version of the protocol, and in case of an error,
reconnect with another version.

These changes are raised some concerns
for me, for example:

1) the version negotiation mechanism implemented on
client can not be made transparent to the end user. We must give the
user an option, so that they could choose from the entire set of
versions supported by the client, a subset of the versions their
application is designed to work with. That will complicate the user
API;

2) the complexity of the client will also be increased, especially
considering unit testing. Certain tests will require a certain version
of Ignite server, so they can no longer be implemented environment-
agnostically, leading to a weird mix on unit and integration levels;

3)
I do not see, or not aware of, the preconditions to such changes. I
think the number of potential applications that can benefit from multi-
protocol client is quite low. Usually all the hassle of version
matching lie on the deployment level, above the app level. Please prove
me wrong on this subject.

Instead of supporting multiple binary
protocols inside one client package, I would strongly prefer to support
each version of the protocol separately, in the corresponding branch of
the package repository. It would still be possible for the end user to
use multiple protocols in one app, but with certain code-level and
deployment-level efforts combined.

Again, I most probably lack awareness
on the subject, and therefore my conclusions may be premature, but
anyway, let us discuss. I will appreciate any bit of knowledge from the
community.

[1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-
23:+Best+Effort+Affinity+for+thin+clients