Re: [DISCUSSION] IEP-119 Binary infrastructure modularization

[Николай Ижиков]

> Looks like current available counter is 119.

Renamed - 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-119+Binary+infrastructure+modularization

> As I understand, all the answers are NO, and this is only a refactoring - 
> correct?

Yes. We must not change anything from the user perspective in the scope of this 
IEP.


> 8 февр. 2024 г., в 21:18, Pavel Tupitsyn  написал(а):
> 
> Could you please clarify:
> - Does this proposal affect the user in any way?
> - Does it introduce or remove functionality?
> - Does it affect compatibility?
> 
> As I understand, all the answers are NO, and this is only a refactoring -
> correct? Let's make this clear in the IEP
> 
> P.S. Separate IEP naming for Ignite 3 is ok, but we have not switched yet,
> and we should not have two proposals with the same ID. Please fix this.
> 
> On Thu, Feb 8, 2024 at 7:17 PM Anton Vinogradov  wrote:
> 
>> Mikhail, as we discussed earlier, Ignite-3 should use it's own counters
>> since it's a different project.
>> Better case for me is to use I3EP-XX naming.
>> 
>> Nikolay, +1 to your proposal.
>> 
>> On Thu, Feb 8, 2024 at 6:57 PM Mikhail Pochatkin 
>> wrote:
>> 
>>> Hello, Nikolay.
>>> 
>>> Thanks for your IEP. I would say that IEP-115 already exists in Apache
>>> Ignite 3 section. Looks like current available counter is 119.
>>> 
>>> 
>>> чт, 8 февр. 2024 г., 18:27 Николай Ижиков :
>>> 
>>>> Hello, Igniters.
>>>> 
>>>> I want to discuss IEP-115 [1] Binary infrastructure modularization
>>>> Two main goal of IEP:
>>>> 
>>>> - remove coupling between specific marshaller(BinaryMarshaller) and
>>> Ignite
>>>> core code.
>>>> - simplify Binary code improvements.
>>>> - clear SerDes abstraction in core code.
>>>> 
>>>> As a result of this IEP we will have an ability to decouple Ignite thin
>>>> client from ignite-core module.
>>>> 
>>>> [1]
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-115+Binary+infrastructure+modularization
>>> 
>> 

[DISCUSSION] IEP-115 Binary infrastructure modularization

[Николай Ижиков]

Hello, Igniters.

I want to discuss IEP-115 [1] Binary infrastructure modularization
Two main goal of IEP:

- remove coupling between specific marshaller(BinaryMarshaller) and Ignite core 
code.
- simplify Binary code improvements.
- clear SerDes abstraction in core code.

As a result of this IEP we will have an ability to decouple Ignite thin client 
from ignite-core module.

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-115+Binary+infrastructure+modularization

Re: [VOTE] Release Apache Ignite 2.16.0 RC0

[Николай Ижиков]

+1 (binding)

Отправлено с iPhone

> 24 дек. 2023 г., в 00:13, Ivan Daschinsky  написал(а):
> 
> +1 from me (binding)
> 
> 1. Checked checksums
> 2. Checked C++/ODBC driver: built from source and ran examples on Ubuntu
> 22.04/gcc 11.4.0/OpenJDK-Temurin 17.0.8
> 3. Checked ODBC driver with pyodbc (python 3.11 + pyodbc 4.0.35). Checked
> on calcite and H2 engines. (Ubuntu 22.04)
> 
> вт, 19 дек. 2023 г. в 13:54, Zhenya Stanilovsky > :
> 
>> 
>> Finally i build it (some local maven problems).
>> Check compilation, run calcite tests.
>> +1 (non-binding)
>> 
>>> 
 
> Hello, Zhenya.
> 
> Java profiles should be activated automatically (on reload maven
> project). Which JDK are you using? Is the project built from maven?
> 
> I can't reproduce on HotSpot 1.8.0_201-b09, OpenJDK 21+35-2513.
> 
> вт, 19 дек. 2023г. в 11:16, Zhenya Stanilovsky <
>> arzamas...@mail.ru.invalid >:
>> 
>> 
>> Hi, probably it`s a kind of problem from my side, but cloning by tag
>> and further steps:
>> * change profile to java-8
>> * Reload all maven Projects
>> * Try to run: ScriptTestSuite will raise guava dependency problem:
>> 
>> /ignite/internal/processors/query/calcite/sql/IgniteSqlRollback.java:20:33
>> java: package com.google.common.collect does not exist
>> ---
>> * change profile to java-9
>> * reload
>> * the same problem :
>> 
>> modules/core/src/test/java/org/apache/ignite/testframework/GridTestUtils.java:79:33
>> java: package com.google.common.collect does not exist
>> 
>> probably i need to clean all and try to rebuild from scratch .. for
>> now it`s not clear for me.
>> 
>>> +1
>>> 
>>> - Started the node from binaries
>>> - Tested .NET examples
>>> - Tested NuGet packages
>>> 
>>> On Sun, Dec 17, 2023 at 2:48PM Nikita Amelchev <
>> namelc...@apache.org >
>>> wrote:
>>> 
 Dear Community,
 
 The release candidate is ready.
 
 I have uploaded release candidate to
 https://dist.apache.org/repos/dist/dev/ignite/2.16.0-rc0
 https://dist.apache.org/repos/dist/dev/ignite/packages_2.16.0-rc0
 
 The following staging can be used for testing:
 
>> https://repository.apache.org/content/repositories/orgapacheignite-1559
 
 Tag name is 2.16.0-rc0:
 https://github.com/apache/ignite/releases/tag/2.16.0-rc0
 
 2.16.0 most important changes:
 * Cache dumps;
 * Calcite engine: added hints, became more stable, covered with
 diagnostic tools;
 * Fixes to support JDK 14-21;
 * Cluster Management API (IEP-81);
 * Java thin client: Service Awareness feature;
 etc.
 
 RELEASE NOTES:
 
 
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.16
 
 Complete list of resolved issues:
 
 
>> https://issues.apache.org/jira/browse/IGNITE-19650?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.16%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority
 
 DEVNOTES
 
 
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.16
 
 The vote is formal, see voting guidelines
 https://www.apache.org/foundation/voting.html
 
 +1 - to accept Apache Ignite 2.16.0-rc0
 0 - don't care either way
 -1 - DO NOT accept Apache Ignite 2.16.0-rc0 (explain why)
 
 See notes on how to verify release here
 https://www.apache.org/info/verification.html
 and
 
 
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
 
 This vote will be open till Sun December 24, 2023, 15:00 UTC.
 
 
>> https://www.timeanddate.com/countdown/vote?iso=20231224T15=0=VOTE+on+the+Apache+Ignite+Release+2.16.0+RC0=sanserif
 
 --
 Best wishes,
 Amelchev Nikita
 
>> 
>> 
>> 
>> 
> 
> 
> --
> Best wishes,
> Amelchev Nikita
 
 
 
 

Re: [PROPOSAL] Move Ignite 2.x to JDK11 compilation

[Николай Ижиков]

+1

> 24 окт. 2023 г., в 19:25, Anton Vinogradov  написал(а):
> 
> ++1 for switch to the OpenJDK framework
> 
> As to original question, I propose to (step-by-step):
> 1) Switch to java 11 testing instead of 8
> 2) Fix java 11 release
> 3) Switch to maven.compiler.source 11 (Vote required)
> 
> On Tue, Oct 24, 2023 at 7:12 PM Dmitriy Pavlov  wrote:
> 
>> ++1 for testing using JDK11
>> 
>> +0 for switching off support of JDK8 runtimes and class version (unless
>> compilation using JDK11 is done with -target=1.8 our users won't be able to
>> run Ignite using Java 8).
>> 
>> I suggest the following next steps: once tests are completely fine on
>> Teamcity using JDK 11, we start a separate thread named like:
>> "[VOTE] discontinuing JDK 8 since the 2.16 release"
>> to make the issue for end users more obvious. Since Nikita volunteers to be
>> a Release Manager, I suppose that 2.16 will be released quite soon.
>> 
>> I personally prefer to switch to newer Java. Using newer runtime helps us
>> to provide speed and scale, since JVM is developed all the time. But I
>> might be unaware of the risks, so let's discuss it later with the entire
>> dev community.
>> 
>> Sincerely,
>> Dmitriy Pavlov
>> 
>> ср, 30 авг. 2023 г. в 19:09, Ivan Daschinsky :
>> 
>>> +1, sounds reasonable for the open source project. IMHO, Eclipse Temurin
>>> JDK is a rule of thumb choice nowadays.
>>> 
>>> ср, 30 авг. 2023 г. в 18:40, Peter Ivanov :
>>> 
 Hi, Igniters!
 
 As part of this proposal I would also like to discuss the JDK vendors
>> we
 are using on our CI platforms TeamCIty.
 Historically, Ignite 2.x is being built under Oracle's edition - mostly
 because of some features like JFR or similar.
 However today we have pretty much solid alternative represented by
>>> OpenJDK
 and its most popular build Eclipse Temurin, which provides every
>> required
 version for our needs (and especially LTS versions 1.8, 11 and 17) with
>>> all
 necessary updates.
 
 So, I suggest we update our build agents for TeamCity accordingly,
>>> provide
 OpenJDK framework for those 3 major versions and discontinue Oracle's
 builds.
 WDYT?
 
 пт, 25 авг. 2023 г. в 12:13, Ivan Daschinsky :
 
> +1. It's time to do it at last.
> 
> пт, 25 авг. 2023 г. в 12:06, Peter Ivanov :
> 
>> Mostly, yes.
>> 
>> In other words - proposal is about starting shipping Apache Ignite
> releases
>> with JDK11 compiled binaries thus dropping JDK8-10 runtime support.
>> 
>> пт, 25 авг. 2023 г. в 12:03, Anton Vinogradov :
>> 
>>> I looks like you're proposing not migration to 2.11 (because it
>> is
>> already
>>> supported), but Java 8-9 support dropping.
>>> 
>>> On Fri, Aug 25, 2023 at 11:54 AM Peter Ivanov <
>> vvei...@apache.org>
>> wrote:
>>> 
 Hi, Igniters!
 
 
 I would like to propose the next, if you don't mind me saying,
 revolutionary step forward in our project: moving Ignite 2.x
>> compilation
>>> to
 JDK11 minimum version.
 I'd rather not make arguments, pros and cons other that
>> currently
> exist
>>> in
 Java world - you know them better than me. Let's just say that
>> it
> seems
 that time has come - consider at least that JDK11 as the LTS
 version
> is
 already about 4 and a half years on the go, and Ignite 3.x
>>> started
> from
 JDK11 right away.
 
 This change may possibly require from us additional efforts on
>> supporting
 the last version with JDK8 in terms of releasing additional
>>> patches
> and
 hotfixes a bit longer than usual. However, this is up to the
> community
>> to
 decide.
 
 Currently, Apache Ignite 2.x (with Extensions as well) is
>> already
>>> prepared
 for being compiled with JDK11 and almost all tests are passing.
>>> If
 we
>>> come
 to an agreement about this proposal and designate the next
>>> version
> that
 will become the first to provide JDK11 compiled binaries - I am
 ready
>> to
 start the process of updating the TeamCity building project
>> accordingly.
 
 
 Please, share your thoughts.
 
>>> 
>> 
> 
> 
> --
> Sincerely yours, Ivan Daschinskiy
> 
 
>>> 
>>> 
>>> --
>>> Sincerely yours, Ivan Daschinskiy
>>> 
>> 

Re: Removal of ignite ml module (or moving it to extensions)

[Николай Ижиков]

A few cents to let you know how abandoned ML module is.

1. Last valuable commit December 9, 2020 - 
https://github.com/apache/ignite/commit/04f6a33851d9f7bd269a09fdc2c74485b1e01a8a

2. Dependencies and current versions of them:

  * com.dropbox.core:dropbox-core-sdk:2.1.1 current version is - 5.4.5 [1]
  * com.github.fommil.netlib:core:1.1.2 - not developed and archived since 
2017. Last version released in 2013 [2] 
  * org.apache.commons:commons-rng-core:1.0 current version is 1.5 [3]
  * com.zaxxer:SparseBitSet:1.0 current version is 1.2 [4]
  * ai.catboost:catboost-prediction:0.24 current version is 1.2 [5] 
  * ai.h2o:h2o-genmodel:3.26.0.8 current version is 3.42.0.2 [6]

ML community make a huge step forward since 2020.
So I doubt ML features and tools integrations works as expected nowadays.
Those type of Ignite features(abandoned or supported partially) has to be in 
extensions.

[1] https://github.com/dropbox/dropbox-sdk-java
[2] https://github.com/fommil/netlib-java
[3] https://commons.apache.org/proper/commons-rng/commons-rng-core/
[4] https://github.com/brettwooldridge/SparseBitSet
[5] https://mvnrepository.com/artifact/ai.catboost/catboost-prediction
[6] https://mvnrepository.com/artifact/ai.h2o/h2o-genmodel


  
> 11 авг. 2023 г., в 12:19, Kseniya Romanova  написал(а):
> 
> As far as I know, the integration was removed from the Tensorflow side.
> 
> On Thu, Aug 10, 2023 at 2:04 PM Andrey Mashenkov 
> wrote:
> 
>> Ivan,
>> 
>>> Actually, I haven't found any integration with tensorflow in AI code.
>> 
>> Ok. You are right.
>> Tensorflow is mentioned in docs: docs/_docs/setup.adoc.
>> 
>> Adapters may require compilation time dependencies, but these dependencies
>> shouldn't be part or release package,
>> regardless whether the ML module is a part of Ignite or extensions. WDYT?
>> 
>> On Thu, Aug 10, 2023 at 1:36 PM Ivan Daschinsky 
>> wrote:
>> 
>>> Actually, I haven't found any integration with tensorflow in AI code.
>>> Actually, all integrations are some adapters that allow to load
>> pretrained
>>> models (h2o, catboost etc.)
>>> 
>>> чт, 10 авг. 2023 г. в 13:08, Ivan Daschinsky :
>>> 
 I am personally for moving to extensions. Alex has already mentioned
>> all
 the reasons why it should be done and all of them are quite important.
 The module seems to be quite independent and there is no problem to
>> move
 it to ignite-extensions.
 So I am +1 for moving to ignite-extensions.
 
 
 чт, 10 авг. 2023 г. в 12:45, Kseniya Romanova :
 
>> 
>> do you know anyone who uses it?
> 
> I know some teams, who do. At the last Ignite Summit we had a talk
> featuring Ml module (from the Groovy community).
> Anyway, We need here the module maintainer opinion
>  + Alex
> 
> On Wed, Aug 9, 2023 at 3:38 PM Andrey Mashenkov <
> andrey.mashen...@gmail.com>
> wrote:
> 
>> -1 for removal.
>> 0 for relocation
>> 
>> imho, TC resources and module size aren't good arguments for
>> removal/moving.
>> ML tests could be run nightly.
>> ML module contains few integrations (with TensorFlow and other),
>> these
>> optional integrations are wighty and could be moved to extension,
>> but core functionality still can be left untouched if it is highly
> coupled
>> with core Ignite and moving to extension is hard.
>> 
>> 
>> On Wed, Aug 9, 2023 at 3:22 PM Anton Vinogradov 
>>> wrote:
>> 
>>> +1 to relocation
>>> 
>>> On Wed, Aug 9, 2023 at 3:09 PM Alex Plehanov <
>>> plehanov.a...@gmail.com
>> 
>>> wrote:
>>> 
 Pavel, do you know anyone who uses it?
 
 Looks like it isn't used at all (no questions on mail lists, no
 tickets), but we spend developers time to build module with
>> every
 Ignite rebuild, we spend users traffic to download module (ML
>> with
 dependencies takes about 1/4 of our binary release package size)
>>> and
 we spend team-city resources to test module.
 
 +1 for removing or moving to extensions.
 
 ср, 9 авг. 2023 г. в 14:19, Pavel Tupitsyn <
>> ptupit...@apache.org
 :
> 
> Does it have any outstanding issues? A stable and useful
>> module
>> should
 not
> be removed just because it does not evolve.
> 
> On Wed, Aug 9, 2023 at 1:46 PM Ivan Daschinsky <
> ivanda...@apache.org
>>> 
 wrote:
> 
>> Igniters, seems that this module is completely abandoned for
> more
>>> than
 2
>> years. But it is enormous and it seems that nobody wants to
>>> take
>> care
 of
>> it. I suggest just removing it or moving it to extensions
>> (as
>>> option).
>> WDYT?
>> 
 
>>> 
>> 
>> 
>> --
>> Best regards,
>> Andrey V. Mashenkov
>> 
> 
 
 
 --
 Sincerely yours, Ivan 

Re: Removal of ignite ml module (or moving it to extensions)

[Николай Ижиков]

Hello, Pavel.

I think the ignite-extension project is right place for ignite-ml.
Useful module that has its own release cycle and don’t contains core Ignite 
features has to be inside extensions.

> 9 авг. 2023 г., в 14:18, Pavel Tupitsyn  написал(а):
> 
> Does it have any outstanding issues? A stable and useful module should not
> be removed just because it does not evolve.
> 
> On Wed, Aug 9, 2023 at 1:46 PM Ivan Daschinsky  wrote:
> 
>> Igniters, seems that this module is completely abandoned for more than 2
>> years. But it is enormous and it seems that nobody wants to take care of
>> it. I suggest just removing it or moving it to extensions (as option).
>> WDYT?
>> 

[DISCUSSION] IEP-109 Cluster in-memory caches snapshots

[Николай Ижиков]

Hello, Igniters.

I’m start working on IEP-109 [1]
Snapshots of in-memory caches.

Your feedback regarding IEP are welcome.



[1] 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-109+Cluster+in-memory+caches+snapshots

Re: [DISCUSSION] Removing IgniteCacheSnapshotManager

[Николай Ижиков]

Ticket created - https://issues.apache.org/jira/browse/IGNITE-19915


> 5 июля 2023 г., в 12:23, Вячеслав Коптилин  
> написал(а):
> 
> Hello,
> 
> I don't have any objections. Let's clean up the code.
> 
> Thanks,
> Slava.
> 
> вт, 4 июл. 2023 г. в 19:47, Николай Ижиков :
> 
>> Hello, Igniters.
>> 
>> IgniteCacheSnapshotManager is internal component that was deprecated more
>> then 3 years for now.
>> I have plans to remove it from codebase.
>> Is there any objections?

[DISCUSSION] Removing IgniteCacheSnapshotManager

[Николай Ижиков]

Hello, Igniters.

IgniteCacheSnapshotManager is internal component that was deprecated more then 
3 years for now.
I have plans to remove it from codebase.
Is there any objections?

Re: [DISCUSSION] [IEP-81] New Cluster Management API (Ignite 2.x)

[Николай Ижиков]

Merged to master.

> 6 июня 2023 г., в 16:19, Nikolay Izhikov  написал(а):
> 
> Hello, Igniters.
> 
> Last couple of weeks we actively worked on Phase 1 of IEP-81.
> Now, after several dozens of feature branch reviews Phase 1 is almost ready.
> 
> If you are interested in new Management API, please, join to the reivew.
> 
> PR - https://github.com/apache/ignite/pull/10675
> IEP - 
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-81+Cluster+Management+API
> 
> 
> 
> On 2021/11/26 13:54:51 Andrey Mashenkov wrote:
>> Hi Maxim,
>> 
>> I like the idea, the IEP looks very useful.
>> 
>> However, I agree with Nikolay on this
>> 
>>> 
>>> Don’t think we should rely on auto scan class path capabilities.
>>>1. How this automatic scanner will distinguish ignite class from
>>> user class if they both in node class path and same package like
>>> «org.apache.ignite.internal»?
>>>2. It seems hard to debug any errors with auto class path scanner.
>>> How we ensure correct order of scanning in case different jar order in
>>> class path?
>> 
>> I think we should go with some kind of  hardcoded «on startup» command
>>> registration.
>> 
>> 
>> Maybe we could use Java ServiceLoader [1] for this?
>> 
>> You can add a list of command classes to e.g.
>> META-INF/services/o.a.i.CommandHandlerProvider,
>> then register all mentioned classes that are found via ServiceLoader.load(
>> CommandHandlerProvider.class)
>> Service loader can return the iterator for all found providers.
>> 
>> This will prevent scanning all the classes in classpath and allow to load
>> classes from 3-rd party extensions.
>> 
>> [1] https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
>> 
>> 
>> On Fri, Nov 26, 2021 at 12:25 PM Nikolay Izhikov 
>> wrote:
>> 
>>> Hello, Maxim.
>>> 
>>> Thanks for the IEP.
>>> I support the intentions and design of the IEP in general but have some
>>> comments:
>>> 
 AnnotationCommandScanner, PackageCommandScanner, URICommandScanner
>>> 
>>> Don’t think we should rely on auto scan class path capabilities.
>>>1. How this automatic scanner will distinguish ignite class from
>>> user class if they both in node class path and same package like
>>> «org.apache.ignite.internal»?
>>>2. It seems hard to debug any errors with auto class path scanner.
>>> How we ensure correct order of scanning in case different jar order in
>>> class path?
>>> 
>>> I think we should go with some kind of  hardcoded «on startup» command
>>> registration.
>>> 
>>> 
 execute(ProxyManagementTask.class, Map atts)
>>> 
>>> 1. Do we have some well-defined place to validate `atts`?
>>> 
>>> 2. Do we really need to rely on `java.lang.Class` parameter?
>>>This means executor (thin client) should have all class
>>> definitions which can be hard for plugins provided class.
>>>Can we rely on some kind of «path array» like `String[] {
>>> «baseline», «set» }` or `String[] { «user», «add» }`
>>>So the API usage will looks like:
>>> 
>>> ```
>>>String execute(String[] path, Map params)
>>> 
>>>Map params = new HashMap();
>>> 
>>>params.put(«—login», «admin»)
>>>params.put(«—password», «mypassw0rd»)
>>> 
>>>String res = execute(new String[] {«user», «add»}, params);
>>> ```
>>> 
>>> 
 Design issues …   it is not possible to add and run new management
>>> tasks at the cluster runtime (e.g. REPL mode for CLI tools can't be used);
>>> 
>>> It seems to me as a good thing?
>>> We shouldn’t have ability to add management tasks in runtime from the user
>>> side because of security reasons.
>>> 
>>> But, we should be able to run any existing task dynamically in some
>>> scripting environment - REPL, CLI as you mentioned.
>>> 
>>> 
 Features
>>> 
>>> I think we should focus on:
>>> 
>>> 1. Subsystem then stores all available management tasks
>>> 2. Unified execution flow that guarantees all required things like
>>> authorization, authentication, logging and event generation.
>>> 3. Creation of the specific protocol implementation that will expose all
>>> existing commands **in the same way** through all supported protocols
>>> **without any additional development**.
>>> 4. Simplification of new command development and improvement of existing.
>>> 
>>> 
 Compatibility
>>> 
>>> IEP doesn’t clearly define compatibility.
>>> Do we keep all existing commands as is?
>>> Is new subsystem a replacement for current API?
>>> 
>>> Is new subsystem will co-exist with current API? For how long?
>>> 
 26 нояб. 2021 г., в 11:44, Maxim Muzafarov 
>>> написал(а):
 
 Igniters,
 
 
 I'd like to propose a new internal cluster management API that will
 simplify new cluster management commands creation and expose new
 commands to CLI, REST, JMX Ignite interfaces without any additional
 actions from the engineer adding a new command. This main goal of the
 IEP is supposed to be available after the implementation of 

Re: Deprecated GridSslContextFactory removal

[Николай Ижиков]

+1

> 24 апр. 2023 г., в 14:56, Taras Ledkov  написал(а):
> 
> +1 for remove

Re: Setting up the project locally.

[Николай Ижиков]

https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines


> 27 марта 2023 г., в 20:06, Maksim Timonin  
> написал(а):
> 
> Hi Gael!
> 
> There are such docs:  DEVNOTES.txt, CONTRIBUTING.md. Please check them in
> the root directory (nearby README.md).
> 
> Maksim.
> 
> On Mon, Mar 27, 2023 at 8:00 PM Gael Yimen Yimga 
> wrote:
> 
>> Hello Ilya,
>> 
>> There is no documentation that explains how to set up the project on a
>> local machine ? If it is the case, it might be a good idea to create a
>> ticket to have that address so that it will be easier for a newbie to
>> onboard in the project. How does that sound ?
>> 
>> Gael.
>> --
>> 
>> 
>> On Mon, 27 Mar 2023 at 09:02, Ilya Kasnacheev 
>> wrote:
>> 
>>> Hello!
>>> 
>>> You may just import it into IntelliJ IDEA or run mvn clean install
>>> -DskipTests.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> пн, 27 мар. 2023 г. в 18:49, Gael Yimen Yimga :
>>> 
 Hello,
 
 Please I'm looking for the documentation to set up the project on my
>>> local
 machine so that I can start the development. I thought I would be able
>> to
 find it in the README.md , but not.
 Please correct me if I'm wrong.
 
 Thanks for your help.
 Gael.
 --
 
>>> 
>> 

Re: Enable master branch protection (disallow force push)

[Николай Ижиков]

Hello, Pavel.

Why do we need any new restrictions?

> 27 марта 2023 г., в 12:32, Pavel Tupitsyn  написал(а):
> 
> Mirza, yes, I've created a ticket for ignite-3 as well:
> 
> https://issues.apache.org/jira/browse/IGNITE-19122
> 
> 
> On Mon, Mar 27, 2023 at 12:27 PM Mirza Aliev  wrote:
> 
>> Thanks! Nice idea!
>> 
>> I wonder if this changes work for ignite-3 repo as well?
>> 
>> пн, 27 мар. 2023 г. в 12:47, Pavel Tupitsyn :
>> 
>>> Igniters,
>>> 
>>> I'm going to enable master branch protection:
>>> - Disallow force push (dangerous)
>>> - Disallow merge commits (against our workflow)
>>> 
>>> Any objections?
>>> 
>>> [1] https://github.com/apache/ignite/pull/10608
>>> [2] https://issues.apache.org/jira/browse/IGNITE-19123
>>> 
>> 

Re: [VOTE] Release bug fix release pyignite-0.6.1-rc0

[Николай Ижиков]

+1 (binding)

> 15 февр. 2023 г., в 11:21, Ivan Daschinsky  написал(а):
> 
> Dear Igniters!
> 
> This is a patch release that contains an important fix for users of
> pyignite
> 
> https://issues.apache.org/jira/browse/IGNITE-18788
> 
> 
> Release candidate binaries for subj are uploaded and ready for vote
> You can find them here:
> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.6.1-rc0
> 
> If you follow the link above, you will find source packages (*.tar.gz and
> *.zip)
> and binary packages (wheels) for windows (amd64), linux (x86_64) amd mac os
> (x86_64)
> for pythons 37, 38, 39, 310 and 311. Also, there are sha512 and gpg
> signatures.
> Code signing keys can be found here -- https://downloads.apache.org/ignite
> /KEYS
> Here you can find instructions how to verify packages
> https://www.apache.org/info/verification.html
> 
> You can install binary package for specific version of python using pip
> For example do this on linux for python 3.8
>>> pip install
> pyignite-0.6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> 
> You can build and install package from source using this command:
>>> pip install pyignite-0.6.1.zip
> You can build wheel on your platform using this command:
>>> pip wheel --no-deps pyignite-0.6.1.zip
> 
> For building C module, you should have python headers and C compiler
> installed.
> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> In Mac OS X xcode-tools and python from homebrew are the best option.
> 
> In order to check whether C module works, use following:
>>> from pyignite import _cutils
>>> print(_cutils.hashcode('test'))
>>> 3556498
> 
> You can find documentation here:
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.6.1.rc0/
> 
> You can find examples here (to check them, you should start ignite locally):
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.6.1.rc0/examples.html
> Also, examples can be found in source archive in examples subfolder.
> docker-compose.yml is supplied in order to start ignite quickly. (Use
> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> down` to shut down it)
> 
> Release notes:
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=86448e9ce51d7223ac49cf4f95da70d3d365e8c1;hb=0d86f44e86270f4d578afbce41aa2d6c424d2615
> 
> Git release tag was created:
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=b0ce094d7a2db3fb07471be7b37ff9edab4180a8
> 
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
> 
> +1 - to accept pyignite-0.6.1-rc0
> 0 - don't care either way
> -1 - DO NOT accept pyignite-0.6.1-rc0
> 
> The vote finishes at 02/17/2021 15:00 UTC
> 
> -- 
> Sincerely yours, Ivan Daschinskiy

Re: Shutdown policy refactoring

[Николай Ижиков]

So why it’s default behavior?
And why we want keep it?

> 18 окт. 2022 г., в 12:27, Aleksandr Polovtsev  
> написал(а):
> 
> But this is the default behaviour right now, do you mean we should change
> it? I'm ok with that if it will not break something
> 
> On Tue, Oct 18, 2022 at 12:13 PM Николай Ижиков  wrote:
> 
>>> I don't know what most people use in the real world, maybe more
>> experienced guys can clarify that.
>> 
>> My question is different.
>> 
>> Why we may want to provide cancel=true?
>> Is there any reason to support this in Ignite?
>> 
>>> 18 окт. 2022 г., в 11:47, Aleksandr Polovtsev 
>> написал(а):
>>> 
>>> Thank you for the response, Nikolay. As I can see, this is the default
>>> behaviour now (ShutdownPolicy = IMMEDIATE, cancel = true). I don't know
>>> what most people use in the real world, maybe more experienced guys can
>>> clarify that.
>>> 
>>> On Tue, Oct 18, 2022 at 11:15 AM Николай Ижиков 
>> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> Why do we need «immediate»?
>>>> Is there real-world scenarios for it?
>>>> 
>>>> Can we do graceful or «save all possible data» shutdown always?
>>>> 
>>>>> 18 окт. 2022 г., в 10:59, Aleksandr Polovtsev >> 
>>>> написал(а):
>>>>> 
>>>>> Hello, dear Igniters!
>>>>> 
>>>>> I'd like to propose a refactoring of the current Shutdown Policy
>>>> mechanism.
>>>>> Currently, node shutdown is controlled by a bunch of flags and enums (I
>>>> may
>>>>> be missing some of them):
>>>>> 
>>>>> 1. ShutdownPolicy enum has only two entries and is basically a flag
>> that
>>>>> dictates if we need to wait for the data to be backed up, so that we
>> will
>>>>> not lose any data when the node goes offline.
>>>>> 2. "cancel" flag is passed to the stop method and indicates that all
>> long
>>>>> running jobs should be interrupted. For example, it prevents the node
>>>>> performing a checkpoint on shutdown. I'm pretty sure this flag is
>> nearly
>>>>> always set to "true", because it is used by the Shutdown Hook.
>>>>> 3. There are also a bunch of system properties that affect the
>> shutdown:
>>>>>  * IGNITE_PDS_SKIP_CHECKPOINT_ON_NODE_STOP - disables checkpoints on
>>>>> node stop, which heavily correlates with the "cancel" flag.
>>>>>  * IGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN - deprecated and duplicates the
>>>>> ShutdownPolicy enum
>>>>>  * IGNITE_NO_SHUTDOWN_HOOK - disables the shutdown hook. I don't know
>>>> if
>>>>> it is really useful.
>>>>> 
>>>>> I think that this approach is counter intuitive and would like to
>>>>> incorporate all aforementioned flags into the ShutdownPolicy enum. For
>>>>> example, it can look as follows:
>>>>> 
>>>>> public enum ShutdownPolicy {
>>>>>  IMMEDIATE // Stops the node and cancels all running jobs
>>>>>  GRACEFUL // Stops the node, does not cancel running jobs and waits
>> for
>>>>> backups
>>>>>  GRACEFUL_NO_BACKUPS // Stops the node, does not cancel running jobs,
>>>>> does not wait for backups
>>>>> }
>>>>> 
>>>>> After that, the "cancel" flag and all corresponding properties can be
>>>>> removed (apart from the IGNITE_NO_SHUTDOWN_HOOK, if it is still
>> needed).
>>>>> 
>>>>> Does this make sense? What do you think?
>>>>> 
>>>>> --
>>>>> With regards,
>>>>> Aleksandr Polovtsev
>>>> 
>>>> 
>>> 
>>> --
>>> With regards,
>>> Aleksandr Polovtsev
>> 
>> 
> 
> -- 
> With regards,
> Aleksandr Polovtsev

Re: Shutdown policy refactoring

[Николай Ижиков]

> I don't know what most people use in the real world, maybe more experienced 
> guys can clarify that.

My question is different.

Why we may want to provide cancel=true?
Is there any reason to support this in Ignite?

> 18 окт. 2022 г., в 11:47, Aleksandr Polovtsev  
> написал(а):
> 
> Thank you for the response, Nikolay. As I can see, this is the default
> behaviour now (ShutdownPolicy = IMMEDIATE, cancel = true). I don't know
> what most people use in the real world, maybe more experienced guys can
> clarify that.
> 
> On Tue, Oct 18, 2022 at 11:15 AM Николай Ижиков  wrote:
> 
>> Hello,
>> 
>> Why do we need «immediate»?
>> Is there real-world scenarios for it?
>> 
>> Can we do graceful or «save all possible data» shutdown always?
>> 
>>> 18 окт. 2022 г., в 10:59, Aleksandr Polovtsev 
>> написал(а):
>>> 
>>> Hello, dear Igniters!
>>> 
>>> I'd like to propose a refactoring of the current Shutdown Policy
>> mechanism.
>>> Currently, node shutdown is controlled by a bunch of flags and enums (I
>> may
>>> be missing some of them):
>>> 
>>> 1. ShutdownPolicy enum has only two entries and is basically a flag that
>>> dictates if we need to wait for the data to be backed up, so that we will
>>> not lose any data when the node goes offline.
>>> 2. "cancel" flag is passed to the stop method and indicates that all long
>>> running jobs should be interrupted. For example, it prevents the node
>>> performing a checkpoint on shutdown. I'm pretty sure this flag is nearly
>>> always set to "true", because it is used by the Shutdown Hook.
>>> 3. There are also a bunch of system properties that affect the shutdown:
>>>   * IGNITE_PDS_SKIP_CHECKPOINT_ON_NODE_STOP - disables checkpoints on
>>> node stop, which heavily correlates with the "cancel" flag.
>>>   * IGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN - deprecated and duplicates the
>>> ShutdownPolicy enum
>>>   * IGNITE_NO_SHUTDOWN_HOOK - disables the shutdown hook. I don't know
>> if
>>> it is really useful.
>>> 
>>> I think that this approach is counter intuitive and would like to
>>> incorporate all aforementioned flags into the ShutdownPolicy enum. For
>>> example, it can look as follows:
>>> 
>>> public enum ShutdownPolicy {
>>>   IMMEDIATE // Stops the node and cancels all running jobs
>>>   GRACEFUL // Stops the node, does not cancel running jobs and waits for
>>> backups
>>>   GRACEFUL_NO_BACKUPS // Stops the node, does not cancel running jobs,
>>> does not wait for backups
>>> }
>>> 
>>> After that, the "cancel" flag and all corresponding properties can be
>>> removed (apart from the IGNITE_NO_SHUTDOWN_HOOK, if it is still needed).
>>> 
>>> Does this make sense? What do you think?
>>> 
>>> --
>>> With regards,
>>> Aleksandr Polovtsev
>> 
>> 
> 
> -- 
> With regards,
> Aleksandr Polovtsev

Re: Shutdown policy refactoring

[Николай Ижиков]

Hello, 

Why do we need «immediate»?
Is there real-world scenarios for it?

Can we do graceful or «save all possible data» shutdown always? 

> 18 окт. 2022 г., в 10:59, Aleksandr Polovtsev  
> написал(а):
> 
> Hello, dear Igniters!
> 
> I'd like to propose a refactoring of the current Shutdown Policy mechanism.
> Currently, node shutdown is controlled by a bunch of flags and enums (I may
> be missing some of them):
> 
> 1. ShutdownPolicy enum has only two entries and is basically a flag that
> dictates if we need to wait for the data to be backed up, so that we will
> not lose any data when the node goes offline.
> 2. "cancel" flag is passed to the stop method and indicates that all long
> running jobs should be interrupted. For example, it prevents the node
> performing a checkpoint on shutdown. I'm pretty sure this flag is nearly
> always set to "true", because it is used by the Shutdown Hook.
> 3. There are also a bunch of system properties that affect the shutdown:
>* IGNITE_PDS_SKIP_CHECKPOINT_ON_NODE_STOP - disables checkpoints on
> node stop, which heavily correlates with the "cancel" flag.
>* IGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN - deprecated and duplicates the
> ShutdownPolicy enum
>* IGNITE_NO_SHUTDOWN_HOOK - disables the shutdown hook. I don't know if
> it is really useful.
> 
> I think that this approach is counter intuitive and would like to
> incorporate all aforementioned flags into the ShutdownPolicy enum. For
> example, it can look as follows:
> 
> public enum ShutdownPolicy {
>IMMEDIATE // Stops the node and cancels all running jobs
>GRACEFUL // Stops the node, does not cancel running jobs and waits for
> backups
>GRACEFUL_NO_BACKUPS // Stops the node, does not cancel running jobs,
> does not wait for backups
> }
> 
> After that, the "cancel" flag and all corresponding properties can be
> removed (apart from the IGNITE_NO_SHUTDOWN_HOOK, if it is still needed).
> 
> Does this make sense? What do you think?
> 
> -- 
> With regards,
> Aleksandr Polovtsev

Re: Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Николай Ижиков]

Hello, Mike.

Ignite provide built-in performance tests - yardstick.
You can read about them in documentation.

As this tests requires some hi-end hardware it executes on the private 
infrastructure.
So, yes, results of the tests discussed privately.

You can reproduce this results if you have enough hardware and time :).

Fixes of performance are done ASAP and you can see them in commit history.
Like the last one - 
https://github.com/apache/ignite/commit/3b0b5ed9a41fd9f3249f8c42bb59ba0e3ceb55d1
 



> 29 сент. 2022 г., в 15:22, Mike Wiesenberg  написал(а):
> 
> Can you elaborate more? What were the performance results you observed? Is
> this discussion taking place elsewhere?
> 
> MW
> 
> On 2022/09/23 08:42:01 Taras Ledkov wrote: > Dear Ignite Community! > >
> Release 2.14 has been delayed due to performance issues (compare with 2.13)
> for IgniteSqlQueryBenchmark in in-memory mode. We're looking. > > Any
> suggestions and support are welcome. >

[DISCUSSION] BPlusTree design: links and papers

[Николай Ижиков]

Hello, Igniters and espesially Ignite veterans.


While study Ignite B+ tree I’m wondering - what was initial design of this data 
structure?
Is it based on some paper or inspired by some existing implementation?
What should I study beside source code to better understand Ignite b+ tree and 
how it can be improved?

Re: [DISCUSSION] Customers Migraton from Thick to Thin Clients

[Николай Ижиков]

Pavel.

> It is not cryptic, it is very straightforward and builds on existing binary 
> type mechanism.

I see the Ivan’s point here.
As I said earlier - It must be absolutely clear for user - PA works or not.

With the logic you propose it will be hiding inside Ignite machinery and logs.

> Unfortunately, this is a breaking change. Currently this scenario works 
> without errors - uses default socket for custom affinity caches.

It’s OK from my point of view to introduce such a change.


> 10 июля 2022 г., в 22:05, Pavel Tupitsyn  написал(а):
> 
>> providing simple function Object -> int32 in cache configuration is wrong
> decision
> 
> Because it is error-prone and does not work for all cases:
> - You have to set it explicitly on the client for every cache.
> - No way to do that if you get an existing cache by name.
> 
>> still being unable to solve the problem
> 
> I believe the proposed solution covers all use cases.
> 
>> cryptic logic
> 
> It is not cryptic, it is very straightforward and builds on existing binary
> type mechanism.
> 
>> feature flags
> 
> What's bad about feature flags?
> 
>> how an ability to obtain a classname of java class can help my python or
> C++ client map key to partition
> 
> Not a java class name, but type id. You can set up type id mapping properly
> and use this in any thin client.
> This will work for .NET client, not sure about Python and C++.
> 
> On Sun, Jul 10, 2022 at 9:33 PM Ivan Daschinsky  wrote:
> 
>> Guys, I simply cant understand why providing simple function Object ->
>> int32 in cache configuration is wrong decision, but implementing cryptic
>> logic with class names, feature flags and still being unable to solve the
>> probem is ok. I don't understand how an ability to obtain a classname of
>> java class can help my python or C++ client map key to partition.
>> 
>> Max's proposal seems to be a good variant, just change naming a little bit,
>> maybe
>> 
>> пт, 8 июл. 2022 г., 15:35 Pavel Tupitsyn :
>> 
>>> Nikolay,
>>> 
>>>> Add ability to sent custom affinity class name.
>>> Can you please elaborate? What is sent where?
>>> 
>>>> If `partitionAwarenessEnabled=true` but custom affinity can’t be
>>> instantiated on the client - throw an exception
>>> Unfortunately, this is a breaking change. Currently this scenario works
>>> without errors - uses default socket for custom affinity caches.
>>> 
>>> 
>>> On Fri, Jul 8, 2022 at 3:23 PM Николай Ижиков 
>> wrote:
>>> 
>>>> Hello, Pavel.
>>>> 
>>>> I support your proposal to transparently create custom AffinityMapper
>>>> based on server node classname, in general.
>>>> Partition Awareness crucial point for a thin client performance so It
>>>> should be absolutely clear for a user - PA works correctly or not.
>>>> 
>>>> So, I think, we should implement the following:
>>>> 
>>>> 0. Add ability to sent custom affinity class name.
>>>> 1. If `partitionAwarenessEnabled=true` but custom affinity can’t be
>>>> instantiated on the client - throw an exception.
>>>> 
>>>> WDYT?
>>>> 
>>>>> 8 июля 2022 г., в 08:37, Pavel Tupitsyn 
>>>> написал(а):
>>>>> 
>>>>> To clarify: you can use a different AffinityFunction implementation
>> on
>>>> the
>>>>> client side. It just has to map to the same binary typeId.
>>>>> Even if there is a legacy setup with AffinityKeyMapper on servers,
>> you
>>>> can
>>>>> craft an AffinityFunction for the client that achieves correct
>>> behavior.
>>>>> So I believe this should cover any use case.
>>>>> 
>>>>> On Thu, Jul 7, 2022 at 10:36 PM Pavel Tupitsyn >> 
>>>> wrote:
>>>>> 
>>>>>> AffinityKeyMapper is just a subset of AffinityFunction, isn't it?
>>>>>> So by supporting AffinityFunction we cover all AffinityKeyMapper use
>>>> cases.
>>>>>> 
>>>>>> 
>>>>>>> managing the classpath, a user should always keep in mind it
>>>>>> I don't consider this a problem. This is how Java works, what can we
>>> do?
>>>>>> You can also stick the class into BinaryConfiguration on the client
>> to
>>>>>> make it explicit and make sure it is loaded.
>>>>>> 
>>>>>>> it still possible havin

Re: [DISCUSSION] Customers Migraton from Thick to Thin Clients

[Николай Ижиков]

interface.
>>>>>>>>> 
>>>>>>>>> public interface ToIntFunction {
>>>>>>>>> int applyAsInt(Object key);
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> Another point that I'd like also to mention is that the cache
>>>>> already
>>>>>>>>> created in a cluster, so it will be better to eliminate the
>>>>> duplicate
>>>>>>>>> affinity function configuration (setting the number of
>>>>> partitions). It
>>>>>>>>> is fully possible to receive the number of partitions from a
>>> server
>>>>>>>>> node (the same way as it currently happening). Thus instead
>>> of the
>>>>>>>>> ToInFunction interface it will be better to use
>>>>>>>>> ToInBiFunction.
>>>>>>>>> 
>>>>>>>>> public interface ToIntBiFunction {
>>>>>>>>> int applyAsInt(Object key, Integer parts);
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I agree with you that "setPartitionAwarenessKeyMapper" may be
>>> is
>>>>> not
>>>>>>>>> good naming here, we can rename it and move it to the public
>>> API,
>>>>> of
>>>>>>>>> course.
>>>>>>>>> 
>>>>>>>>> On Thu, 7 Jul 2022 at 20:06, Pavel Tupitsyn <
>>> ptupit...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Maxim,
>>>>>>>>>> 
>>>>>>>>>> I am confused. We were talking about a custom Affinity
>>> Function.
>>>>>>>>>> As you noted, AffinityKeyMapper is deprecated, why do we add
>>>>> something
>>>>>>>>>> named "setPartitionAwarenessKeyMapper"?
>>>>>>>>>> 
>>>>>>>>>> Internal API approach is hacky.
>>>>>>>>>> IMO we should either develop a proper feature with a good
>>> public
>>>>> API, or
>>>>>>>>>> not add anything at all.
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jul 7, 2022 at 6:34 PM Maxim Muzafarov <
>>>>> mmu...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thank you for your comments.
>>>>>>>>>>> 
>>>>>>>>>>> First of all, I'd like to point out that if the cache is
>>>>> created with
>>>>>>>>>>> a custom affinity function or the legacy AffinityKeyMapper
>>>>> interface,
>>>>>>>>>>> the thin client should be able to provide only a
>>>>> `key-to-partition`
>>>>>>>>>>> mapping function to handle the case described above. The
>>>>>>>>>>> `partition-to-node` mappings the client will receive from
>>> a
>>>>> server
>>>>>>>>>>> node it connected to. This will simplify a bit the final
>>>>> solution.
>>>>>>>>>>> 
>>>>>>>>>>> ==
>>>>>>>>>>> 
>>>>>>>>>>> I've checked your suggestions and it looks like both of
>>> them
>>>>> have some
>>>>>>>>>>> sufficient drawbacks:
>>>>>>>>>>> 
>>>>>>>>>>> 1. Using SystemProperty looks really hacky and we are
>>>>> explicitly
>>>>>>>>>>> complicate a thin client configuration for an end user.
>>>>>>>>>>> 
>>>>>>>>>>> 2. Extending the ClientCacheConfiguration is a very good
>>> and
>>>>>>>>>>> straightforward idea, however it doesn't fit the case
>>>>> described above.
>>>>>>>>>>> Caches previously 

Re: Access to wiki to write a IEP draft

[Николай Ижиков]

Hello, please, provide you Jira ID.

> 8 июля 2022 г., в 12:18, Sergey Korotkov  
> написал(а):
> 
> Hello Igniters,
> 
> I would like to prepare the IEP for some improved loading testing framework 
> for Ignite.
> 
> Would you please give me a write access to https://cwiki.apache.org/confluence
> 
> Thanks,
> 
> --
> 
>Sergey Korotkov
> 

Re: [DISCUSSION] Customers Migraton from Thick to Thin Clients

[Николай Ижиков]

+1 to have ability to specify custom affinity for PA on thin client.

It seems to me custom affinity is a rare use-case of Ignite API.
Propose to have SystemProperty that can specify affinity implementation for a 
thin client.


> 29 июня 2022 г., в 18:53, Maxim Muzafarov  написал(а):
> 
> Igniters,
> 
> 
> I've faced with a customer's cluster which has more than 150 nodes and
> most for them are the thick-client nodes. Due to each thick-client is
> a full-fledged cluster topology participant it affects the cluster
> discovery machinery during the system operation and adding an
> additional overhead for using/deploying a new nodes in Kubernetes
> environment. However, the main thing from my point of view it prevents
> updating the client side and server side components independently
> (Apache Ignite doesn't support rolling upgrade).
> 
> Accordingly to the assumptions above using thin clients become a
> necessary. This looks even more attractive, since the thin client has
> a fairly rich API over the past few releases.
> 
> 
> The MAIN ISSUE here that blocks thin client usage is that for some of
> cache groups a custom affinity function (and an AffinityKeyMapper) was
> used which prevents enabling the Partition Awareness thin client
> feature. Thus each cache request will have two hops.
> 
> Of course, we can force users to migrate to a new API, but this
> becomes more difficult when Apache Ignite is part of a much larger
> architectural solution and thus it is doent' looks so friendly.
> 
> The MAIN QUESTION here - does anyone know our users who have
> encountered with the same issue? I want to solve such a problem once
> and make all such users happy by implementing the general approach.
> 
> 
> = Possible solutions =
> 
> 
> 1. Making an affinity function pluggable (mapping calculations) on the
> thin clients side. Currently the RendezvousAffinityFunction [1] is
> only supported
> for the partition awareness. A user's affinity function seems to be
> the stateless function due to there is no machinery to transfer states
> to the thin client.
> 
> Pros - a general solution for all such cases;
> Cons - unnecessary complexity, extending public API;
> 
> 
> 2. Creating an Ignite extension which will extend the thin client API
> thus a user will have a full control over a destination node to which
> requests being sent.
> 
> Pros - isolated solution, simple implementation;
> Cons - hard to support spring-boot-thin-client etc. and other
> extensions based on the thin client API;
> 
> 
> Folks, please share your thoughts.
> 
> 
> [1] 
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePartitionsRequest.java#L206

Re: Question about the sources of the Apache Ignite version 2.12.0-1

[Николай Ижиков]

Hello.

Apache Ignite doesn’t have version 2.12.0-1

https://ignite.apache.org/download.cgi  
- here you can find existing releases. Links to source archives provided, also. 

https://github.com/apache/ignite/tags  - 
same sources on GitHub.

> 17 июня 2022 г., в 16:53, Koltermann, BENJAMIN 
>  написал(а):
> 
> Hi Apache Ignite Dev Team,
>  
> I have a question to the sources of the Apache Ignite build 2.12.0-1. Which 
> sources were used for the build. Were the same sources used as in version 
> 2.12.0 or is there a difference?
> With best regards,
> Benjamin Koltermann
> 
> Siemens Mobility GmbH
> SMO RI ML ADC ETCS AR
> Ackerstr. 22
> 38126 Braunschweig, Germany
> Mobile: +49 (173) 4147755
> mailto:benjamin.kolterm...@siemens.com 
> 
> www.siemens.com 
> 
> Siemens Mobility GmbH; Chairman of the Supervisory Board: Roland Busch; 
> Management Board: Karl Blaim, Michael Peter; Registered office: Munich, 
> Germany; Commercial registry Munich, HRB 237219; WEEE-Reg.-No. DE 92917817

[DISCUSSION] index-reader and orphaned pages

[Николай Ижиков]

Hello Igniters.

We have very useful `index-reader` utility that can check index file 
consistency.
I try to run it on several test and production Ignite deployments and found 
that frequently util report orphaned pages inside index.bin file.

Example of output of production deployment:

```
Exception occurred on step 981750: Possibly orphan InlineInnerIO page, 
pageId=844420636146422
Exception occurred on step 981759: Possibly orphan InlineInnerIO page, 
pageId=844420636146431
Exception occurred on step 981775: Possibly orphan InlineInnerIO page, 
pageId=844420636146447
Exception occurred on step 981790: Possibly orphan InlineInnerIO page, 
pageId=844420636146462
Total pages encountered during sequential scan: 981844
Total errors occurred during sequential scan: 44853

```

Test deployment:

```
Error [step=10876, msg=Possibly orphan InlineInnerIO page, 
pageId=844420635175548]
Error [step=11048, msg=Possibly orphan InlineInnerIO page, 
pageId=844420635175720]
Error [step=11081, msg=Possibly orphan InlineInnerIO page, 
pageId=844420635175753]
Error [step=9, msg=Possibly orphan InlineInnerIO page, 
pageId=844420635175791]
Error [step=11169, msg=Possibly orphan InlineInnerIO page, 
pageId=844420635175841]

Total pages encountered during sequential scan:11543
Total errors occurred during sequential scan: 88
```

So my questions are

- Do we know the way how orphaned pages sneaks into index.bin?
- Do we have tickets/fixes to prevent this situation?
- How Ignite admin should fix this in real-world deployments?

Re: Control.sh command that schedules index rebuild in the maintenance mode

[Николай Ижиков]

Hello, Семен.

Why index rebuild should be done in maintenance mode?

> 31 мая 2022 г., в 18:24, Данилов Семён  написал(а):
> 
> Hello, igniters!
> 
> I want to propose a new control.sh cache sub-command that schedules a rebuild 
> of cache indexes in the maintenance mode. 
> We already have a force rebuild command but sometimes (for example, under 
> load) user may want to first stop the node and then rebuild indexes.
> For this scenario we can add a command that creates a maintenance task and 
> then after restart node will enter the maintenance mode and rebuild specified 
> indexes.
> 
> I propose the following API:
> --cache schedule_indexes_rebuild --node-id nodeId --target 
> cacheName1=index1,...indexN
> Schedules rebuild of the indexes for specified caches via the Maintenance 
> Mode.
> 
> Parameters:
>--node-id - (Optional) Specify node for indexes rebuild.
>--target- Cache name with optionally specified indexes. If indexes 
> are not specified then all indexes of the cache will be scheduled for the 
> rebuild operation.
> 
> As you can see, the user can provide a node id to schedule the rebuild on a 
> specific node (otherwise the rebuild will be scheduled for all of the nodes) 
> and targets (specific indexes of a cache or all of the indexes of a cache).
> There is a ticket[1] and a pull request[2] for my proposal.
> 
> WDYT?
> 
> Kind regards,
> Semyon.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-17002
> [2] https://github.com/apache/ignite/pull/10042
> 

Re: restful api get key result is null,but java thin clien get key has result

[Николай Ижиков]

Hello.

Please, make sure you have equal value of BinaryConfiguration#compactFooter 
both for server node and for thin client.
Set specific value (true) in both configuration
They are not equals, by default for the backward compatibility reasons.

Having different compactFooter value can lead to described behavior. 

> 23 мая 2022 г., в 15:16, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Please write to user list (u...@ignite.apache.org)
> 
> Having said that, I think you need to qualify this key class with package
> name when doing REST.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пн, 23 мая 2022 г. в 13:59, wkhapy...@163.com :
> 
>> i use java thin client i
>> 
>> 
>> 
>> wkhapy...@163.com
>> 

Re: [DISCUSSION] Move the ignite-yarn, ignite-geospatial, ignite-schedule to the Ignite Extensions

[Николай Ижиков]

+1

> 13 мая 2022 г., в 17:51, Maxim Muzafarov  написал(а):
> 
> Folks,
> 
> I've found that the following modules are rarely used and haven't been
> change for a few years, but  they are released each time Ignite being
> released. It seems we can safely move all of them to the Extensions
> project.
> 
> The list:
> - ignite-yarn
> - ignite-geospatial
> - ignite-schedule
> 
> WDYT?

Re: [VOTE] Release Extensions Ignite Spring Data 2.0.0 RC1, Ignite Spring Session 1.0.0 RC1, Ignite Zookeeper IP Finder 1.0.0 RC1

[Николай Ижиков]

+1

> 13 мая 2022 г., в 12:53, Maxim Muzafarov  написал(а):
> 
> Dear Community,
> 
> 
> The release candidates are ready.
> 
> The ignite-zookeeper-ip-finder and ignite-spring-session haven't been
> changed since the last vote. All the issues mentioned in the previous
> vote related to the ignite-spring-data-ext have also fixed.
> 
> See the links below.
> 
> 
> == Ignite Spring Data Extension Module 2.0.0 ==
> 
> The release candidate is uploaded to:
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-data-ext-2.0.0-rc1/
> 
> The following Maven staging can be used:
> https://repository.apache.org/content/repositories/orgapacheignite-1550/org/apache/ignite/ignite-spring-data-ext/
> 
> Tag:
> https://github.com/apache/ignite-extensions/releases/tag/ignite-spring-data-ext-2.0.0-rc1
> 
> RELEASE_NOTES:
> https://github.com/apache/ignite-extensions/blob/release/ignite-spring-data-ext-2.0.0/modules/spring-data-ext/spring-data/RELEASE_NOTES.txt
> 
> DEVNOTES:
> https://github.com/apache/ignite-extensions/blob/release/ignite-spring-data-ext-2.0.0/DEVNOTES.md
> 
> Release Checker Results:
> https://github.com/apache/ignite-extensions/runs/6420862446?check_suite_focus=true
> 
> 
> 
> == Ignite Spring Session Extension Module 1.0.0 ==
> 
> The release candidate is uploaded to:
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-session-ext-1.0.0-rc1/
> 
> The following Maven staging can be used:
> https://repository.apache.org/content/repositories/orgapacheignite-1548/org/apache/ignite/ignite-spring-session-ext/
> 
> Tag:
> https://github.com/apache/ignite-extensions/releases/tag/ignite-spring-session-ext-1.0.0-rc1
> 
> RELEASE_NOTES:
> https://github.com/apache/ignite-extensions/blob/release/ignite-spring-session-ext-1.0.0/modules/spring-session-ext/RELEASE_NOTES.txt
> 
> DEVNOTES:
> https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md
> 
> Release Checker Results:
> https://github.com/apache/ignite-extensions/runs/6229613223?check_suite_focus=true
> 
> 
> == Ignite Zookeeper Ip Finder Extension Module 1.0.0 ==
> 
> The release candidate is uploaded to:
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-zookeeper-ip-finder-ext-1.0.0-rc1/
> 
> The following Maven staging can be used:
> https://repository.apache.org/content/repositories/orgapacheignite-1549/org/apache/ignite/ignite-zookeeper-ip-finder-ext/
> 
> Tag:
> https://github.com/apache/ignite-extensions/releases/tag/ignite-zookeeper-ip-finder-ext-1.0.0-rc1
> 
> RELEASE_NOTES:
> https://github.com/apache/ignite-extensions/blob/release/ignite-zookeeper-ip-finder-ext-1.0.0/modules/zookeeper-ip-finder-ext/zookeeper-ip-finder/RELEASE_NOTES.txt
> 
> DEVNOTES:
> https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md
> 
> Release Checker Results:
> https://github.com/apache/ignite-extensions/runs/6210193576?check_suite_focus=true
> 
> 
> 
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
> 
> +1 - to accept (Ignite Spring Data 2.0.0 RC1, Ignite Spring Session
> 1.0.0 RC1, Ignite Zookeeper IP Finder 1.0.0 RC1)
> 0 - don't care either way
> -1 - DO NOT accept (explain why)
> 
> See notes on how to verify release here
> https://www.apache.org/info/verification.html
> and
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> 
> 
> This vote will be open for at least 4 days till Tue May 17, 2022,
> 14:00 Moscow TZ. Please, write me down the thread if you need
> additional time to check the release.
> 
> https://www.timeanddate.com/countdown/vote?iso=20220517T14=166=%5BVOTE%5D+Release+Extensions+Ignite+Spring+Data+2.0.0+RC1%2C+Ignite+Spring+Session+1.0.0+RC1%2C+Ignite+Zookeeper+IP+Finder+1.0.0+RC1=serif

Re: [VOTE] Release Ignite AWS 1.0.0 RC1, Ignite Azure 1.0.0 RC1, Ignite GCE 1.0.0 RC1

[Николай Ижиков]

+1

> 4 мая 2022 г., в 17:43, Andrey Gura  написал(а):
> 
> +1
> 
> On Fri, Apr 29, 2022 at 7:22 PM Nikita Amelchev  wrote:
>> 
>> +1
>> 
>> пт, 29 апр. 2022 г. в 15:51, Anton Vinogradov :
>>> 
>>> +1
>>> 
>>> On Fri, Apr 29, 2022 at 3:48 PM Maxim Muzafarov  wrote:
>>> 
 Dear Community,
 
 
 The release candidates are ready. See the links below.
 
 
 == Ignite AWS Extension Module 1.0.0 ==
 
 The release candidate is uploaded to:
 
 https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-aws-ext-1.0.0-rc1
 
 The following Maven staging can be used:
 
 https://repository.apache.org/content/repositories/orgapacheignite-1543/org/apache/ignite/ignite-aws-ext/
 
 Tag:
 
 https://github.com/apache/ignite-extensions/releases/tag/ignite-aws-ext-1.0.0-rc1
 
 RELEASE_NOTES:
 
 https://github.com/apache/ignite-extensions/blob/release/ignite-aws-ext-1.0.0/modules/aws-ext/RELEASE_NOTES.txt
 
 DEVNOTES:
 https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md
 
 Release Checker Results:
 
 https://github.com/apache/ignite-extensions/runs/6227581198?check_suite_focus=true
 
 
 == Ignite Azure Extension Module 1.0.0 ==
 
 The release candidate is uploaded to:
 
 https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-azure-ext-1.0.0-rc1/
 
 The following Maven staging can be used:
 
 https://repository.apache.org/content/repositories/orgapacheignite-1545/org/apache/ignite/ignite-azure-ext/
 
 Tag:
 
 https://github.com/apache/ignite-extensions/releases/tag/ignite-azure-ext-1.0.0-rc1
 
 RELEASE_NOTES:
 
 https://github.com/apache/ignite-extensions/blob/release/ignite-azure-ext-1.0.0/modules/azure-ext/RELEASE_NOTES.txt
 
 DEVNOTES:
 https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md
 
 Release Checker Results:
 
 https://github.com/apache/ignite-extensions/runs/6228019576?check_suite_focus=true
 
 
 == Ignite GCE Extension Module 1.0.0 ==
 
 The release candidate is uploaded to:
 
 https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-gce-ext-1.0.0-rc1/
 
 The following Maven staging can be used:
 
 https://repository.apache.org/content/repositories/orgapacheignite-1546/org/apache/ignite/ignite-gce-ext/
 
 Tag:
 
 https://github.com/apache/ignite-extensions/releases/tag/ignite-gce-ext-1.0.0-rc1
 
 RELEASE_NOTES:
 
 https://github.com/apache/ignite-extensions/blob/release/ignite-gce-ext-1.0.0/modules/gce-ext/RELEASE_NOTES.txt
 
 DEVNOTES:
 https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md
 
 Release Checker Results:
 
 https://github.com/apache/ignite-extensions/runs/6228169980?check_suite_focus=true
 
 
 The vote is formal, see voting guidelines
 https://www.apache.org/foundation/voting.html
 
 +1 - to accept (Ignite AWS 1.0.0 RC1, Ignite Azure 1.0.0 RC1, Ignite
 GCE 1.0.0 RC1)
 0 - don't care either way
 -1 - DO NOT accept (explain why)
 
 See notes on how to verify release here
 https://www.apache.org/info/verification.html
 and
 
 https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
 
 
 This vote will be open for at least 7 days till Fri May 7, 2022,
 14:00 Moscow TZ. Please, write me down the thread if you need
 additional time to check the release.
 
 
 https://www.timeanddate.com/countdown/vote?iso=20220507T14=166=%5BVOTE%5D+Release+Ignite+AWS+1.0.0+RC1%2C+Ignite+Azure+1.0.0+RC1%2C+Ignite+GCE+1.0.0+RC1=sanserif
 
>> 
>> 
>> 
>> --
>> Best wishes,
>> Amelchev Nikita

Re: [DISCUSSION] Move ignite-hibernate modules to the Ignite Extensions

[Николай Ижиков]

+1

> 26 апр. 2022 г., в 16:24, Maxim Muzafarov  написал(а):
> 
> Hello Igniters,
> 
> 
> Currently we have a batch of ignite-hibernate modules which provide
> Hibernate second-level cache (L2 cache) implementation based on the
> Apache Ignite for different version of hibernate.
> 
> The following list of modules we have:
> - ignite-hibernate_4.2
> - ignite-hibernate_5.1
> - ignite-hibernate_5.3
> - ignite-hibernate-core (a common part for all hibernate modules)
> 
> I propose to use the same approach as was used for the
> ignite-spring-data module [1] and move all these modules to the Ignite
> Extesions project.
> 
> In details:
> 
> - remove all these modules from the Ignite project.
> - create ignite-hibernate extension.
> - move ignite-hibernate-core + ignite-hibernate_4.2 to
> release/ignite-hibernate-4.2.0 branch (the version of ignite-hibernate
> extension will be 4.2.0) and release it on demand;
> - move ignite-hibernate-core + ignite-hibernate_5.1 to
> release/ignite-hibernate-5.1.0 branch (the version of ignite-hibernate
> extension will be 5.1.0) and release it on demand;
> - move ignite-hibernate-core + ignite-hibernate_5.3 to the master
> branch and to the release/ignite-hibernate-5.3.0 branch (the version
> of ignite-hibernate extension will be 5.3.0) and release it
> immediately;
> 
> 
> Is it OK or am I missing something?
> WDYT?
> 
> 
> [1] https://lists.apache.org/thread/b03bny7xj9x5ty371srtdn9ysd72qht7

Re: [VOTE] Release Apache Ignite 2.13.0 RC2

[Николай Ижиков]

+1 (binding)

> 23 апр. 2022 г., в 17:32, Nikita Amelchev  написал(а):
> 
> Hi Igniters, PMCs,
> 
> Please, verify a new release candidate.
> 
> The vote will continue until there are enough votes for a resolution. [1]
> 
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> 
> пт, 22 апр. 2022 г. в 08:24, Zhenya Stanilovsky :
>> 
>> 
>> Nikita thanks for your effort !
>> Download sources, run examples.
>> +1 from me.
>> 
>>> Dear Community,
>>> 
>>> The release candidate is ready.
>>> 
>>> I have uploaded a release candidate to:
>>> https://dist.apache.org/repos/dist/dev/ignite/2.13.0-rc2/
>>> https://dist.apache.org/repos/dist/dev/ignite/packages_2.13.0-rc2/
>>> 
>>> The following staging can be used for testing:
>>> https://repository.apache.org/content/repositories/orgapacheignite-1542
>>> 
>>> Tag name is 2.13.0-rc1:
>>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.13.0-rc2
>>> 
>>> RELEASE_NOTES:
>>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.13
>>> 
>>> Complete list of resolved issues:
>>> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.13%27))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority
>>> 
>>> DEVNOTES:
>>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.13
>>> 
>>> Additional checks have been performed (available for users included
>>> into the release group on TeamCity).
>>> 
>>> TC [Check RC: Licenses, compile, chksum]
>>> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum/6399934
>>> 
>>> TC [2] Compare w/ Previous Release
>>> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/6399932
>>> 
>>> The vote is formal, see voting guidelines
>>> https://www.apache.org/foundation/voting.html
>>> 
>>> +1 - to accept Apache Ignite 2.13.0-rc2
>>> 0 - don't care either way
>>> -1 - DO NOT accept Apache Ignite Ignite 2.13.0-rc2 (explain why)
>>> 
>>> See notes on how to verify release here
>>> https://www.apache.org/info/verification.html
>>> and
>>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
>>> 
>>> This vote will be open for at least 3 days till Fri Apr 23, 2022,
>>> 20:00 UTC. Please, write me down the thread if you need additional
>>> time to check the
>>> release.
>>> https://www.timeanddate.com/countdown/vote?iso=20220423T20=0=VOTE+on+the+Apache+Ignite+Release+2.13.0+RC2=sanserif
>>> 
>>> --
>>> Best wishes,
>>> Amelchev Nikita
>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> Best wishes,
> Amelchev Nikita

Re: [DISCUSSION] Remove scalar module

[Николай Ижиков]

Ticket created - https://issues.apache.org/jira/browse/IGNITE-16820 
<https://issues.apache.org/jira/browse/IGNITE-16820>

> 7 апр. 2022 г., в 16:43, Petr Ivanov  написал(а):
> 
> Hi!
> 
> 
> +1 for removal.
> That module is long time obsolete and prevents project from JDK upgrades at 
> least.
> 
>> On 7 Apr 2022, at 15:15, Ilya Kasnacheev  wrote:
>> 
>> Hello!
>> 
>> I think it is a good idea, it takes ages to compile and I assume the Scala
>> version we use is really outdated now.
>> 
>> Regards,
>> -- 
>> Ilya Kasnacheev
>> 
>> 
>> чт, 7 апр. 2022 г. в 14:07, Николай Ижиков :
>> 
>>> Hello, Igniters.
>>> 
>>> Looks like scalar module abandoned for at least for 4 years - no user-list
>>> questions, no source updates.
>>> I propose to remove it completely.
>>> 
>>> WDYT?
> 

[DISCUSSION] Remove scalar module

[Николай Ижиков]

Hello, Igniters.

Looks like scalar module abandoned for at least for 4 years - no user-list 
questions, no source updates.
I propose to remove it completely.

WDYT?

Re: [VOTE] Release pyignite 0.5.2-rc0

[Николай Ижиков]

+1

Отправлено с iPhone

> 14 сент. 2021 г., в 11:39, Pavel Tupitsyn  написал(а):
> 
> +1
> 
> Checked on Ubuntu 20.04, ran a few examples.
> 
>> On Tue, Sep 14, 2021 at 11:12 AM Ivan Daschinsky 
>> wrote:
>> 
>> +1 from me
>> Checked on windows 10 x86_64 (visual studio 2017) and ubuntu 20.04 x86_64
>> and on pythons 3.6, 3.7, 3.8, 3.9 (both win and linux):
>> 1. Installing from source -- passe
>> 2. Building wheels from source -- passed
>> 3. Installing wheels -- passed
>> 
>> Checked on each steps C module and examples. -- passed
>> 
>> Checked hashsums and gpg signatures -- passed
>> 
>> пт, 10 сент. 2021 г. в 19:08, Ivan Daschinsky :
>> 
>>> Dear Igniters!
>>> 
>>> Release candidate binaries for subj are uploaded and ready for vote
>>> You can find them here:
>>> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.2-rc0
>>> 
>>> If you follow the link above, you will find source packages (*.tar.gz and
>>> *.zip)
>>> and binary packages (wheels) for windows (amd64) and linux (x86_64)
>>> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
>>> Code signing keys can be found here --
>>> https://downloads.apache.org/ignite/KEYS
>>> Here you can find instructions how to verify packages
>>> https://www.apache.org/info/verification.html
>>> 
>>> You can install binary package for specific version of python using pip
>>> For example do this on linux for python 3.8
> pip install pyignite-0.5.2-cp38-cp38-manylinux1_x86_64.whl
>>> 
>>> You can build and install package from source using this command:
> pip install pyignite-0.5.2.tar.gz
>>> You can build wheel on your platform using this command:
> pip wheel --no-deps pyignite-0.5.2.tar.gz
>>> 
>>> For building C module, you should have python headers and C compiler
>>> installed.
>>> (i.e. for ubuntu sudo apt install build-essential python3-dev)
>>> In Mac OS X xcode-tools and python from homebrew are the best option.
>>> 
>>> In order to check whether C module works, use following:
> from pyignite import _cutils
> print(_cutils.hashcode('test'))
> 3556498
>>> 
>>> You can find documentation here:
>>> 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/
>>> 
>>> You can find examples here (to check them, you should start ignite
>>> locally):
>>> 
>>> 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/examples.html
>>> Also, examples can be found in source archive in examples subfolder.
>>> docker-compose.yml is supplied in order to start ignite quickly. (Use
>>> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
>>> down` to shut down it)
>>> 
>>> Release notes:
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=a67624a9c141ad5457cdda1a50bd57af5ac62615;hb=1222f29abca4c44a8a2f23e413eafb6acd332e76
>>> 
>>> Git release tag was created:
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.2.rc0
>>> 
>>> The vote is formal, see voting guidelines
>>> https://www.apache.org/foundation/voting.html
>>> 
>>> +1 - to accept pyignite-0.5.2-rc0
>>> 0 - don't care either way
>>> -1 - DO NOT accept pyignite-0.5.2-rc0
>>> 
>>> The vote finishes at 09/15/2021 15:00 UTC
>>> 
>> 
>> 
>> --
>> Sincerely yours, Ivan Daschinskiy
>> 

Re: [Announcement] Apache Ignite 2.11 Code Freeze started

[Николай Ижиков]

+1 to fix both prior to release

Отправлено с iPhone

> 22 июля 2021 г., в 02:36, Maxim Muzafarov  написал(а):
> 
> Folks,
> 
> We've faced [1][2] issues related to the new functionality added to
> the 2.11 release (it will be safe to merge to the release branch).
> From my point of view, both of them are critical and must be included
> in the 2.11 release.
> The [2] is ready for merge. The [1] will be completed by the end of this week.
> 
> Please share your thoughts.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15146
> [2] https://issues.apache.org/jira/browse/IGNITE-15170
> 
>> On Tue, 20 Jul 2021 at 15:08, Maxim Muzafarov  wrote:
>> 
>> Folks,
>> 
>> Since the release branch was created some time ago, should we bump up
>> the master branch version to the next one?
>> 
>>> On Thu, 1 Jul 2021 at 17:15, Alexey Gidaspov  wrote:
>>> 
>>> Hi, Pavel.
>>> 
>>> I think, it looks like blocker. Please cherry-pick it to 2.11 release branch
>>> 
>>> On 2021/07/01 09:29:57, Pavel Pereslegin  wrote:
 Hello, Alexey!
 
 Is it possible to include a hotfix that corrects the command syntax
 output in the control script? [1]
 
 This bug can significantly complicate the use of the snapshot restore
 function (one of the important features of 2.11). In addition, this
 may raise a number of questions to the product support, which we can
 prevent by adding this patch in 2.11.
 
 This patch does not affect any functions other than the "help" output
 of the control script.
 
 [1] https://issues.apache.org/jira/browse/IGNITE-14989
 
 чт, 1 июл. 2021 г. в 11:56, Alexey Gidaspov :
> 
> Hi, Iilya!
> 
> As I can see, this feature highly improves debugging during incidents. So 
> I think we can call it blocker and cherry-pick to ignite-2.11 branch
> 
> On 2021/06/30 20:26:43, Shishkov Ilya  wrote:
>> Hello, Alexey!
>> Is it possible to add system views for BaselineNode attributes [1] and
>> corresponding documentation [2] to 2.11?
>> 
>> 1. https://issues.apache.org/jira/browse/IGNITE-15007
>> 2. https://issues.apache.org/jira/browse/IGNITE-15028
>> 
>> ср, 30 июн. 2021 г. в 11:07, Nikita Amelchev :
>> 
>>> Thanks! I have cherry-picked the commit to the 2.11 branch.
>>> 
>>> ср, 30 июн. 2021 г. в 11:00, Alexey Gidaspov :
 
 Hi, Nikita!
 
 I think it's important fix and should be included in 2.11 release. I've
>>> tagged ticket by fixVersion 2.11. Can you cherry-pick it to ignite-2.11
>>> branch? And please fill release notes or delete flag.
 
 On 2021/06/30 07:55:04, Nikita Amelchev  wrote:
> Hello, Alexey.
> 
> I suggest adding to the 2.11 scope the resolved issue [1]. It fixes
> incorrect values of cache, cache groups, data region metrics after
> cluster reactivation.
> WDYT?
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-14990
> 
> вт, 29 июн. 2021 г. в 15:09, Alexey Gidaspov :
>> 
>> Ok, we can add this fix to release scope.
>> 
>> On 2021/06/29 08:36:00, Сурков Александр
>>> Викторович wrote:
>>> Hi Alexey.
>>> 
>>> I think need add ticket: JmxMetricExporter fails to export
>>> discovery metrics - https://issues.apache.org/jira/browse/IGNITE-14376
>>> 
>>> On 2021/06/15 14:43:55, Alexey Gidaspov  wrote:
 Apache Ignite 2.11 Code Freeze started now>
 
> 
> 
> 
> --
> Best wishes,
> Amelchev Nikita
> 
>>> 
>>> 
>>> 
>>> --
>>> Best wishes,
>>> Amelchev Nikita
>>> 
>> 
 

Re: [DISCUSSION] Code style. Variable abbrevations

[Николай Ижиков]

Hello, Ivan.

We can create checkstyle rule to enforce usage of abbreviations.
Internal/public code differs by package.

PoC of rule [1]

[1] https://github.com/apache/ignite/pull/9153

> 17 июня 2021 г., в 07:01, Ivan Pavlukhin  написал(а):
> 
> Nikita,
> 
> Do you have a plan in your mind how to make Ignite codebase consistent?
> 
> AFAIR, we had it intentionally inconsistent for a long time at least
> for one sake: for internal code we used special conventions (e.g.
> abbreviations, shorten getters) and common Java conventions for public
> API and examples (e.g. no abbreviations and usual getters).
> 
> 2021-06-16 23:37 GMT+03:00, Nikita Ivanov :
>> Consistency is what makes it easier to contribute to the project and
>> attract new members. Consistency implies strong, well defined and
>> universally enforced rules. Just because we have some individuals who
>> are lazy or inexperienced - it does not mean that the entire project
>> should relax the basic-level engineering discipline.
>> 
>> On a personal node - nothing screams "immaturity" louder that a code
>> that uses inconsistent naming, commenting, code style & organization,
>> etc.
>> --
>> Nikita Ivanov
>> 
>>> On Wed, Jun 16, 2021 at 5:56 AM Andrey Gura  wrote:
>>> 
>>> Hi,
>>> 
>>> I understand that this rule seems too strict or may be weird. But
>>> removing the rule could lead to review comments like "use future
>>> instead of fut". So my proposal is to change rule from "required" to
>>> "recommended".
>>> 
>>> On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko
>>>  wrote:
 
 Konstantin, thanks for chiming in.
 
 That's exactly my concern. Overformalization is generally not a good
 thing.
 Someone used "mess" to abbreviate "message"? That is surely not a good
 coding style, but that's what code reviews are for. I believe that our
 committers are more than capable of identifying such issues.
 
 At the same time, with the current rules, we are *forced* to use
 abbreviations like "locAddrGrpMgr", whether we like it or not. All it
 does
 is makes it harder to contribute to Ignite, without providing any clear
 value.
 
 -Val
 
 On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov 
 wrote:
 
> +1 for making this optional
> 
> Common abbreviation rules is good for sure, but sometimes I getting
> sick
> of all those locAddrGrpMgr.
> 
> 
> --
> Regards,
> Konstantin Orlov
> 
> 
> 
> 
>> On 7 Jun 2021, at 14:31, Nikolay Izhikov 
>> wrote:
>> 
>> Hello, Anton, Alexei.
>> 
>> Thanks for the feedback.
>> 
>> Personally, I’m pretty happy current abbreviation rules too.
>> Let see what we can do to make our codebase even more consistent.
>> 
>>> 7 июня 2021 г., в 13:23, Alexei Scherbakov <
> alexey.scherbak...@gmail.com> написал(а):
>>> 
>>> -1
>>> Common abbrevs add quality to the code.
>>> 
>>> пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov :
>>> 
 -1 here.
 We can fix the code and set up the rule.
 
 This will help to prevent having a weird abbreviation like "mess"
 (from
 "message") or "ign" (from "Ignite").
 Also, the abbreviations list (hardcoded at IDEA plugin) allows to
 have
> same
 names across the whole code, this should simplify the reading.
 
 On Sat, Jun 5, 2021 at 10:49 PM Valentin Kulichenko <
 valentin.kuliche...@gmail.com> wrote:
 
> I also support removing this requirement. It’s not the first
> time
> someone
> brings this up, and so far we haven’t been able to fix it. Not
> worth
> it
 in
> my view.
> 
> -Val
> 
> On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov
> 
> wrote:
> 
>> Hello, guys.
>> 
>> Thanks for the feedback.
>> 
>> Dmitry,
>> 
>>> Manual rule enforcement saves a couple of symbols in code
>> 
>> I’m talking about automatic check.
>> We can implement it easily.
>> So, if we decide to keep this rule all we need is to fix
>> current
>> violations (several thousand!).
>> After it, checkstyle will automatically enforce this rule.
>> As you may know, currently, any PR checked according to our
> checkstyle
>> rules.
>> Please, take a look at little green check sign after PR name.
>> 
>> 
>>> 5 июня 2021 г., в 00:57, Dmitry Pavlov 
> написал(а):
>>> 
>>> +1 for removal. Cnt and count both seem to be fine.
>>> 
>>> Manual rule enforcement saves a couple of symbols in code, but
 requires
>> some time from both contributor and reviewer.
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov

Re: Thin Client ping operation?

[Николай Ижиков]

Hello, Pavel.

SQL drivers usually use “SELECT 1” query to ensure connection is alive.

Can we use similar approach?

Отправлено с iPhone

> 13 сент. 2020 г., в 13:26, Pavel Tupitsyn  написал(а):
> 
> Igniters,
> 
> There is a feature request for a thin client Ping operation on the user
> list [1].
> I think that is a good idea - IgniteClient.ping() will be a valuable
> addition.
> 
> Any objections?
> 
> [1]
> http://apache-ignite-users.70518.x6.nabble.com/Feature-request-method-to-test-active-connection-in-Ignite-thin-client-td33985.html

Re: [DISCUSSION] Output IgniteSystemProperties via ignite.sh

[Николай Ижиков]

Yes, of course.
I think we have to make IgniteSystemProperties enum and traverse it in runtime.

Отправлено с iPhone

> 21 авг. 2020 г., в 17:54, Zhenya Stanilovsky  
> написал(а):
> 
> 
> 
> Good catch, as for me, do you plan some autogeneration here?
>  
>>  
>>>  
 Hello, Igniters.
 
 For now, we have dozens of the `IgniteSystemProperties` [1] that can tweak 
 Ignite behaviour in the very wide limits.
 But, the issue, for the administrator is the following
 
 - Documentation about existing properties can be outdated.
 - The only place of the truth is the source code.
 - It’s hard to understand what flag is supported in what version.
 
 I propose to implement output of all available properties with it’s 
 descriptions in the `ignite.sh -X` command.
 
 Example of the JVM output:
 
 ```
 [16:25:49]~/src/ignite:[master]$ java -X
 
 -Xbatch disable background compilation
 -Xbootclasspath/a:
   append to end of bootstrap class path
 -Xcheck:jni perform additional checks for JNI functions
 -Xcomp forces compilation of methods on first invocation
 -Xdebug provided for backward compatibility
 -Xdiag show additional diagnostic messages
 -Xfuture enable strictest checks, anticipating future default
 -Xint interpreted mode execution only
 -Xinternalversion
   displays more detailed JVM version information than 
 the
 
 [16:28:45]~/src/ignite:[master]$ java -XX:+UnlockDiagnosticVMOptions 
 -XX:+PrintFlagsFinal -version
 
 [Global flags]
 ccstrlist AOTLibrary = {product} {default}
  bool AbortVMOnCompilationFailure = false {diagnostic} {default}
 ccstr AbortVMOnException = {diagnostic} {default}
 ccstr AbortVMOnExceptionMessage = {diagnostic} {default}
  bool AbortVMOnSafepointTimeout = false {diagnostic} {default}
  bool AbortVMOnVMOperationTimeout = false {diagnostic} {default}
  intx AbortVMOnVMOperationTimeoutDelay = 1000 {diagnostic} {default}
   int ActiveProcessorCount = -1 {product} {default}
 
 ```
 
 Example of the Ignite output:
 
 
> ignite.sh -X IGNITE_CONFIG_URL - System property to hold optional 
> configuration URL.
 IGNITE_SSH_HOST - System property to hold SSH host for visor-started nodes.
 IGNITE_MIN_BUFFERED_COMMUNICATION_MSG_CNT - [DEPRECATED] System property 
 to disable buffered communication if node sends less messages count than 
 specified by this property. Default value is {@code 512}.
 
 …
 
 ```
 
 WDYT?
 
 [1]  
 https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java#L56
  
>>>  
>>>  
>>>  
>>>  

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey.

With this API we are trying to fill the GAP Alexey pointed out at the start of 
this discussion.
I think it worth to be reviewed and merged.

Can we, please, come back to the discussion of the changes itself?
I think API changes should be discussed in the other thread.

> Why do you think this is a wrong usage pattern? From the top of my head, here 
> is a few cases of direct metric API usage that I know are currently being 
> used in production:
> * A custom task execution scheduling service with load balancing based on 
> utilization metrics readings from Java code
> * Cleanup task trigger based on metrics readings
> * A custom health-check endpoint for an application with an embedded Ignite 
> node for Kubernetes/Spring Application/etc

[1] https://github.com/apache/ignite/pull/7283

> 24 янв. 2020 г., в 18:08, Andrey Gura  написал(а):
> 
>> My point - that your contribution that took a long time, already, can’t be 
>> an argument to postpone creation of the API in the current release.
> 
> Argument is not about time. But about API which is known will be changed.
> Second argument: why we should add this experimental API to the
> already existing experimental API? Just to making API more
> experimental? As I told already it is commit for commit and doesn't
> bring any value but brings some inconvenience to me (e.g. merge
> problems).
> 
> BTW, does it exist issue about marking IEP-35 API's as experimental?
> 
> On Thu, Jan 23, 2020 at 8:33 PM Николай Ижиков  wrote:
>> 
>>> You want say didn't discuss?
>> 
>> Yes.
>> 
>>> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?
>> 
>> No, of course.
>> My point - that your contribution that took a long time, already, can’t be 
>> an argument to postpone creation of the API in the current release.
>> 
>> 
>>> 23 янв. 2020 г., в 18:11, Andrey Gura  написал(а):
>>> 
>>>> We don’t discuss your changes and your proposals for the Metric API.
>>> 
>>> You want say didn't discuss? Actually we did it [1] but you told ok
>>> let's see code :)
>>> 
>>>> So I don’t think we can make the existence of some PR is an argument to 
>>>> introduce(or not introduce) this facade.
>>> 
>>> You definitely right in case of competition development. But I think
>>> that collaborative development is more effective. Isn't it?
>>> 
>>>> Moreover, As far as I know, you developing changes for the Metric API for 
>>>> 5 months without public discussion, for now. Let's start it.
>>> 
>>> Firsty, with discussion. See above.
>>> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?
>>> 
>>>> Let’s do the following:
>>>> 1. Review `IgniteMetric` facade and introduce it to the users as an 
>>>> experimental API.
>>> 
>>> It just doesn't make sense. Just commit for commit.
>>> 
>>>> 2. Discuss your proposal to the Metric API in the separate thread you are 
>>>> referencing.
>>> 
>>> You a re welcome to thread [1] again. But please, take into account.
>>> because discussion is postponed by you it's late enough to discuss
>>> proposed solution. But of course I'll answer on your questions and
>>> will be glade to your comments and suggestions.
>>> 
>>>> I think we should start a discussion from the simplified explanation of:
>>>> 1. API intentions - What we want to gain with it? What is our focus?
>>>> 2. Simple, minimal example of API main interfaces and desired usages.
>>> 
>>> All this already described at [1]. You also can take a look on PR (see
>>> MetricSource implementations, there are complex and simple ones).
>>> 
>>> [1] 
>>> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-35-Metrics-management-in-Ignite-tp43243.html
>>> 
>>> On Thu, Jan 23, 2020 at 5:42 PM Николай Ижиков  wrote:
>>>> 
>>>> Andrey.
>>>> 
>>>>> IGNITE-11927 implementation so your changes are incompatible with my 
>>>>> changes
>>>> 
>>>> We don’t discuss your changes and your proposals for the Metric API.
>>>> So I don’t think we can make the existence of some PR is an argument to 
>>>> introduce(or not introduce) this facade.
>>>> Moreover, As far as I know, you developing changes for the Metric API for 
>>>> 5 months without public discussion, for now. Let's start it.
>>>> 
>>>> Let’s do the following:
>>>

Re: Internal classes are exposed in public API

[Николай Ижиков]

> You want say didn't discuss?

Yes.

> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?

No, of course.
My point - that your contribution that took a long time, already, can’t be an 
argument to postpone creation of the API in the current release.


> 23 янв. 2020 г., в 18:11, Andrey Gura  написал(а):
> 
>> We don’t discuss your changes and your proposals for the Metric API.
> 
> You want say didn't discuss? Actually we did it [1] but you told ok
> let's see code :)
> 
>> So I don’t think we can make the existence of some PR is an argument to 
>> introduce(or not introduce) this facade.
> 
> You definitely right in case of competition development. But I think
> that collaborative development is more effective. Isn't it?
> 
>> Moreover, As far as I know, you developing changes for the Metric API for 5 
>> months without public discussion, for now. Let's start it.
> 
> Firsty, with discussion. See above.
> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?
> 
>> Let’s do the following:
>> 1. Review `IgniteMetric` facade and introduce it to the users as an 
>> experimental API.
> 
> It just doesn't make sense. Just commit for commit.
> 
>> 2. Discuss your proposal to the Metric API in the separate thread you are 
>> referencing.
> 
> You a re welcome to thread [1] again. But please, take into account.
> because discussion is postponed by you it's late enough to discuss
> proposed solution. But of course I'll answer on your questions and
> will be glade to your comments and suggestions.
> 
>> I think we should start a discussion from the simplified explanation of:
>> 1. API intentions - What we want to gain with it? What is our focus?
>> 2. Simple, minimal example of API main interfaces and desired usages.
> 
> All this already described at [1]. You also can take a look on PR (see
> MetricSource implementations, there are complex and simple ones).
> 
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-35-Metrics-management-in-Ignite-tp43243.html
> 
> On Thu, Jan 23, 2020 at 5:42 PM Николай Ижиков  wrote:
>> 
>> Andrey.
>> 
>>> IGNITE-11927 implementation so your changes are incompatible with my changes
>> 
>> We don’t discuss your changes and your proposals for the Metric API.
>> So I don’t think we can make the existence of some PR is an argument to 
>> introduce(or not introduce) this facade.
>> Moreover, As far as I know, you developing changes for the Metric API for 5 
>> months without public discussion, for now. Let's start it.
>> 
>> Let’s do the following:
>> 
>> 1. Review `IgniteMetric` facade and introduce it to the users as an 
>> experimental API.
>> 
>> 2. Discuss your proposal to the Metric API in the separate thread you are 
>> referencing.
>> 
>> I think we should start a discussion from the simplified explanation of:
>> 
>> 1. API intentions - What we want to gain with it? What is our focus?
>> 2. Simple, minimal example of API main interfaces and desired usages.
>> 
>> This will help to the community and me personally better understand your 
>> idea and share feedback.
>> 
>> Thanks.
>> 
>>> 23 янв. 2020 г., в 17:15, Andrey Gura  написал(а):
>>> 
>>> Nikolay,
>>> 
>>> as I wrote early in this thread API significantly changed during
>>> IGNITE-11927 implementation so your changes are incompatible with my
>>> changes.
>>> 
>>> ReadOnlyMetricRegistry will be removed at all (still exists in a
>>> couple of places where it uses).
>>> 
>>> Also I don't want to introduce IgniteMetric facade in this rush. In
>>> current implementation this interface just provides access to the
>>> ReadOnlyMetricManager instance (which will be removed) but it is not
>>> enough.
>>> 
>>> I agree with Denis. We should mark current API as experimental and
>>> continue IEP-35 development. See my process proposal [1] described
>>> early this thread. We can release Apache Ignite 2.8.1 specially for
>>> this changes.
>>> Public APIs require deeper thinking in order to provide comprehensive,
>>> consistent and convenient way of metrics management for end users.
>>> 
>>> Let's add IgniteExperimental, release 2.8 and finish metrics related
>>> issues after it.
>>> 
>>> [1] 
>>> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html
>>> 
>>> 
>>> On Wed, Jan 22, 2020 

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey.

> IGNITE-11927 implementation so your changes are incompatible with my changes

We don’t discuss your changes and your proposals for the Metric API.
So I don’t think we can make the existence of some PR is an argument to 
introduce(or not introduce) this facade.
Moreover, As far as I know, you developing changes for the Metric API for 5 
months without public discussion, for now. Let's start it.

Let’s do the following:

1. Review `IgniteMetric` facade and introduce it to the users as an 
experimental API.

2. Discuss your proposal to the Metric API in the separate thread you are 
referencing.

I think we should start a discussion from the simplified explanation of:

1. API intentions - What we want to gain with it? What is our focus?
2. Simple, minimal example of API main interfaces and desired usages.

This will help to the community and me personally better understand your idea 
and share feedback.

Thanks.

> 23 янв. 2020 г., в 17:15, Andrey Gura  написал(а):
> 
> Nikolay,
> 
> as I wrote early in this thread API significantly changed during
> IGNITE-11927 implementation so your changes are incompatible with my
> changes.
> 
> ReadOnlyMetricRegistry will be removed at all (still exists in a
> couple of places where it uses).
> 
> Also I don't want to introduce IgniteMetric facade in this rush. In
> current implementation this interface just provides access to the
> ReadOnlyMetricManager instance (which will be removed) but it is not
> enough.
> 
> I agree with Denis. We should mark current API as experimental and
> continue IEP-35 development. See my process proposal [1] described
> early this thread. We can release Apache Ignite 2.8.1 specially for
> this changes.
> Public APIs require deeper thinking in order to provide comprehensive,
> consistent and convenient way of metrics management for end users.
> 
> Let's add IgniteExperimental, release 2.8 and finish metrics related
> issues after it.
> 
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html
> 
> 
> On Wed, Jan 22, 2020 at 5:21 PM Николай Ижиков  wrote:
>> 
>> Hello, Igniters.
>> 
>> * IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the 
>> master and cherry-picked to the 2.8.
>> So the main issue with the Metric API solved.
>> 
>> * I raised the PR [1] to fix second issue with the new Metric API: absence 
>> of the public Java API to get metrics.
>> This PR introduces the following changes:
>> 
>> 1. IgniteMetric interface created: it provides Java API to access Ignite 
>> metrics created with the new Metric API.
>> 
>> ```
>> public interface IgniteMetric extends Iterable {
>>@Nullable ReadOnlyMetricRegistry registry(String name);
>> }
>> ```
>> 
>> 2. All deprecation javadocs regarding metrics now reference to the public 
>> IgniteMetric instead of internal GridMetricManager:
>> 
>>> @deprecated Use {@link IgniteMetric} instead.
>> 
>> 3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
>> possible.
>> 
>> Please, review.
>> 
>> [1]  https://github.com/apache/ignite/pull/7283
>> 
>>> 21 янв. 2020 г., в 17:51, Николай Ижиков  
>>> написал(а):
>>> 
>>> Hello, Igniters.
>>> 
>>> Alexey approved my PR [1] regarding fixing public API for metric exporters.
>>> I’m waiting for a bot visa and merge this PR.
>>> 
>>> As we discussed, the metrics API will be marked with IgniteExperimental 
>>> annotation.
>>> 
>>> If anyone has any objection to this plan, please provide your feedback.
>>> 
>>> [1] https://github.com/apache/ignite/pull/7269
>>> 
>>>> 21 янв. 2020 г., в 13:45, Николай Ижиков  
>>>> написал(а):
>>>> 
>>>> Thanks, for the review Alexey.
>>>> 
>>>> I will fix your comment and  try to implement Monitoring facade, shortly.
>>>> 
>>>>> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
>>>>> написал(а):
>>>>> 
>>>>> Nikolay,
>>>>> 
>>>>> I left a single comment in the PR about the histogram metric. I think the
>>>>> API looks much cleaner now.
>>>>> 
>>>>> I will take care of the @IgniteExperimental annotation.
>>>>> 
>>>>> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
>>>>> 
>>>>>> Alexey.
>>>>>> 
>>>>>> PR [1] is waiting for your review.
>>>>>> Please, take a look.
>>

Re: Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Hello, Saikat.

Thank you so much for the review.

I answered your questions and resolve all the comments.
Please, take a look, one more time.

> 22 янв. 2020 г., в 07:58, Saikat Maitra  написал(а):
> 
> Hi Nikolay,
> 
> I have reviewed the PR and shared comments.
> 
> Please let me know if you have any feedback.
> 
> Regards,
> Saikat
> 
> On Mon, Jan 20, 2020 at 2:42 PM Николай Ижиков  wrote:
> 
>> Hello, Saikat.
>> 
>> Thanks, for feedback.
>> 
>> I raised a PR [1] to `ignite-extensions`.
>> 
>> You can find description of the new module below(examples can be found at
>> [2]):
>> 
>> Module provides the ability to integrate `Ignite` into you spring-boot
>> application with zero(or minimal) configuration.
>> 
>> After you add this module as a dependency to your spring-boot application
>> `Ignite` node will be configured and injected into `BeanFactory`.
>> 
>> Algorithm to configure `Ignite` is the following:
>>  1. If `IgniteConfiguration` bean exists in the `BeanFactory` it will be
>> used.
>>  2. If `IgniteConfiguration` bean doesn't exist following rules are
>> applied:
>>2.1. Default `Ignite` configuration created.
>>2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will be
>> used to customize `IgniteConfiguration`.
>> If a user wants to set custom SPI instances or similar hardcoded
>> values
>> one should do it with `IgniteConfigurer` implementation.
>>2.3  Application properties applied to `IgniteConfiguration`. Prefix
>> for the properties is `ignite`.
>> 
>> 
>> [1] https://github.com/apache/ignite-extensions/pull/6
>> [2] https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example
>> 
>> 
>>> 18 янв. 2020 г., в 06:44, Saikat Maitra 
>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> Thank you for your email. As part of Ignite Extensions migration we are
>> migrating Ignite Extensions to following repo.
>>> 
>>> https://github.com/apache/ignite-extensions
>>> 
>>> We have added flink and pub-sub modules and few additional modules are
>> open in PR.
>>> 
>>> You can refer to this PR to see how we are migrating the modules
>> https://github.com/apache/ignite-extensions/pull/5
>>> 
>>> I wanted to connect and discuss the changes to understand the spring
>> boot auto configure feature. We currently have an ignite spring module that
>> allows resource injection capabilities and provides a parser for Spring
>> based xml configuration files. Can you please review and share if the
>> changes you are proposing can be added as part of Ignite spring module or
>> it make sense to make it a separate spring boot auto configure module.
>>> 
>>> https://github.com/apache/ignite/tree/master/modules/spring
>>> 
>>> Regards,
>>> Saikat
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков 
>> wrote:
>>> Tests added.
>>> Please, review.
>>> 
>>> Saikat, can you help with this PR [1]?
>>> 
>>> I think it should be added as a separate module as we do with the flink
>> integration.
>>> Can you help me with it?
>>> Do we have some how-to for it?
>>> 
>>> [1] https://github.com/apache/ignite/pull/7237
>>> 
>>>> 16 янв. 2020 г., в 16:51, Николай Ижиков 
>> написал(а):
>>>> 
>>>> Hello, Denis.
>>>> 
>>>> Thanks, for the feedback.
>>>> 
>>>> Alexey, it seems, PR is ready to be reviewed, but I need some time(a
>> day or two) to write tests.
>>>> You can start with the core code review if you wish.
>>>> 
>>>> Here are autoconfigurer requirements:
>>>> 
>>>> 1. Start usage of Ignite with minimal(or zero) configuration.
>>>> 2. Configure Ignite configuration properties with the standard spring
>> boot application properties.
>>>> 3. Configure Ignite SPI implementation and so on that can’t be
>> configured via #2.
>>>> 
>>>> After some consultation with the Spring experts from the
>> community(Maxim Stepachev thanks for the idea)
>>>> I updated the PR with the logic described below:
>>>> 
>>>> 1. To enable Ignite auto-configuration user should add
>> `org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to dependencies.
>>>>   After it Ignite node wi

Re: Internal classes are exposed in public API

[Николай Ижиков]

Hello, Igniters.

* IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the master 
and cherry-picked to the 2.8.
So the main issue with the Metric API solved.

* I raised the PR [1] to fix second issue with the new Metric API: absence of 
the public Java API to get metrics.
This PR introduces the following changes:

1. IgniteMetric interface created: it provides Java API to access Ignite 
metrics created with the new Metric API.

```
public interface IgniteMetric extends Iterable {
@Nullable ReadOnlyMetricRegistry registry(String name);
}
```

2. All deprecation javadocs regarding metrics now reference to the public 
IgniteMetric instead of internal GridMetricManager:

  > @deprecated Use {@link IgniteMetric} instead.

3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
possible.

Please, review.

[1]  https://github.com/apache/ignite/pull/7283

> 21 янв. 2020 г., в 17:51, Николай Ижиков  написал(а):
> 
> Hello, Igniters.
> 
> Alexey approved my PR [1] regarding fixing public API for metric exporters.
> I’m waiting for a bot visa and merge this PR.
> 
> As we discussed, the metrics API will be marked with IgniteExperimental 
> annotation.
> 
> If anyone has any objection to this plan, please provide your feedback.
> 
> [1] https://github.com/apache/ignite/pull/7269
> 
>> 21 янв. 2020 г., в 13:45, Николай Ижиков  написал(а):
>> 
>> Thanks, for the review Alexey.
>> 
>> I will fix your comment and  try to implement Monitoring facade, shortly.
>> 
>>> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
>>> написал(а):
>>> 
>>> Nikolay,
>>> 
>>> I left a single comment in the PR about the histogram metric. I think the
>>> API looks much cleaner now.
>>> 
>>> I will take care of the @IgniteExperimental annotation.
>>> 
>>> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
>>> 
>>>> Alexey.
>>>> 
>>>> PR [1] is waiting for your review.
>>>> Please, take a look.
>>>> 
>>>> I think we should do the following before 2.8 release
>>>> 
>>>> * Introduce new @IgniteExperimental annotation as discussed.
>>>> * Mark Monitoring API with it.
>>>> * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
>>>> * merge «[IEP-35] public Java metric API» to 2.8
>>>> 
>>>> [1] https://github.com/apache/ignite/pull/7269
>>>> 
>>>>> 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
>>>> написал(а):
>>>>> 
>>>>> Nikolay,
>>>>> 
>>>>> Should we wait for both of the tickets given that we agreed that we are
>>>>> putting an experimental marker on the new APIs? I'm ok to fix only the
>>>>> first one and add the experimental marker so that we do not delay 2.8
>>>>> release.
>>>>> 
>>>>> пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
>>>>> 
>>>>>> Andrey.
>>>>>> 
>>>>>> Let’s move from the long letters to the code.
>>>>>> If you want to change API - please, propose this changes.
>>>>>> I think everybody wins if we make our API better.
>>>>>> 
>>>>>> Please, describe proposed changes.
>>>>>> It would be great if you have some examples or PR for it.
>>>>>> 
>>>>>> As for this release, I have plans to contribute tickets
>>>>>> «[IEP-35] Expose MetricRegistry to the public API» [1] and
>>>>>> «[IEP-35] public Java metric API» [2] for it.
>>>>>> 
>>>>>> Any objections on it?
>>>>>> 
>>>>>> [1] https://github.com/apache/ignite/pull/7269
>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-12553
>>>>>> 
>>>>>> 
>>>>>>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
>>>>>>> 
>>>>>>> It solves problem.
>>>>>>> 
>>>>>>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>>>>>>>  wrote:
>>>>>>>> 
>>>>>>>> After giving it some thought, I agree with Denis - there is nothing
>>>>>> wrong
>>>>>>>> with exposing the new APIs, we just need to make it clear that we are
>>>>>> still
>>>>>>>> going to change it.
>>>>>>>> 
>>>>>>>> Should we Introduce som

Re: Internal classes are exposed in public API

[Николай Ижиков]

Hello, Igniters.

Alexey approved my PR [1] regarding fixing public API for metric exporters.
I’m waiting for a bot visa and merge this PR.

As we discussed, the metrics API will be marked with IgniteExperimental 
annotation.

If anyone has any objection to this plan, please provide your feedback.

[1] https://github.com/apache/ignite/pull/7269

> 21 янв. 2020 г., в 13:45, Николай Ижиков  написал(а):
> 
> Thanks, for the review Alexey.
> 
> I will fix your comment and  try to implement Monitoring facade, shortly.
> 
>> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
>> написал(а):
>> 
>> Nikolay,
>> 
>> I left a single comment in the PR about the histogram metric. I think the
>> API looks much cleaner now.
>> 
>> I will take care of the @IgniteExperimental annotation.
>> 
>> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
>> 
>>> Alexey.
>>> 
>>> PR [1] is waiting for your review.
>>> Please, take a look.
>>> 
>>> I think we should do the following before 2.8 release
>>> 
>>> * Introduce new @IgniteExperimental annotation as discussed.
>>> * Mark Monitoring API with it.
>>> * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
>>> * merge «[IEP-35] public Java metric API» to 2.8
>>> 
>>> [1] https://github.com/apache/ignite/pull/7269
>>> 
>>>> 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
>>> написал(а):
>>>> 
>>>> Nikolay,
>>>> 
>>>> Should we wait for both of the tickets given that we agreed that we are
>>>> putting an experimental marker on the new APIs? I'm ok to fix only the
>>>> first one and add the experimental marker so that we do not delay 2.8
>>>> release.
>>>> 
>>>> пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
>>>> 
>>>>> Andrey.
>>>>> 
>>>>> Let’s move from the long letters to the code.
>>>>> If you want to change API - please, propose this changes.
>>>>> I think everybody wins if we make our API better.
>>>>> 
>>>>> Please, describe proposed changes.
>>>>> It would be great if you have some examples or PR for it.
>>>>> 
>>>>> As for this release, I have plans to contribute tickets
>>>>> «[IEP-35] Expose MetricRegistry to the public API» [1] and
>>>>> «[IEP-35] public Java metric API» [2] for it.
>>>>> 
>>>>> Any objections on it?
>>>>> 
>>>>> [1] https://github.com/apache/ignite/pull/7269
>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-12553
>>>>> 
>>>>> 
>>>>>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
>>>>>> 
>>>>>> It solves problem.
>>>>>> 
>>>>>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>>>>>>  wrote:
>>>>>>> 
>>>>>>> After giving it some thought, I agree with Denis - there is nothing
>>>>> wrong
>>>>>>> with exposing the new APIs, we just need to make it clear that we are
>>>>> still
>>>>>>> going to change it.
>>>>>>> 
>>>>>>> Should we Introduce something like @IgniteExperimental annotation (I
>>>>>>> believe it has been discussed a dozen of times on the dev-list)?
>>>>>>> 
>>>>>>> пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
>>>>>>> 
>>>>>>>> +1 to mark feature or whole release as EA.
>>>>>>>> 
>>>>>>>> пт, 17 янв. 2020 г., 23:00 Denis Magda :
>>>>>>>> 
>>>>>>>>> Folks, if you don't mind I'll share some thoughts/suggestions as an
>>>>>>>>> observer who was not involved in the feature development.
>>>>>>>>> 
>>>>>>>>> It's absolutely 'ok' to deprecate an API that is replaced with a
>>> much
>>>>>>>>> better version. However, we should do this only when the new APIs
>>> are
>>>>>>>>> production-ready. If there are still many limitations or open items
>>>>> then
>>>>>>>>> don't deprecate anything that exists and release the new APIs
>>>>> labeling as
>>>>>>>>> early access. What if release the improvements labeling 

Re: Internal classes are exposed in public API

[Николай Ижиков]

Thanks, for the review Alexey.

I will fix your comment and  try to implement Monitoring facade, shortly.

> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
> написал(а):
> 
> Nikolay,
> 
> I left a single comment in the PR about the histogram metric. I think the
> API looks much cleaner now.
> 
> I will take care of the @IgniteExperimental annotation.
> 
> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
> 
>> Alexey.
>> 
>> PR [1] is waiting for your review.
>> Please, take a look.
>> 
>> I think we should do the following before 2.8 release
>> 
>> * Introduce new @IgniteExperimental annotation as discussed.
>> * Mark Monitoring API with it.
>> * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
>> * merge «[IEP-35] public Java metric API» to 2.8
>> 
>> [1] https://github.com/apache/ignite/pull/7269
>> 
>>> 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
>> написал(а):
>>> 
>>> Nikolay,
>>> 
>>> Should we wait for both of the tickets given that we agreed that we are
>>> putting an experimental marker on the new APIs? I'm ok to fix only the
>>> first one and add the experimental marker so that we do not delay 2.8
>>> release.
>>> 
>>> пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
>>> 
>>>> Andrey.
>>>> 
>>>> Let’s move from the long letters to the code.
>>>> If you want to change API - please, propose this changes.
>>>> I think everybody wins if we make our API better.
>>>> 
>>>> Please, describe proposed changes.
>>>> It would be great if you have some examples or PR for it.
>>>> 
>>>> As for this release, I have plans to contribute tickets
>>>> «[IEP-35] Expose MetricRegistry to the public API» [1] and
>>>> «[IEP-35] public Java metric API» [2] for it.
>>>> 
>>>> Any objections on it?
>>>> 
>>>> [1] https://github.com/apache/ignite/pull/7269
>>>> [2] https://issues.apache.org/jira/browse/IGNITE-12553
>>>> 
>>>> 
>>>>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
>>>>> 
>>>>> It solves problem.
>>>>> 
>>>>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>>>>>  wrote:
>>>>>> 
>>>>>> After giving it some thought, I agree with Denis - there is nothing
>>>> wrong
>>>>>> with exposing the new APIs, we just need to make it clear that we are
>>>> still
>>>>>> going to change it.
>>>>>> 
>>>>>> Should we Introduce something like @IgniteExperimental annotation (I
>>>>>> believe it has been discussed a dozen of times on the dev-list)?
>>>>>> 
>>>>>> пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
>>>>>> 
>>>>>>> +1 to mark feature or whole release as EA.
>>>>>>> 
>>>>>>> пт, 17 янв. 2020 г., 23:00 Denis Magda :
>>>>>>> 
>>>>>>>> Folks, if you don't mind I'll share some thoughts/suggestions as an
>>>>>>>> observer who was not involved in the feature development.
>>>>>>>> 
>>>>>>>> It's absolutely 'ok' to deprecate an API that is replaced with a
>> much
>>>>>>>> better version. However, we should do this only when the new APIs
>> are
>>>>>>>> production-ready. If there are still many limitations or open items
>>>> then
>>>>>>>> don't deprecate anything that exists and release the new APIs
>>>> labeling as
>>>>>>>> early access. What if release the improvements labeling as EA
>> instead
>>>> of
>>>>>>>> hiding them completely?
>>>>>>>> 
>>>>>>>> I would also encourage us to put aside emotions, don't blame or
>> point
>>>>>>>> fingers. This IEP is a great initiative and you both have already
>>>> done a
>>>>>>>> tremendous job by developing, architecting and reviewing changes.
>>>> Just be
>>>>>>>> respectful. Nobody is trying to block the feature from being
>> released.
>>>>>>>> Everyone would be glad to tap into improvements and start using them
>>>> in
>>>>>>>> prod. But if more time is needed for the GA

Re: Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Hello, Saikat.

Thanks, for feedback.

I raised a PR [1] to `ignite-extensions`.

You can find description of the new module below(examples can be found at [2]):

Module provides the ability to integrate `Ignite` into you spring-boot 
application with zero(or minimal) configuration.

After you add this module as a dependency to your spring-boot application
`Ignite` node will be configured and injected into `BeanFactory`.

Algorithm to configure `Ignite` is the following:
  1. If `IgniteConfiguration` bean exists in the `BeanFactory` it will be used.
  2. If `IgniteConfiguration` bean doesn't exist following rules are applied:
2.1. Default `Ignite` configuration created.
2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will be used to 
customize `IgniteConfiguration`.
 If a user wants to set custom SPI instances or similar hardcoded values
 one should do it with `IgniteConfigurer` implementation.
2.3  Application properties applied to `IgniteConfiguration`. Prefix for 
the properties is `ignite`.


[1] https://github.com/apache/ignite-extensions/pull/6
[2] https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example


> 18 янв. 2020 г., в 06:44, Saikat Maitra  написал(а):
> 
> Hi Nikolay,
> 
> Thank you for your email. As part of Ignite Extensions migration we are 
> migrating Ignite Extensions to following repo.
> 
> https://github.com/apache/ignite-extensions
> 
> We have added flink and pub-sub modules and few additional modules are open 
> in PR.
> 
> You can refer to this PR to see how we are migrating the modules 
> https://github.com/apache/ignite-extensions/pull/5
> 
> I wanted to connect and discuss the changes to understand the spring boot 
> auto configure feature. We currently have an ignite spring module that allows 
> resource injection capabilities and provides a parser for Spring based xml 
> configuration files. Can you please review and share if the changes you are 
> proposing can be added as part of Ignite spring module or it make sense to 
> make it a separate spring boot auto configure module.
> 
> https://github.com/apache/ignite/tree/master/modules/spring
> 
> Regards,
> Saikat
> 
> 
> 
> 
> 
> 
> 
> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков  wrote:
> Tests added.
> Please, review.
> 
> Saikat, can you help with this PR [1]?
> 
> I think it should be added as a separate module as we do with the flink 
> integration.
> Can you help me with it?
> Do we have some how-to for it?
> 
> [1] https://github.com/apache/ignite/pull/7237
> 
> > 16 янв. 2020 г., в 16:51, Николай Ижиков  
> > написал(а):
> > 
> > Hello, Denis.
> > 
> > Thanks, for the feedback.
> > 
> > Alexey, it seems, PR is ready to be reviewed, but I need some time(a day or 
> > two) to write tests.
> > You can start with the core code review if you wish.
> > 
> > Here are autoconfigurer requirements:
> > 
> > 1. Start usage of Ignite with minimal(or zero) configuration.
> > 2. Configure Ignite configuration properties with the standard spring boot 
> > application properties.
> > 3. Configure Ignite SPI implementation and so on that can’t be configured 
> > via #2.
> > 
> > After some consultation with the Spring experts from the community(Maxim 
> > Stepachev thanks for the idea)
> > I updated the PR with the logic described below:
> > 
> > 1. To enable Ignite auto-configuration user should add 
> > `org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to dependencies.
> >After it Ignite node will be started during spring-boot application 
> > startup.
> > 
> > 2. IgniteConfiguration initialization logic: 
> > 
> > 2.1 If {@link IgniteConfiguration} bean exists in {@link BeanFactory} it 
> > will be used for the node start.
> > 2.2 If {@link IgniteConfiguration} bean doesn't exist following rules are 
> > applied:
> >  * Newly introducer IgniteConfigurer bean will be used to customize an 
> > empty IgniteConfiguration instance. 
> >If a user wants to set custom SPI instances or similar hardcoded values 
> > one should do it IgniteConfigurer implementation.
> > 
> >  * Application properties will override config values. Prefix for 
> > properties names is "ignite».
> > 
> > PS. Similar logic applied for the second module -  
> > `org.apache.ignite:spring-boot-ignite-client-autoconfigure:2.9.0`.
> > It provides the same features but for the autoconfiguration of the 
> > IgniteClient
> > 
> > 
> >> 15 янв. 2020 г., в 03:03, Denis Magda  написал(а):
> >> 
> >> Nikolay,
> >> 
> >> Thanks for c

Re: Internal classes are exposed in public API

[Николай Ижиков]

Alexey.

PR [1] is waiting for your review.
Please, take a look.

I think we should do the following before 2.8 release

* Introduce new @IgniteExperimental annotation as discussed.
* Mark Monitoring API with it.
* merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
* merge «[IEP-35] public Java metric API» to 2.8

[1] https://github.com/apache/ignite/pull/7269

> 20 янв. 2020 г., в 17:09, Alexey Goncharuk  
> написал(а):
> 
> Nikolay,
> 
> Should we wait for both of the tickets given that we agreed that we are
> putting an experimental marker on the new APIs? I'm ok to fix only the
> first one and add the experimental marker so that we do not delay 2.8
> release.
> 
> пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
> 
>> Andrey.
>> 
>> Let’s move from the long letters to the code.
>> If you want to change API - please, propose this changes.
>> I think everybody wins if we make our API better.
>> 
>> Please, describe proposed changes.
>> It would be great if you have some examples or PR for it.
>> 
>> As for this release, I have plans to contribute tickets
>> «[IEP-35] Expose MetricRegistry to the public API» [1] and
>> «[IEP-35] public Java metric API» [2] for it.
>> 
>> Any objections on it?
>> 
>> [1] https://github.com/apache/ignite/pull/7269
>> [2] https://issues.apache.org/jira/browse/IGNITE-12553
>> 
>> 
>>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
>>> 
>>> It solves problem.
>>> 
>>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>>>  wrote:
>>>> 
>>>> After giving it some thought, I agree with Denis - there is nothing
>> wrong
>>>> with exposing the new APIs, we just need to make it clear that we are
>> still
>>>> going to change it.
>>>> 
>>>> Should we Introduce something like @IgniteExperimental annotation (I
>>>> believe it has been discussed a dozen of times on the dev-list)?
>>>> 
>>>> пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
>>>> 
>>>>> +1 to mark feature or whole release as EA.
>>>>> 
>>>>> пт, 17 янв. 2020 г., 23:00 Denis Magda :
>>>>> 
>>>>>> Folks, if you don't mind I'll share some thoughts/suggestions as an
>>>>>> observer who was not involved in the feature development.
>>>>>> 
>>>>>> It's absolutely 'ok' to deprecate an API that is replaced with a much
>>>>>> better version. However, we should do this only when the new APIs are
>>>>>> production-ready. If there are still many limitations or open items
>> then
>>>>>> don't deprecate anything that exists and release the new APIs
>> labeling as
>>>>>> early access. What if release the improvements labeling as EA instead
>> of
>>>>>> hiding them completely?
>>>>>> 
>>>>>> I would also encourage us to put aside emotions, don't blame or point
>>>>>> fingers. This IEP is a great initiative and you both have already
>> done a
>>>>>> tremendous job by developing, architecting and reviewing changes.
>> Just be
>>>>>> respectful. Nobody is trying to block the feature from being released.
>>>>>> Everyone would be glad to tap into improvements and start using them
>> in
>>>>>> prod. But if more time is needed for the GA let's reiterate a bit.
>>>>>> 
>>>>>> -
>>>>>> Denis
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jan 17, 2020 at 12:24 PM Николай Ижиков 
>>>>>> wrote:
>>>>>> 
>>>>>>>> Also I agree with Alexey about introducing public IgniteMonitoring
>>>>>> facade
>>>>>>> 
>>>>>>> This is not an issue with the API.
>>>>>>> Just the new feature that doesn’t affects API.
>>>>>>> Moreover, I create a ticket to fix it, already.
>>>>>>> 
>>>>>>>> It's right. But if you add checking of statisticsEnabling property
>>>>> then
>>>>>>> test will fail. It's just not good tests.
>>>>>>> 
>>>>>>> My changes doesn’t affect any `staticticsEnabled` property.
>>>>>>> I don’ understand your point here.
>>>>>>> 
>>>>>>>> Redundant ReadOnlyMetricRegistry.
>>>>>>> 
>&g

Master compilation error

[Николай Ижиков]

Hello. Igniters.

Master build fails:

https://ci.ignite.apache.org/viewLog.html?buildId=4944107=IgniteTests24Java8_BuildApacheIgnite=buildLog_IgniteTests24Java8=pull%2F7269%2Fhead

[13:37:02][Step 3/4] [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project ignite-core: Compilation failure: Compilation 
failure: 
[13:37:02][Step 3/4] [ERROR] 
/opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[25,1]
 cannot find symbol
[13:37:02][Step 3/4] [ERROR]   symbol:   static 
IGNITE_BASELINE_AUTO_ADJUST_ENABLED
[13:37:02][Step 3/4] [ERROR]   location: class
[13:37:02][Step 3/4] [ERROR] 
/opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateAbstractTest.java:[162,28]
 cannot find symbol
[13:37:02][Step 3/4] [ERROR]   symbol:   variable 
IGNITE_BASELINE_AUTO_ADJUST_ENABLED
[13:37:02][Step 3/4] [ERROR]   location: class 
org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateAbstractTest
[13:37:02][Step 3/4] [ERROR] 
/opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[40,28]
 cannot find symbol
[13:37:02][Step 3/4] [ERROR]   symbol:   variable 
IGNITE_BASELINE_AUTO_ADJUST_ENABLED
[13:37:02][Step 3/4] [ERROR]   location: class 
org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
[13:37:02][Step 3/4] [ERROR] 
/opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[47,30]
 cannot find symbol
[13:37:02][Step 3/4] [ERROR]   symbol:   variable 
IGNITE_BASELINE_AUTO_ADJUST_ENABLED
[13:37:02][Step 3/4] [ERROR]   location: class 
org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey.

Let’s move from the long letters to the code.
If you want to change API - please, propose this changes.
I think everybody wins if we make our API better.

Please, describe proposed changes.
It would be great if you have some examples or PR for it.

As for this release, I have plans to contribute tickets 
«[IEP-35] Expose MetricRegistry to the public API» [1] and 
«[IEP-35] public Java metric API» [2] for it.

Any objections on it?

[1] https://github.com/apache/ignite/pull/7269
[2] https://issues.apache.org/jira/browse/IGNITE-12553


> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
> 
> It solves problem.
> 
> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>  wrote:
>> 
>> After giving it some thought, I agree with Denis - there is nothing wrong
>> with exposing the new APIs, we just need to make it clear that we are still
>> going to change it.
>> 
>> Should we Introduce something like @IgniteExperimental annotation (I
>> believe it has been discussed a dozen of times on the dev-list)?
>> 
>> пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
>> 
>>> +1 to mark feature or whole release as EA.
>>> 
>>> пт, 17 янв. 2020 г., 23:00 Denis Magda :
>>> 
>>>> Folks, if you don't mind I'll share some thoughts/suggestions as an
>>>> observer who was not involved in the feature development.
>>>> 
>>>> It's absolutely 'ok' to deprecate an API that is replaced with a much
>>>> better version. However, we should do this only when the new APIs are
>>>> production-ready. If there are still many limitations or open items then
>>>> don't deprecate anything that exists and release the new APIs labeling as
>>>> early access. What if release the improvements labeling as EA instead of
>>>> hiding them completely?
>>>> 
>>>> I would also encourage us to put aside emotions, don't blame or point
>>>> fingers. This IEP is a great initiative and you both have already done a
>>>> tremendous job by developing, architecting and reviewing changes. Just be
>>>> respectful. Nobody is trying to block the feature from being released.
>>>> Everyone would be glad to tap into improvements and start using them in
>>>> prod. But if more time is needed for the GA let's reiterate a bit.
>>>> 
>>>> -
>>>> Denis
>>>> 
>>>> 
>>>> On Fri, Jan 17, 2020 at 12:24 PM Николай Ижиков 
>>>> wrote:
>>>> 
>>>>>> Also I agree with Alexey about introducing public IgniteMonitoring
>>>> facade
>>>>> 
>>>>> This is not an issue with the API.
>>>>> Just the new feature that doesn’t affects API.
>>>>> Moreover, I create a ticket to fix it, already.
>>>>> 
>>>>>> It's right. But if you add checking of statisticsEnabling property
>>> then
>>>>> test will fail. It's just not good tests.
>>>>> 
>>>>> My changes doesn’t affect any `staticticsEnabled` property.
>>>>> I don’ understand your point here.
>>>>> 
>>>>>> Redundant ReadOnlyMetricRegistry.
>>>>> 
>>>>> It’s not redundant.
>>>>> It required for exporters and provide read only access to
>>> MetricRegistry
>>>>> existing in the node.
>>>>> 
>>>>> 
>>>>>> MetricExporterSpi that requires ReadOnlyMetricRegistry.
>>>>> 
>>>>>> Absence of newly created metrics in old MBeans that forces user use
>>>>>> exporter SPI while his code base uses old MBeans.
>>>>> 
>>>>> Why this is issue?
>>>>> 
>>>>>> Inconsistent MetricRegistry API.
>>>>> 
>>>>> It’s consistent.
>>>>> 
>>>>>> Metrics lookups from map instead of using direct reference
>>>>>> (performance problem).
>>>>> 
>>>>> 1. We(You and I) did this choice together to simplify creation of the
>>> new
>>>>> metrics.
>>>>> 2. This is not public API issue.
>>>>> 
>>>>> 
>>>>>> Ignoring of statisticsEnabled flag.
>>>>> 
>>>>> I don’t ignore this flag.
>>>>> It just doesn’t exists in new framework(because of scope).
>>>>> I don’t think it’s an issue.
>>>>> 
>>>>>> JmxExporterSpi creates beans that doesn't satisfy best MBeans
>>

Re: [DISCUSSION] API to KILL any user started computation

[Николай Ижиков]

Hello, Andrey.

Thanks for positive feedback.
Appreciate it.

> we can't cancel service or service's method

I understand it. 
Seems, we have several obvious options here:

* Try to do it and if fail answer to the user: «I’ve tried and fail. Sorry»
* Kill hanging thread.

> This invocations not register anywhere and can't be tracked and killed.

Yes.
So we should invent tracking for those invocation to be able to kill it :)


> 17 янв. 2020 г., в 21:03, Andrey Gura  написал(а):
> 
> It would be grate.
> 
> Only one comment: we can't cancel service or service's method
> execution because service grid API doesn't imply any requirement to
> service implementation. So if user do something like infinite cycle or
> blocking but non-interruptible operation it's impossible to interrupt
> it. Also service method invocation itself isn't cluster wide or local
> node specific operation. This invocations not register anywhere and
> can't be tracked and killed.
> 
> Unfortunately, the same is valid for some jobs in sense of infinite or
> non-interruptible operations.
> 
> On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков  wrote:
>> 
>> Hello, Igniters.
>> 
>> As you may know, we put a lot of effort to improve Ignite metric and 
>> diagnostic API.
>> We have created the following API:
>>* Metric manager
>>* System view manager
>> As far as I know, we would have tracing capabilities soon.
>> 
>> I think it's time to take the next step.
>> We should provide to the administrator the API to cancel(stop, kill) 
>> arbitrary user started computation.
>> 
>> Right now we have API to do it for:
>>* transactions `TransactionsMXBean#getActiveTransactions`.
>>* SQL queries: `KILL QUERY` sql command and visor task - 
>> `VisorQueryCancelTask`
>> 
>> Please, note, these features are implemented via different API.
>> 
>> I think we should introduce uniform Cancel API for the following 
>> computations:
>> 
>>* service.
>>* specific service method execution.
>>* compute job(compute task).
>>* query(scan, continuous, text).
>> 
>> We should  also rework kill transaction and kill SQL queries API to make 
>> them similar to each other.
>> I have plans to write an IEP for it and implement it.
>> What do you think?
>> 

Re: Internal classes are exposed in public API

[Николай Ижиков]

> Also I agree with Alexey about introducing public IgniteMonitoring facade

This is not an issue with the API.
Just the new feature that doesn’t affects API.
Moreover, I create a ticket to fix it, already.

> It's right. But if you add checking of statisticsEnabling property then test 
> will fail. It's just not good tests.

My changes doesn’t affect any `staticticsEnabled` property.
I don’ understand your point here.

> Redundant ReadOnlyMetricRegistry.

It’s not redundant.
It required for exporters and provide read only access to MetricRegistry 
existing in the node.


> MetricExporterSpi that requires ReadOnlyMetricRegistry.

> Absence of newly created metrics in old MBeans that forces user use
> exporter SPI while his code base uses old MBeans.

Why this is issue?

> Inconsistent MetricRegistry API.

It’s consistent.

> Metrics lookups from map instead of using direct reference
> (performance problem).

1. We(You and I) did this choice together to simplify creation of the new 
metrics.
2. This is not public API issue. 


> Ignoring of statisticsEnabled flag.

I don’t ignore this flag.
It just doesn’t exists in new framework(because of scope).
I don’t think it’s an issue.

> JmxExporterSpi creates beans that doesn't satisfy best MBeans practices.

Please, clarify your statement.
What is best MBeans practices you are talking about?

> Not finished IGNITE-11927


How this can be API issue?


> 17 янв. 2020 г., в 20:52, Andrey Gura  написал(а):
> 
>>> All issues Alexey mentioned in starting letter are fixed with my PR [1].
>>> I don’t think other issues you mentioned are blocker for the release.
> 
> As I mentioned already IGNITE-11927 is part of IEP-35 and
> implementation seriously affects API's. Also I agree with Alexey about
> introducing public IgniteMonitoring facade. So thiss PR doesn't fix
> all issues.
> 
>>> I talk about ignored existing functionality.
>> There is no existing tests that was broken by this contribution.
> 
> It's right. But if you add checking of statisticsEnabling property
> then test will fail. It's just not good tests.
> 
>> If you know the issues with it, feel free to create a ticket I will fix it 
>> ASAP.
> 
> I already fix it all in IGNITE-11927
> 
>>> 1. Moving IEP-35 API's to the internal package.
>> We should move the product forward and shouldn't hide major contribution 
>> from the user just because your second guess «I don’t like the API I just 
>> reviewed and approved».
> 
> We should move the product forward with with really finished features,
> not pieces of features.
> 
>> So I am against this proposal.
> 
> It is not argument.
> 
>> But, I’m ready to see your proposal for the API change and discuss them.
> 
> We can prepare it together. But we can't block release.
> 
>>> Because IGNITE-11927 doesn't solve all problems
>> What is *ALL* problems?
> 
> Redundant ReadOnlyMetricRegistry.
> MetricExporterSpi that requires ReadOnlyMetricRegistry.
> Absence of newly created metrics in old MBeans that forces user use
> exporter SPI while his code base uses old MBeans.
> Inconsistent MetricRegistry API.
> Metrics lookups from map instead of using direct reference
> (performance problem).
> Ignoring of statisticsEnabled flag.
> JmxExporterSpi creates beans that doesn't satisfy best MBeans practices.
> Not finished IGNITE-11927
> 
> It's enough I believe.
> 
> On Fri, Jan 17, 2020 at 7:28 PM Николай Ижиков  wrote:
>> 
>> Andrey.
>> 
>> All issues Alexey mentioned in starting letter are fixed with my PR [1].
>> I don’t think other issues you mentioned are blocker for the release.
>> 
>>> I talk about ignored existing functionality.
>> 
>> There is no existing tests that was broken by this contribution.
>> If you know the issues with it, feel free to create a ticket I will fix it 
>> ASAP.
>> 
>>> 1. Moving IEP-35 API's to the internal package.
>> 
>> We should move the product forward and shouldn't hide major contribution 
>> from the user just because your second guess «I don’t like the API I just 
>> reviewed and approved».
>> So I am against this proposal.
>> But, I’m ready to see your proposal for the API change and discuss them.
>> 
>>> Because IGNITE-11927 doesn't solve all problems
>> 
>> What is *ALL* problems?
>> Seems, we never be able to solve *ALL* problems.
>> But, we should move the product forward.
>> 
>> As for your steps [1-6].
>> I’m always following these steps during my contribution.
>> 
>> [1] https://github.com/apache/ignite/pull/7269
>> 
>>> 17 янв. 2020 г., в 19:08, Andrey Gura  написал(а):
>&

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey.

All issues Alexey mentioned in starting letter are fixed with my PR [1].
I don’t think other issues you mentioned are blocker for the release.

> I talk about ignored existing functionality.

There is no existing tests that was broken by this contribution.
If you know the issues with it, feel free to create a ticket I will fix it ASAP.

> 1. Moving IEP-35 API's to the internal package.

We should move the product forward and shouldn't hide major contribution from 
the user just because your second guess «I don’t like the API I just reviewed 
and approved».
So I am against this proposal.
But, I’m ready to see your proposal for the API change and discuss them.

> Because IGNITE-11927 doesn't solve all problems

What is *ALL* problems? 
Seems, we never be able to solve *ALL* problems.
But, we should move the product forward.

As for your steps [1-6].
I’m always following these steps during my contribution.

[1] https://github.com/apache/ignite/pull/7269

> 17 янв. 2020 г., в 19:08, Andrey Gura  написал(а):
> 
> The discussion is hot and can be endless. So in separate post I want
> propose my solution.
> 
> 1. Moving IEP-35 API's to the internal package.
> 2. Create special feature branch B.
> 3. In branch B will be merged IGNITE-11927
> 4. Because IGNITE-11927 doesn't solve all problems we should propose
> solution and implement it in branch B.
> 5. Testing, usability testing, fixing, etc iteratively.
> 6. Merge it to master and in new release branch if needed.
> 
> Independent step. There are some problem which should be fixed in 2.8
> before release otherwise we introduce problems with compatibility
> which will haunt us till next major release. I'll create tickets.
> 
> On Fri, Jan 17, 2020 at 7:03 PM Andrey Gura  wrote:
>> 
>>>> Because it is brand new API and it requires rewrite client code.
>>> We doesn’t break backward compatibility.
>>> The message is - this interface would be remove in the next major release.
>> 
>> We don't know anything about development processes our users. I can
>> admit that process could require that deprecated methods/APIs are not
>> allowed for example.
>> 
>>>> ReadOnlyMetricRegistry
>>>> Form user stand point it is very strange interface which don't give me any 
>>>> information about it’s purpose and responsibilities.
>>>> Seems, I should explain proposed changes [1] more clear:
>> 
>> I understand this. But I'm not Ignite user, I'm Ignite developer. The
>> key moment in my message *from user stand point*. From my point of
>> view it is very not intuitive solution and this interface is
>> redundant.
>> 
>>>> Actually not. We have statisticsEnabled for caches for example. There are 
>>>> other entities with such flag.
>>> They still works as expected.
>> 
>> Actually not. I fixed many such issues during IGNITE-11927 implementation.
>> 
>>>> Why do you decided do in such way?
>>> Because of the scope.
>>> The ability to disable/enable metrics is the matter of the other ticket.
>> 
>> I talk not about ability. I talk about ignored existing functionality.
>> So scope is not case here.
>> 
>>>> But they should not be exported by MetricExporterSpi implementations.
>>> Actually, it’s a responsibility of the exporter to decide.
>>> JMX exporter can exports ObjectMetric while OpenCensus exporter can’t.
>> 
>> Actually list is not metric at all as I already told.
>> 
>> On Fri, Jan 17, 2020 at 5:26 PM Николай Ижиков  wrote:
>>> 
>>> Andrey.
>>> 
>>>> Because it is brand new API and it requires rewrite client code.
>>> 
>>> We doesn’t break backward compatibility.
>>> The message is - this interface would be remove in the next major release.
>>> 
>>>> ReadOnlyMetricRegistry
>>>> Form user stand point it is very strange interface which don't give me any 
>>>> information about it’s purpose and responsibilities.
>>> 
>>> Seems, I should explain proposed changes [1] more clear:
>>> 
>>> Each SPI would have a `ReadOnlyMetricManager` which provides access to 
>>> collection of `ReadOnlyMetricRegistry`
>>> which has a collection of `Metric`.
>>> 
>>> So we reflects two-level structure we have in the internal API
>>> 
>>> GridMetricManager -> Collection[MetricRegistry] -> Collection[Metric]
>>> ReadOnlyMetricManager -> Collection[ReadOnlyMetricRegistry] -> 
>>> Collection[Metric]
>>> 
>>>> Actually not. We have statisticsEnabled for caches for example. Ther

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey.

> Because it is brand new API and it requires rewrite client code.

We doesn’t break backward compatibility.
The message is - this interface would be remove in the next major release.

> ReadOnlyMetricRegistry
> Form user stand point it is very strange interface which don't give me any 
> information about it’s purpose and responsibilities.

Seems, I should explain proposed changes [1] more clear:

Each SPI would have a `ReadOnlyMetricManager` which provides access to 
collection of `ReadOnlyMetricRegistry`
which has a collection of `Metric`.

So we reflects two-level structure we have in the internal API

GridMetricManager -> Collection[MetricRegistry] -> Collection[Metric]
ReadOnlyMetricManager -> Collection[ReadOnlyMetricRegistry] -> 
Collection[Metric]

> Actually not. We have statisticsEnabled for caches for example. There are 
> other entities with such flag.

They still works as expected.

> Why do you decided do in such way?

Because of the scope.
The ability to disable/enable metrics is the matter of the other ticket.

> But they should not be exported by MetricExporterSpi implementations.

Actually, it’s a responsibility of the exporter to decide.
JMX exporter can exports ObjectMetric while OpenCensus exporter can’t.

[1] 
https://github.com/apache/ignite/pull/7269/files#diff-0ae5657231fc4c1f650493b02190b81bR25

> 17 янв. 2020 г., в 16:57, Andrey Gura  написал(а):
> 
>> If I’m not missing something, you were one of the active reviewers of the 
>> Metric API.
> 
> Yes. But if I'm not missing some thing you were major developer of
> Metric API :) Shit happens. And it happened.
> 
>>> The first, I agree with Alexey about deprecation of APIs that are still 
>>> supported and don't offer reasonable substitution.
>> It has - MetricExporterSPI.
> 
> There is such concept - backward compatibility. I understand that
> deprecation of some interface doesn't break backward compatibility but
> it leads to question^ what should I use instead of this. And
> MetricExporterSpi is not answer for this question. Because it is brand
> new API and it requires rewrite client code.
> 
>>> ReadOnlyMetricRegistry interface is redundant.
>> It’s an interface that exposes internal MetricRegistry  to the exporters.
> 
> No it is not. It's completely artificial thing which allow iterate via
> all metric registries. GridMetricManager implements this interface
> while it is not metric registry. Form user stand point it is very
> strange interface which don't give me any information about it's
> purpose and responsibilities.
> 
>>> Exporters expose metrics if they are disabled.
>> We don’t have an ability to disable metrics.
> 
> Actually not. We have statisticsEnabled for caches for example. There
> are other entities with such flag.
> 
>> And that done, intentionally.
> 
> Why do you decided do in such way? Why you ignore existing
> functionality? It affects user expectations and experience.
> 
>> You are working on this issue, aren’t you?
> 
> Yes? I'm working. Unfortunately we are not synchronized in this
> context and I should redo all metrics related changes in order to
> merge it with my changes. Anyway, my change doesn't solve all problems
> (e.g. it doesn't introduce IgniteMonitoring facade).
> 
>> I can fix this issue, by myself.
> 
> Unfortunately it will be just fix. In my solution it is redesign. Stop
> fixing issues, let's do things. It requires deeper changes. My changes
> blocks AI 2.8 release because it big enough. So it retargeted on the
> next release. And it is one more reason for moving the changes to the
> internal packages. And it isn't good news for me because I will go
> throughout pan and tiers of merge. But it is right.
> 
>> Metrics of type lists are not metric at all.
> 
> They are created to deal with backward compatibility.
> 
>>> Metrics of type lists are not metric at all.
>> They are created to deal with backward compatibility.
> 
> Yes, I know. But they should not be exported by MetricExporterSpi
> implementations.
> 
> On Fri, Jan 17, 2020 at 3:37 PM Николай Ижиков  wrote:
>> 
>> Andrey, thanks for your opinion and your ownest critisism.
>> I can’t wait for your contribution.
>> 
>> If I’m not missing something, you were one of the active reviewers of the 
>> Metric API.
>> 
>>> The first, I agree with Alexey about deprecation of APIs that are still 
>>> supported and don't offer reasonable substitution.
>> 
>> It has - MetricExporterSPI.
>> 
>>> The second, from my point of view, we can't recommend MetricExporterSpi's 
>>> because it are still not-production ready.
>> 
>> It’s ready.
>

Re: Internal classes are exposed in public API

[Николай Ижиков]

Andrey, thanks for your opinion and your ownest critisism.
I can’t wait for your contribution.

If I’m not missing something, you were one of the active reviewers of the 
Metric API.

> The first, I agree with Alexey about deprecation of APIs that are still 
> supported and don't offer reasonable substitution.

It has - MetricExporterSPI.

> The second, from my point of view, we can't recommend MetricExporterSpi's 
> because it are still not-production ready.

It’s ready.

> The third, moving of MetricRegistry to the public API doesn't solve the 
> problem because this interface exposes internal Metric interface 
> implementations.

Not, its’ not.
Please, see `org.apache.ignite.spi.metric.LongMetric` and other public 
interface.

> API of MetricRegistry is inconsistent. 

MetricRegistry is the internal API.
Feel free to create ticket for an issues with it and I will try to fix it.

> ReadOnlyMetricRegistry interface is redundant.

It’s an interface that exposes internal MetricRegistry  to the exporters.

> Exporters expose metrics if they are disabled.

We don’t have an ability to disable metrics.
And that done, intentionally.
You are working on this issue, aren’t you?
I can fix this issue, by myself.

> Metrics of type lists are not metric at all.

They are created to deal with backward compatibility.

> 17 янв. 2020 г., в 15:09, Andrey Gura  написал(а):
> 
> Hi,
> 
> The first, I agree with Alexey about deprecation of APIs that are
> still supported and don't offer reasonable substitution.
> 
> The second, from my point of view, we can't recommend
> MetricExporterSpi's because it are still not-production ready. There
> are some issues with it and usage of ReadOnlyMetricRegistry interface
> just one of them.
> 
> The third, moving of MetricRegistry to the public API doesn't solve
> the problem because this interface exposes internal Metric interface
> implementations. So your PR is incomplete.
> Moreover, API of MetricRegistry is inconsistent. E.g. register(name,
> supplier, desc) method returns registered metric for some types and
> doesn't for other. register(metric) method is inconsistent in sense of
> metric naming. ReadOnlyMetricRegistry interface is redundant.
> MetricExporterSpi should be revised because it absolutely not
> intuitive because it requires ReadOnlyMetricRegistry and it's purpose
> is undefined.
> 
> One more point. IEP-35 is still not fully implemented. Some things are
> not taken into account. Exporters expose metrics if they are disabled.
> JMX beans exposes values that don't confirm to best practices [1].
> Metrics of type lists are not metric at all. Ubiquitous merics lookup
> from hash map instead of usage reference for getting metrics values
> (it is just performance issue). Also IGNITE-11927 is not implemented
> which also changes interfaces significantly.
> 
> Let's just admit that the implementation is immature and must be moved
> to the internal packages.
> 
> And because we already merged partially implemented IEP to the master
> branch we *must move all currently public APIs to the internal API*
> while it will not be ready for publication.
> 
> And the last but not least. What is happening indicates a immature
> development process which must be revised. I don't want discuss it in
> this thread but we must not allow merge of change to the master branch
> before it will completed, that is we must use feature branches for
> full IEP not only for particular tickets. And also we should
> reformulate IEP process in order to avoid things like this.
> 
> [1] 
> https://www.oracle.com/technetwork/java/javase/tech/best-practices-jsp-136021.html
> 
> On Fri, Jan 17, 2020 at 12:49 PM Николай Ижиков  wrote:
>> 
>> Alex.
>> 
>> OK, I may leverage your experience and create pure Java API.
>> Ticket [1] created.
>> 
>> But, personally, I don’t agree with you.
>> Ignite has dozens of the API that theoretically have a usage scenario, but 
>> in real-world have 0 custom implementation and usages.
>> Moreover, many APIs that were created with the intentions you mentioned is 
>> abandoned now and confuses users.
>> 
>> You can just see count of the tests we just mute on the TC.
>> 
>> Can you, please, take a look at the fix regarding puck API issue you 
>> mentioned in your first letter [2], [3]
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-12553
>> [2] https://github.com/apache/ignite/pull/7269
>> [3] https://issues.apache.org/jira/browse/IGNITE-12552
>> 
>> 
>>> 17 янв. 2020 г., в 12:12, Alexey Goncharuk  
>>> написал(а):
>>> 
>>> Nikolay,
>>> 
>>> Why do you think this is a wrong usage pattern? From the top of m

Re: Internal classes are exposed in public API

[Николай Ижиков]

Alex.

OK, I may leverage your experience and create pure Java API.
Ticket [1] created. 

But, personally, I don’t agree with you.
Ignite has dozens of the API that theoretically have a usage scenario, but in 
real-world have 0 custom implementation and usages.
Moreover, many APIs that were created with the intentions you mentioned is 
abandoned now and confuses users.

You can just see count of the tests we just mute on the TC.

Can you, please, take a look at the fix regarding puck API issue you mentioned 
in your first letter [2], [3] 

[1] https://issues.apache.org/jira/browse/IGNITE-12553
[2] https://github.com/apache/ignite/pull/7269
[3] https://issues.apache.org/jira/browse/IGNITE-12552


> 17 янв. 2020 г., в 12:12, Alexey Goncharuk  
> написал(а):
> 
> Nikolay,
> 
> Why do you think this is a wrong usage pattern? From the top of my head,
> here is a few cases of direct metric API usage that I know are currently
> being used in production:
> * A custom task execution scheduling service with load balancing based on
> utilization metrics readings from Java code
> * Cleanup task trigger based on metrics readings
> * A custom health-check endpoint for an application with an embedded
> Ignite node for Kubernetes/Spring Application/etc

Re: Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Tests added.
Please, review.

Saikat, can you help with this PR [1]?

I think it should be added as a separate module as we do with the flink 
integration.
Can you help me with it?
Do we have some how-to for it?

[1] https://github.com/apache/ignite/pull/7237

> 16 янв. 2020 г., в 16:51, Николай Ижиков  написал(а):
> 
> Hello, Denis.
> 
> Thanks, for the feedback.
> 
> Alexey, it seems, PR is ready to be reviewed, but I need some time(a day or 
> two) to write tests.
> You can start with the core code review if you wish.
> 
> Here are autoconfigurer requirements:
> 
> 1. Start usage of Ignite with minimal(or zero) configuration.
> 2. Configure Ignite configuration properties with the standard spring boot 
> application properties.
> 3. Configure Ignite SPI implementation and so on that can’t be configured via 
> #2.
> 
> After some consultation with the Spring experts from the community(Maxim 
> Stepachev thanks for the idea)
> I updated the PR with the logic described below:
> 
> 1. To enable Ignite auto-configuration user should add 
> `org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to dependencies.
>After it Ignite node will be started during spring-boot application 
> startup.
> 
> 2. IgniteConfiguration initialization logic: 
> 
> 2.1 If {@link IgniteConfiguration} bean exists in {@link BeanFactory} it will 
> be used for the node start.
> 2.2 If {@link IgniteConfiguration} bean doesn't exist following rules are 
> applied:
>  * Newly introducer IgniteConfigurer bean will be used to customize an empty 
> IgniteConfiguration instance. 
>If a user wants to set custom SPI instances or similar hardcoded values 
> one should do it IgniteConfigurer implementation.
> 
>  * Application properties will override config values. Prefix for properties 
> names is "ignite».
> 
> PS. Similar logic applied for the second module -  
> `org.apache.ignite:spring-boot-ignite-client-autoconfigure:2.9.0`.
> It provides the same features but for the autoconfiguration of the 
> IgniteClient
> 
> 
>> 15 янв. 2020 г., в 03:03, Denis Magda  написал(а):
>> 
>> Nikolay,
>> 
>> Thanks for contributing in this direction! That's one of the gaps on our
>> end and the user community will be certainly thankful once we fill it in.
>> 
>> *Alexey Kuznetsov*, as one of the Spring Boot experts, could you spend some
>> time reviewing the changes?
>> 
>> As for the extensions/modularization activities, please join Saikat in the
>> discussions ([1] and [2]). He is contributing the foundation and moving our
>> existing integrations to that new repository. The Spring Boot improvements
>> might be moved or, another option, we might add this class to the core?
>> 
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-12361-Migrate-Flume-module-to-ignite-extensions-td45010.html
>> [2]
>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td44064.html
>> 
>> -
>> Denis
>> 
>> 
>> On Sat, Jan 11, 2020 at 10:44 AM Николай Ижиков  wrote:
>> 
>>> Hello, Igniters.
>>> 
>>> During Ignite meetup I took part in there was a request from the users.
>>> They propose to create a custom spring boot autoconfigurer module for
>>> Ignite.
>>> This module should provide a smooth injection of Ignite to any spring-boot
>>> application.
>>> 
>>> I've implemented a tiny straightforward prototype of the module [1]
>>> Examples of the usage of integration can be found in the example
>>> application [2]
>>> 
>>> For now, the module provides the following features:
>>> 
>>> 1. Starts Ignite node and inject it in the spring ApplicationContext if
>>> bean of the type IgniteConfiguration exists in the context.
>>>   This can be achieved in two ways:
>>>   * create `IgniteConfiguration` from java code [3]
>>>   * add `ignite.xml` file to the application context [4]
>>> 
>>> 2. Starts IgniteClient instance and injects it int the spring Application
>>> if:
>>>   * ClientConfiguration bean exists in the context [5]
>>>   * `spring.data.ignite.clientAddresses` exists in the application
>>> properties. [6]
>>> 
>>> I have a following questions regards new module:
>>> 
>>>   1. We have an extension initiative so where is the right place for the
>>> new module?
>>>   2. Do we have spring experts in the community? What other features for
>>> this autoconfigurer module

Re: Internal classes are exposed in public API

[Николай Ижиков]

Pavel, 

As I said before - I don’t see any real use-cases for your example.

That’s why

> I don’t create pure Java API for metric intentionally.

What real-world use-case you are keeping in mind?
Why do you want to do it in that way?

The API you’re talking about is very simple and straightforward.
It’s an easy thing to create, but this will open to the user's wrong usage 
pattern.
From my point of view, to get metrics values one should use MetricExporterSPI.

> 17 янв. 2020 г., в 09:40, Pavel Tupitsyn  написал(а):
> 
> Николай, can you provide a full example - how do I get a metric value in my
> code as an Ignite user?
> 
> var ignite = Ignition.ignite();
> var totalAllocatedPages = ???;
> 
> On Fri, Jan 17, 2020 at 12:47 AM Николай Ижиков  wrote:
> 
>> Hello, Pavel.
>> 
>>> there should be an obvious public API to retrieve metrics
>> 
>> What do you mean by «retrieve»?
>> I don’t create pure Java API for metric intentionally.
>> We have MetricExporterSPI for this purpose.
>> It very simple from my point of view.
>> What use cases for `Ignite.monitoring()` API exists?
>> 
>> ```
>> public interface MetricExporterSpi extends IgniteSpi {
>>public void setMetricRegistry(ReadOnlyMetricRegistry registry);
>>public void setExportFilter(Predicate filter);
>> }
>> ```
>> 
>> ```
>> public interface ReadOnlyMetricRegistry extends Iterable {
>>public void addMetricRegistryCreationListener(Consumer
>> lsnr);
>>public void addMetricRegistryRemoveListener(Consumer
>> lsnr);
>> }
>> ```
>> 
>> ```
>> public class MetricRegistry implements Iterable {
>>public String name();
>>@Nullable public  M findMetric(String name);
>> }
>> ```
>> 
>>> 16 янв. 2020 г., в 20:32, Pavel Tupitsyn 
>> написал(а):
>>> 
>>> Agree with Alexey, there should be an obvious public API to retrieve
>> metrics
>>> 
>>> On Thu, Jan 16, 2020 at 8:23 PM Alexey Goncharuk <
>> alexey.goncha...@gmail.com>
>>> wrote:
>>> 
>>>> That's an option, I agree. Yet for me, from the usability point of
>> view, it
>>>> would be quite strange - to get an instance of the monitoring
>> interface, I
>>>> would need to do
>>>> JmxMetricExporterSpi metrics =
>>>> (JmxMetricExporterSpi)ignite.configuration().getMetricExporterSpi()[0];
>>>> which is too verbose and fragile if someone changes configuration (say,
>>>> inserts another SPI to the first position).
>>>> 
>>>> чт, 16 янв. 2020 г. в 20:14, Николай Ижиков :
>>>> 
>>>>> May be we should enable JMX exporter, by default?
>>>>> This would provide the same value for the user as `IgniteMonitoring`
>> you
>>>>> propose.
>>>>> 
>>>>>> 16 янв. 2020 г., в 20:06, Alexey Goncharuk <
>> alexey.goncha...@gmail.com
>>>>> 
>>>>> написал(а):
>>>>>> 
>>>>>> I think it would make sense to make something like a new
>>>> IgniteMonitoring
>>>>>> facade on Ignite interface and add an ability to obtain a metric value
>>>>>> through this facade, not through an exporter. This would be a
>>>>>> straightforward replacement for all old metrics interfaces.
>>>>>> 
>>>>>> чт, 16 янв. 2020 г. в 17:13, Николай Ижиков :
>>>>>> 
>>>>>>> Ticket to export MetricRegistry to the public API created -
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-12552
>>>>>>> 
>>>>>>>> 16 янв. 2020 г., в 16:58, Николай Ижиков 
>>>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Do you propose to keep these interfaces forever?
>>>>>>>> 
>>>>>>>>> 16 янв. 2020 г., в 16:52, Alexey Goncharuk <
>>>>> alexey.goncha...@gmail.com>
>>>>>>> написал(а):
>>>>>>>>> 
>>>>>>>>> Let's say I am upgrading from 2.7 to 2.8. Given that I figured out
>>>> to
>>>>>>>>> obtain an instance of the JMX exporter SPI, how should I map the
>> old
>>>>>>>>> 'interface + method name' to the new metric name? I think
>>>> deprecation
>>>>> of
>>>>>>>>> the public API may be a bit harsh in this case.
>>>>>>>>> 
>

Re: Internal classes are exposed in public API

[Николай Ижиков]

Hello, Pavel.

> there should be an obvious public API to retrieve metrics

What do you mean by «retrieve»?
I don’t create pure Java API for metric intentionally.
We have MetricExporterSPI for this purpose.
It very simple from my point of view.
What use cases for `Ignite.monitoring()` API exists?

```
public interface MetricExporterSpi extends IgniteSpi {
public void setMetricRegistry(ReadOnlyMetricRegistry registry);
public void setExportFilter(Predicate filter);
}
```

```
public interface ReadOnlyMetricRegistry extends Iterable {
public void addMetricRegistryCreationListener(Consumer 
lsnr);
public void addMetricRegistryRemoveListener(Consumer lsnr);
}
```

```
public class MetricRegistry implements Iterable {
public String name();
@Nullable public  M findMetric(String name);
}
```

> 16 янв. 2020 г., в 20:32, Pavel Tupitsyn  написал(а):
> 
> Agree with Alexey, there should be an obvious public API to retrieve metrics
> 
> On Thu, Jan 16, 2020 at 8:23 PM Alexey Goncharuk 
> wrote:
> 
>> That's an option, I agree. Yet for me, from the usability point of view, it
>> would be quite strange - to get an instance of the monitoring interface, I
>> would need to do
>> JmxMetricExporterSpi metrics =
>> (JmxMetricExporterSpi)ignite.configuration().getMetricExporterSpi()[0];
>> which is too verbose and fragile if someone changes configuration (say,
>> inserts another SPI to the first position).
>> 
>> чт, 16 янв. 2020 г. в 20:14, Николай Ижиков :
>> 
>>> May be we should enable JMX exporter, by default?
>>> This would provide the same value for the user as `IgniteMonitoring` you
>>> propose.
>>> 
>>>> 16 янв. 2020 г., в 20:06, Alexey Goncharuk >> 
>>> написал(а):
>>>> 
>>>> I think it would make sense to make something like a new
>> IgniteMonitoring
>>>> facade on Ignite interface and add an ability to obtain a metric value
>>>> through this facade, not through an exporter. This would be a
>>>> straightforward replacement for all old metrics interfaces.
>>>> 
>>>> чт, 16 янв. 2020 г. в 17:13, Николай Ижиков :
>>>> 
>>>>> Ticket to export MetricRegistry to the public API created -
>>>>> https://issues.apache.org/jira/browse/IGNITE-12552
>>>>> 
>>>>>> 16 янв. 2020 г., в 16:58, Николай Ижиков 
>>>>> написал(а):
>>>>>> 
>>>>>> Do you propose to keep these interfaces forever?
>>>>>> 
>>>>>>> 16 янв. 2020 г., в 16:52, Alexey Goncharuk <
>>> alexey.goncha...@gmail.com>
>>>>> написал(а):
>>>>>>> 
>>>>>>> Let's say I am upgrading from 2.7 to 2.8. Given that I figured out
>> to
>>>>>>> obtain an instance of the JMX exporter SPI, how should I map the old
>>>>>>> 'interface + method name' to the new metric name? I think
>> deprecation
>>> of
>>>>>>> the public API may be a bit harsh in this case.
>>>>>>> 
>>>>>>> чт, 16 янв. 2020 г. в 16:44, Николай Ижиков :
>>>>>>> 
>>>>>>>> Hello, Alexey.
>>>>>>>> 
>>>>>>>>> * DataRegionMetric public interface is deprecated, however, the
>>>>>>>> suggested replacement class GridMetricsManager is internal and
>> cannot
>>>>> be
>>>>>>>> acquired by a user. This makes impossible for the user to fix their
>>>>> code to
>>>>>>>> not use deprecated API
>>>>>>>> 
>>>>>>>> May be it’s not clear text here.
>>>>>>>> My point here, that user should obtain metrics values via
>> configured
>>>>>>>> metric exporters and don’t use *Metric interfaces.
>>>>>>>> 
>>>>>>>>> * In ReadOnlyMetricsRegistry there is a Consumer,
>>> but
>>>>>>>> MetricRegistry is again an internal class.
>>>>>>>> 
>>>>>>>> This is a real issue.
>>>>>>>> We should expose MetricRegistry interface to the public API and
>> keep
>>>>>>>> internal implementation.
>>>>>>>> 
>>>>>>>> I will create ticket and fix it, shortly.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 16 янв. 2020 г., в 16:39, Alexey Goncharuk <
>>>>> alexey.goncha...@gmail.com>
>>>>>>>> написал(а):
>>>>>>>>> 
>>>>>>>>> Igniters, Nikolay,
>>>>>>>>> 
>>>>>>>>> I was adding a new metric in the scope of the ticket [1] and was
>>>>>>>> surprised by a few things:
>>>>>>>>> * DataRegionMetric public interface is deprecated, however, the
>>>>>>>> suggested replacement class GridMetricsManager is internal and
>> cannot
>>>>> be
>>>>>>>> acquired by a user. This makes impossible for the user to fix their
>>>>> code to
>>>>>>>> not use deprecated API
>>>>>>>>> * In ReadOnlyMetricsRegistry there is a Consumer,
>>> but
>>>>>>>> MetricRegistry is again an internal class. This will cause the user
>>>>> code
>>>>>>>> compilation to break when MetricRegistry is changed
>>>>>>>>> 
>>>>>>>>> These things violate the public-private API separation and need to
>>> be
>>>>>>>> fixed prior 2.8 is released. Let's discuss what changes need to be
>>>>> made to
>>>>>>>> the API or perhaps move incomplete IEP-35 interfaces to the private
>>>>> package
>>>>>>>> and remove deprecations until the API is ready.
>>>>>>>>> 
>>>>>>>>> --AG
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: Internal classes are exposed in public API

[Николай Ижиков]

May be we should enable JMX exporter, by default?
This would provide the same value for the user as `IgniteMonitoring` you 
propose.

> 16 янв. 2020 г., в 20:06, Alexey Goncharuk  
> написал(а):
> 
> I think it would make sense to make something like a new IgniteMonitoring
> facade on Ignite interface and add an ability to obtain a metric value
> through this facade, not through an exporter. This would be a
> straightforward replacement for all old metrics interfaces.
> 
> чт, 16 янв. 2020 г. в 17:13, Николай Ижиков :
> 
>> Ticket to export MetricRegistry to the public API created -
>> https://issues.apache.org/jira/browse/IGNITE-12552
>> 
>>> 16 янв. 2020 г., в 16:58, Николай Ижиков 
>> написал(а):
>>> 
>>> Do you propose to keep these interfaces forever?
>>> 
>>>> 16 янв. 2020 г., в 16:52, Alexey Goncharuk 
>> написал(а):
>>>> 
>>>> Let's say I am upgrading from 2.7 to 2.8. Given that I figured out to
>>>> obtain an instance of the JMX exporter SPI, how should I map the old
>>>> 'interface + method name' to the new metric name? I think deprecation of
>>>> the public API may be a bit harsh in this case.
>>>> 
>>>> чт, 16 янв. 2020 г. в 16:44, Николай Ижиков :
>>>> 
>>>>> Hello, Alexey.
>>>>> 
>>>>>> * DataRegionMetric public interface is deprecated, however, the
>>>>> suggested replacement class GridMetricsManager is internal and cannot
>> be
>>>>> acquired by a user. This makes impossible for the user to fix their
>> code to
>>>>> not use deprecated API
>>>>> 
>>>>> May be it’s not clear text here.
>>>>> My point here, that user should obtain metrics values via configured
>>>>> metric exporters and don’t use *Metric interfaces.
>>>>> 
>>>>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>>>>> MetricRegistry is again an internal class.
>>>>> 
>>>>> This is a real issue.
>>>>> We should expose MetricRegistry interface to the public API and keep
>>>>> internal implementation.
>>>>> 
>>>>> I will create ticket and fix it, shortly.
>>>>> 
>>>>> 
>>>>>> 16 янв. 2020 г., в 16:39, Alexey Goncharuk <
>> alexey.goncha...@gmail.com>
>>>>> написал(а):
>>>>>> 
>>>>>> Igniters, Nikolay,
>>>>>> 
>>>>>> I was adding a new metric in the scope of the ticket [1] and was
>>>>> surprised by a few things:
>>>>>> * DataRegionMetric public interface is deprecated, however, the
>>>>> suggested replacement class GridMetricsManager is internal and cannot
>> be
>>>>> acquired by a user. This makes impossible for the user to fix their
>> code to
>>>>> not use deprecated API
>>>>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>>>>> MetricRegistry is again an internal class. This will cause the user
>> code
>>>>> compilation to break when MetricRegistry is changed
>>>>>> 
>>>>>> These things violate the public-private API separation and need to be
>>>>> fixed prior 2.8 is released. Let's discuss what changes need to be
>> made to
>>>>> the API or perhaps move incomplete IEP-35 interfaces to the private
>> package
>>>>> and remove deprecations until the API is ready.
>>>>>> 
>>>>>> --AG
>>>>> 
>>>>> 
>>> 
>> 
>> 

Re: Internal classes are exposed in public API

[Николай Ижиков]

Ticket to export MetricRegistry to the public API created - 
https://issues.apache.org/jira/browse/IGNITE-12552

> 16 янв. 2020 г., в 16:58, Николай Ижиков  написал(а):
> 
> Do you propose to keep these interfaces forever?
> 
>> 16 янв. 2020 г., в 16:52, Alexey Goncharuk  
>> написал(а):
>> 
>> Let's say I am upgrading from 2.7 to 2.8. Given that I figured out to
>> obtain an instance of the JMX exporter SPI, how should I map the old
>> 'interface + method name' to the new metric name? I think deprecation of
>> the public API may be a bit harsh in this case.
>> 
>> чт, 16 янв. 2020 г. в 16:44, Николай Ижиков :
>> 
>>> Hello, Alexey.
>>> 
>>>> * DataRegionMetric public interface is deprecated, however, the
>>> suggested replacement class GridMetricsManager is internal and cannot be
>>> acquired by a user. This makes impossible for the user to fix their code to
>>> not use deprecated API
>>> 
>>> May be it’s not clear text here.
>>> My point here, that user should obtain metrics values via configured
>>> metric exporters and don’t use *Metric interfaces.
>>> 
>>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>>> MetricRegistry is again an internal class.
>>> 
>>> This is a real issue.
>>> We should expose MetricRegistry interface to the public API and keep
>>> internal implementation.
>>> 
>>> I will create ticket and fix it, shortly.
>>> 
>>> 
>>>> 16 янв. 2020 г., в 16:39, Alexey Goncharuk 
>>> написал(а):
>>>> 
>>>> Igniters, Nikolay,
>>>> 
>>>> I was adding a new metric in the scope of the ticket [1] and was
>>> surprised by a few things:
>>>> * DataRegionMetric public interface is deprecated, however, the
>>> suggested replacement class GridMetricsManager is internal and cannot be
>>> acquired by a user. This makes impossible for the user to fix their code to
>>> not use deprecated API
>>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>>> MetricRegistry is again an internal class. This will cause the user code
>>> compilation to break when MetricRegistry is changed
>>>> 
>>>> These things violate the public-private API separation and need to be
>>> fixed prior 2.8 is released. Let's discuss what changes need to be made to
>>> the API or perhaps move incomplete IEP-35 interfaces to the private package
>>> and remove deprecations until the API is ready.
>>>> 
>>>> --AG
>>> 
>>> 
> 

Re: Internal classes are exposed in public API

[Николай Ижиков]

Do you propose to keep these interfaces forever?

> 16 янв. 2020 г., в 16:52, Alexey Goncharuk  
> написал(а):
> 
> Let's say I am upgrading from 2.7 to 2.8. Given that I figured out to
> obtain an instance of the JMX exporter SPI, how should I map the old
> 'interface + method name' to the new metric name? I think deprecation of
> the public API may be a bit harsh in this case.
> 
> чт, 16 янв. 2020 г. в 16:44, Николай Ижиков :
> 
>> Hello, Alexey.
>> 
>>> * DataRegionMetric public interface is deprecated, however, the
>> suggested replacement class GridMetricsManager is internal and cannot be
>> acquired by a user. This makes impossible for the user to fix their code to
>> not use deprecated API
>> 
>> May be it’s not clear text here.
>> My point here, that user should obtain metrics values via configured
>> metric exporters and don’t use *Metric interfaces.
>> 
>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>> MetricRegistry is again an internal class.
>> 
>> This is a real issue.
>> We should expose MetricRegistry interface to the public API and keep
>> internal implementation.
>> 
>> I will create ticket and fix it, shortly.
>> 
>> 
>>> 16 янв. 2020 г., в 16:39, Alexey Goncharuk 
>> написал(а):
>>> 
>>> Igniters, Nikolay,
>>> 
>>> I was adding a new metric in the scope of the ticket [1] and was
>> surprised by a few things:
>>> * DataRegionMetric public interface is deprecated, however, the
>> suggested replacement class GridMetricsManager is internal and cannot be
>> acquired by a user. This makes impossible for the user to fix their code to
>> not use deprecated API
>>> * In ReadOnlyMetricsRegistry there is a Consumer, but
>> MetricRegistry is again an internal class. This will cause the user code
>> compilation to break when MetricRegistry is changed
>>> 
>>> These things violate the public-private API separation and need to be
>> fixed prior 2.8 is released. Let's discuss what changes need to be made to
>> the API or perhaps move incomplete IEP-35 interfaces to the private package
>> and remove deprecations until the API is ready.
>>> 
>>> --AG
>> 
>> 

Re: Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Hello, Denis.

Thanks, for the feedback.

Alexey, it seems, PR is ready to be reviewed, but I need some time(a day or 
two) to write tests.
You can start with the core code review if you wish.

Here are autoconfigurer requirements:

1. Start usage of Ignite with minimal(or zero) configuration.
2. Configure Ignite configuration properties with the standard spring boot 
application properties.
3. Configure Ignite SPI implementation and so on that can’t be configured via 
#2.

After some consultation with the Spring experts from the community(Maxim 
Stepachev thanks for the idea)
I updated the PR with the logic described below:

1. To enable Ignite auto-configuration user should add 
`org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to dependencies.
After it Ignite node will be started during spring-boot application startup.

2. IgniteConfiguration initialization logic: 

2.1 If {@link IgniteConfiguration} bean exists in {@link BeanFactory} it will 
be used for the node start.
2.2 If {@link IgniteConfiguration} bean doesn't exist following rules are 
applied:
  * Newly introducer IgniteConfigurer bean will be used to customize an empty 
IgniteConfiguration instance. 
If a user wants to set custom SPI instances or similar hardcoded values one 
should do it IgniteConfigurer implementation.

  * Application properties will override config values. Prefix for properties 
names is "ignite».

PS. Similar logic applied for the second module -  
`org.apache.ignite:spring-boot-ignite-client-autoconfigure:2.9.0`.
It provides the same features but for the autoconfiguration of the IgniteClient


> 15 янв. 2020 г., в 03:03, Denis Magda  написал(а):
> 
> Nikolay,
> 
> Thanks for contributing in this direction! That's one of the gaps on our
> end and the user community will be certainly thankful once we fill it in.
> 
> *Alexey Kuznetsov*, as one of the Spring Boot experts, could you spend some
> time reviewing the changes?
> 
> As for the extensions/modularization activities, please join Saikat in the
> discussions ([1] and [2]). He is contributing the foundation and moving our
> existing integrations to that new repository. The Spring Boot improvements
> might be moved or, another option, we might add this class to the core?
> 
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-12361-Migrate-Flume-module-to-ignite-extensions-td45010.html
> [2]
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td44064.html
> 
> -
> Denis
> 
> 
> On Sat, Jan 11, 2020 at 10:44 AM Николай Ижиков  wrote:
> 
>> Hello, Igniters.
>> 
>> During Ignite meetup I took part in there was a request from the users.
>> They propose to create a custom spring boot autoconfigurer module for
>> Ignite.
>> This module should provide a smooth injection of Ignite to any spring-boot
>> application.
>> 
>> I've implemented a tiny straightforward prototype of the module [1]
>> Examples of the usage of integration can be found in the example
>> application [2]
>> 
>> For now, the module provides the following features:
>> 
>> 1. Starts Ignite node and inject it in the spring ApplicationContext if
>> bean of the type IgniteConfiguration exists in the context.
>>This can be achieved in two ways:
>>* create `IgniteConfiguration` from java code [3]
>>* add `ignite.xml` file to the application context [4]
>> 
>> 2. Starts IgniteClient instance and injects it int the spring Application
>> if:
>>* ClientConfiguration bean exists in the context [5]
>>* `spring.data.ignite.clientAddresses` exists in the application
>> properties. [6]
>> 
>> I have a following questions regards new module:
>> 
>>1. We have an extension initiative so where is the right place for the
>> new module?
>>2. Do we have spring experts in the community? What other features for
>> this autoconfigurer module required?
>> 
>> [1] https://github.com/apache/ignite/pull/7237/files
>> [2] https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example
>> [3]
>> https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/configfrombean
>> [4]
>> https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/configfromfile
>> [5]
>> https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/thinclientfrombean
>> [6]
>> https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/thinclientfromconfig
>> 
>> 

Re: Internal classes are exposed in public API

[Николай Ижиков]

Hello, Alexey.

>  * DataRegionMetric public interface is deprecated, however, the suggested 
> replacement class GridMetricsManager is internal and cannot be acquired by a 
> user. This makes impossible for the user to fix their code to not use 
> deprecated API

May be it’s not clear text here.
My point here, that user should obtain metrics values via configured metric 
exporters and don’t use *Metric interfaces.

>  * In ReadOnlyMetricsRegistry there is a Consumer, but 
> MetricRegistry is again an internal class. 

This is a real issue.
We should expose MetricRegistry interface to the public API and keep internal 
implementation.

I will create ticket and fix it, shortly.


> 16 янв. 2020 г., в 16:39, Alexey Goncharuk  
> написал(а):
> 
> Igniters, Nikolay,
> 
> I was adding a new metric in the scope of the ticket [1] and was surprised by 
> a few things:
>  * DataRegionMetric public interface is deprecated, however, the suggested 
> replacement class GridMetricsManager is internal and cannot be acquired by a 
> user. This makes impossible for the user to fix their code to not use 
> deprecated API
>  * In ReadOnlyMetricsRegistry there is a Consumer, but 
> MetricRegistry is again an internal class. This will cause the user code 
> compilation to break when MetricRegistry is changed
> 
> These things violate the public-private API separation and need to be fixed 
> prior 2.8 is released. Let's discuss what changes need to be made to the API 
> or perhaps move incomplete IEP-35 interfaces to the private package and 
> remove deprecations until the API is ready.
> 
> --AG

Re: [DISCUSSION] API to KILL any user started computation

[Николай Ижиков]

Alexey.

I think, yes.
We certainly should be able to use system view data for the new KILL API.

I think we should support both SQL and Java(JMX) API for this KILL command.


> 16 янв. 2020 г., в 15:28, Alexei Scherbakov  
> написал(а):
> 
> Nikolaj,
> 
> Can we use system views instead of implementing something new ?
> 
> Each user operation has an unique ID.
> 
> It's possible to introduce universal SQL kill something like:
> 
> kill transaction ID
> 
> where id is taken from system view.
> 
> 
> чт, 16 янв. 2020 г. в 14:19, Николай Ижиков :
> 
>> Hello, Alexey.
>> 
>> I’m talking about *administrator* API.
>> 
>> For example, the User has a cluster that is used by several applications.
>> Some application starts buggy compute tasks that consume all CPU resources.
>> Right now, administrator doesn’t have the ability to kill this task.
>> 
>> This can lead to instability of the whole cluster.
>> 
>> 
>>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com> написал(а):
>>> 
>>> Transactions can be also rolled back using tx.close where close is
>>> java.lang.AutoCloseable#close
>>> It looks for me to the definition of uniform cancel API.
>>> 
>>> 
>>> 
>>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com
>>>> :
>>> 
>>>> Nikolaj,
>>>> 
>>>> We already have cancellation possibilities for almost every user
>>>> computation.
>>>> Transactions are cancelled using tx.rollback()
>>>> Queries are cancelled using query.close()
>>>> Task is cancellable through ComputeTaskSession
>>>> 
>>>> What is uniform cancel API ? Why do we need it ?
>>>> 
>>>> 
>>>> 
>>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков :
>>>> 
>>>>> Hello, Igniters.
>>>>> 
>>>>> As you may know, we put a lot of effort to improve Ignite metric and
>>>>> diagnostic API.
>>>>> We have created the following API:
>>>>>   * Metric manager
>>>>>   * System view manager
>>>>> As far as I know, we would have tracing capabilities soon.
>>>>> 
>>>>> I think it's time to take the next step.
>>>>> We should provide to the administrator the API to cancel(stop, kill)
>>>>> arbitrary user started computation.
>>>>> 
>>>>> Right now we have API to do it for:
>>>>>   * transactions `TransactionsMXBean#getActiveTransactions`.
>>>>>   * SQL queries: `KILL QUERY` sql command and visor task -
>>>>> `VisorQueryCancelTask`
>>>>> 
>>>>> Please, note, these features are implemented via different API.
>>>>> 
>>>>> I think we should introduce uniform Cancel API for the following
>>>>> computations:
>>>>> 
>>>>>   * service.
>>>>>   * specific service method execution.
>>>>>   * compute job(compute task).
>>>>>   * query(scan, continuous, text).
>>>>> 
>>>>> We should  also rework kill transaction and kill SQL queries API to
>> make
>>>>> them similar to each other.
>>>>> I have plans to write an IEP for it and implement it.
>>>>> What do you think?
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> 
>>>> Best regards,
>>>> Alexei Scherbakov
>>>> 
>>> 
>>> 
>>> --
>>> 
>>> Best regards,
>>> Alexei Scherbakov
>> 
>> 
> 
> -- 
> 
> Best regards,
> Alexei Scherbakov

Re: [DISCUSSION] API to KILL any user started computation

[Николай Ижиков]

Hello, Alexey.

I’m talking about *administrator* API.

For example, the User has a cluster that is used by several applications.
Some application starts buggy compute tasks that consume all CPU resources.
Right now, administrator doesn’t have the ability to kill this task.

This can lead to instability of the whole cluster.


> 16 янв. 2020 г., в 13:42, Alexei Scherbakov  
> написал(а):
> 
> Transactions can be also rolled back using tx.close where close is
> java.lang.AutoCloseable#close
> It looks for me to the definition of uniform cancel API.
> 
> 
> 
> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov > :
> 
>> Nikolaj,
>> 
>> We already have cancellation possibilities for almost every user
>> computation.
>> Transactions are cancelled using tx.rollback()
>> Queries are cancelled using query.close()
>> Task is cancellable through ComputeTaskSession
>> 
>> What is uniform cancel API ? Why do we need it ?
>> 
>> 
>> 
>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков :
>> 
>>> Hello, Igniters.
>>> 
>>> As you may know, we put a lot of effort to improve Ignite metric and
>>> diagnostic API.
>>> We have created the following API:
>>>* Metric manager
>>>* System view manager
>>> As far as I know, we would have tracing capabilities soon.
>>> 
>>> I think it's time to take the next step.
>>> We should provide to the administrator the API to cancel(stop, kill)
>>> arbitrary user started computation.
>>> 
>>> Right now we have API to do it for:
>>>* transactions `TransactionsMXBean#getActiveTransactions`.
>>>* SQL queries: `KILL QUERY` sql command and visor task -
>>> `VisorQueryCancelTask`
>>> 
>>> Please, note, these features are implemented via different API.
>>> 
>>> I think we should introduce uniform Cancel API for the following
>>> computations:
>>> 
>>>* service.
>>>* specific service method execution.
>>>* compute job(compute task).
>>>* query(scan, continuous, text).
>>> 
>>> We should  also rework kill transaction and kill SQL queries API to make
>>> them similar to each other.
>>> I have plans to write an IEP for it and implement it.
>>> What do you think?
>>> 
>>> 
>> 
>> --
>> 
>> Best regards,
>> Alexei Scherbakov
>> 
> 
> 
> -- 
> 
> Best regards,
> Alexei Scherbakov

[DISCUSSION] API to KILL any user started computation

[Николай Ижиков]

Hello, Igniters.

As you may know, we put a lot of effort to improve Ignite metric and diagnostic 
API.
We have created the following API:
* Metric manager
* System view manager
As far as I know, we would have tracing capabilities soon.

I think it's time to take the next step.
We should provide to the administrator the API to cancel(stop, kill) arbitrary 
user started computation.

Right now we have API to do it for:
* transactions `TransactionsMXBean#getActiveTransactions`.
* SQL queries: `KILL QUERY` sql command and visor task - 
`VisorQueryCancelTask` 

Please, note, these features are implemented via different API.

I think we should introduce uniform Cancel API for the following computations:

* service.
* specific service method execution.
* compute job(compute task).
* query(scan, continuous, text).

We should  also rework kill transaction and kill SQL queries API to make them 
similar to each other.
I have plans to write an IEP for it and implement it. 
What do you think?

Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Hello, Igniters.

During Ignite meetup I took part in there was a request from the users.
They propose to create a custom spring boot autoconfigurer module for Ignite.
This module should provide a smooth injection of Ignite to any spring-boot 
application.

I've implemented a tiny straightforward prototype of the module [1] 
Examples of the usage of integration can be found in the example application 
[2] 

For now, the module provides the following features:

1. Starts Ignite node and inject it in the spring ApplicationContext if bean of 
the type IgniteConfiguration exists in the context.
This can be achieved in two ways:
* create `IgniteConfiguration` from java code [3] 
* add `ignite.xml` file to the application context [4] 

2. Starts IgniteClient instance and injects it int the spring Application if: 
* ClientConfiguration bean exists in the context [5] 
* `spring.data.ignite.clientAddresses` exists in the application 
properties. [6] 

I have a following questions regards new module:

1. We have an extension initiative so where is the right place for the new 
module?
2. Do we have spring experts in the community? What other features for this 
autoconfigurer module required?

[1] https://github.com/apache/ignite/pull/7237/files
[2] https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example
[3] 
https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/configfrombean
[4] 
https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/configfromfile
[5] 
https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/thinclientfrombean
[6] 
https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example/tree/master/src/main/java/org/apache/ignite/spring/boot/thinclientfromconfig

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Николай Ижиков]

Hello, Igniters.

I’m -1 to include the read-only patch to 2.8.
I think we shouldn’t accept any patches to 2.8 except bug fixes for blockers 
and major issues.

Guys, we don’t release Apache Ignite for 13 months!
We should focus on the release and make it ASAP.

We can’t extend the scope anymore.

> 10 янв. 2020 г., в 04:29, Sergey Antonov  
> написал(а):
> 
> Hello, Maxim!
> 
>> This PR [2] doesn't look a very simple +5,517 −2,038, 111 files
> changed.
> Yes, PR is huge, but I wrote a lot of new tests and reworked already
> presented. Changes in product code are minimal - only 30 changed files in
> /src/main/ part. And most of them are new control.sh commands and
> configuration.
> 
>> Do we have customer requests for this feature or maybe users who are
> waiting for exactly that ENUM values exactly in 2.8 release (not the 2.8.1
> for instance)?
> Can we introduce in new features in maintanance release (2.8.1)? Cluster
> read-only mode will be new feature, if we remove IgniteCluster#readOnly in
> 2.8 release. If all ok with that, lets remove  IgniteCluster#readOnly and
> move ticket [1] to 2.8.1 release.
> 
>> Do we have extended test results report (on just only TC.Bot green visa)
> on this feature to be sure that we will not add any blocker issues to the
> release?
> I'm preparing patch for 2.8 release and I will get new TC Bot visa vs
> release branch.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-12225
> 
> 
> 
> чт, 9 янв. 2020 г. в 19:38, Maxim Muzafarov :
> 
>> Folks,
>> 
>> 
>> Let me remind you that we are working on the 2.8 release branch
>> stabilization currently (please, keep it in mind).
>> 
>> 
>> Do we have a really STRONG reason for adding such a change [1] to the
>> ignite-2.8 branch? This PR [2] doesn't look a very simple +5,517
>> −2,038, 111 files changed.
>> Do we have customer requests for this feature or maybe users who are
>> waiting for exactly that ENUM values exactly in 2.8 release (not the
>> 2.8.1 for instance)?
>> Can we just simply remove IgniteCluster#readOnly to eliminate any
>> backward compatibility issues between 2.8 and 2.9 releases?
>> Do we have extended test results report (on just only TC.Bot green
>> visa) on this feature to be sure that we will not add any blocker
>> issues to the release? For instance, on pre-production environment.
>> 
>> I'd like to notice that we also have more than enough the release
>> blocker issues [3] which are still `in progress` and such a release
>> run becomes endless. Such changes without strong reasons looks too
>> scary for me a special after scope and code freeze dates.
>> 
>> Please, dispel my doubts.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-12225
>> [2] https://github.com/apache/ignite/pull/7194
>> [3]
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolvedissues(notrelatedtodocumentation)
>> 
>> On Thu, 9 Jan 2020 at 19:01, Alexey Zinoviev 
>> wrote:
>>> 
>>> +1
>>> 
>>> чт, 9 янв. 2020 г. в 18:52, Sergey Antonov :
>>> 
 +1
 
 I'm preparing patch for 2.8 branch now. TC Bot visa for 2.8 branch
>> will be
 at 13 Jan
 
 чт, 9 янв. 2020 г., 21:06 Ivan Pavlukhin :
 
> +1
> 
> чт, 9 янв. 2020 г. в 16:38, Ivan Rakov :
>> 
>> Maxim M. and anyone who is interested,
>> 
>> I suggest to include this fix to 2.8 release:
>> https://issues.apache.org/jira/browse/IGNITE-12225
>> Basically, it's a result of the following discussion:
>> 
> 
 
>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Single-point-in-API-for-changing-cluster-state-td43665.html
>> 
>> The fix affects public API: IgniteCluster#readOnly methods that
>> work
 with
>> boolean are replaced with ones that work with enum.
>> If we include it, we won't be obliged to keep deprecated boolean
 version
> of
>> API in the code (which is currently present in 2.8 branch) as it
>> wasn't
>> published in any release.
>> 
>> On Tue, Dec 31, 2019 at 3:54 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
>> wrote:
>> 
>>> Hello!
>>> 
>>> I have ran dependency checker plugin and quote the following:
>>> 
>>> One or more dependencies were identified with known
>> vulnerabilities
 in
>>> ignite-urideploy:
>>> One or more dependencies were identified with known
>> vulnerabilities
 in
>>> ignite-spring:
>>> One or more dependencies were identified with known
>> vulnerabilities
 in
>>> ignite-spring-data:
>>> One or more dependencies were identified with known
>> vulnerabilities
 in
>>> ignite-aop:
>>> One or more dependencies were identified with known
>> vulnerabilities
 in
>>> ignite-visor-console:
>>> 
>>> spring-core-4.3.18.RELEASE.jar
>>> (pkg:maven/org.springframework/spring-core@4.3.18.RELEASE,
>>> 
> 
>> cpe:2.3:a:pivotal_software:spring_framework:4.3.18.release:*:*:*:*:*:*:*,

Re: Visor plugin

[Николай Ижиков]

> 2. Move all colde from visorcmd to  controls.sh
> 3. Drop visorcm

Hello, Zhenya, Alexey, can you, please, clarify, why you suggest dropping of 
visorcmd?
What are the problems of the visorcmd that can’t be solved?


> 27 дек. 2019 г., в 09:26, Zhenya Stanilovsky  
> написал(а):
> 
> 
> +1 here.
> 
>  
>> Hi, All!
>> 
>> I think that the best way is to do the following:
>> 1. Move controls.sh to separate module (this will allow us to use any third
>> party libs for argument parsing, interactive mode, and other stuff).
>> 2. Move all colde from visorcmd to controls.sh
>> 3. Drop visorcmd
>> 
>> On Thu, Dec 26, 2019 at 11:29 PM Николай Ижиков < nizhi...@apache.org > 
>> wrote:
>>  
>>> +1 to keep visor cmd and fix it’s issues.
>>> 
>>>> 26 дек. 2019 г., в 19:23, Denis Magda < dma...@apache.org > написал(а):
>>>> 
>>>> Personally, I see no reason for deprecating Visor CLI and moving all its
>>>> capabilities to the control.sh script. It's better to merge the script's
>>>> capabilities into Visor CLI and rework the connectivity/communication
>>>> protocol of the latter. If to recall the reason for the control.sh
>>> creation
>>>> it was the Visor's daemon-mode way of interaction with the cluster that
>>> is
>>>> cumbersome and complicated.
>>>> 
>>>> -
>>>> Denis
>>>> 
>>>> 
>>>> On Thu, Dec 26, 2019 at 1:49 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com >
>>>> wrote:
>>>> 
>>>>> Hello!
>>>>> 
>>>>> I think that control.sh should be extended instead of Visor CLI, it is a
>>>>> tool which sees a lot more activity currently.
>>>>> 
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>> 
>>>>> 
>>>>> чт, 26 дек. 2019 г. в 12:45, Ivan Pavlukhin < vololo...@gmail.com >:
>>>>> 
>>>>>> Folks,
>>>>>> 
>>>>>> Do not we have plans to discontinue Visor? If so, it might be better
>>>>>> to add a desired functionality to another management API?
>>>>>> 
>>>>>> вт, 24 дек. 2019 г. в 08:02, Denis Magda < dma...@apache.org >:
>>>>>>> 
>>>>>>> Forwarding to the dev list. How exactly would you like to expand Visor
>>>>>> CMD?
>>>>>>> Please describe your idea and we can mover from that point.
>>>>>>> 
>>>>>>> Denis
>>>>>>> 
>>>>>>> On Monday, December 23, 2019, sgtech19 < rajado...@gmail.com > wrote:
>>>>>>> 
>>>>>>>> Hello team,
>>>>>>>> I would like to add a new feature to the existing
>>>>>> visor
>>>>>>>> commands. Could you give me some direction on how to achieve this if
>>>>>> its
>>>>>>>> possible? Do we need a visor plugin ? if so,please provide any
>>>>> example
>>>>>> of
>>>>>>>> this plugin .
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Sent from:  http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> -
>>>>>>> Denis
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Best regards,
>>>>>> Ivan Pavlukhin
>>>>>> 
>>>>> 
>>> 
>>> 
>> --
>> Alexey Kuznetsov
>>   
>  
>  
>  
>  

Re: Visor plugin

[Николай Ижиков]

+1 to keep visor cmd and fix it’s issues.

> 26 дек. 2019 г., в 19:23, Denis Magda  написал(а):
> 
> Personally, I see no reason for deprecating Visor CLI and moving all its
> capabilities to the control.sh script. It's better to merge the script's
> capabilities into Visor CLI and rework the connectivity/communication
> protocol of the latter. If to recall the reason for the control.sh creation
> it was the Visor's daemon-mode way of interaction with the cluster that is
> cumbersome and complicated.
> 
> -
> Denis
> 
> 
> On Thu, Dec 26, 2019 at 1:49 AM Ilya Kasnacheev 
> wrote:
> 
>> Hello!
>> 
>> I think that control.sh should be extended instead of Visor CLI, it is a
>> tool which sees a lot more activity currently.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> чт, 26 дек. 2019 г. в 12:45, Ivan Pavlukhin :
>> 
>>> Folks,
>>> 
>>> Do not we have plans to discontinue Visor? If so, it might be better
>>> to add a desired functionality to another management API?
>>> 
>>> вт, 24 дек. 2019 г. в 08:02, Denis Magda :
 
 Forwarding to the dev list. How exactly would you like to expand Visor
>>> CMD?
 Please describe your idea and we can mover from that point.
 
 Denis
 
 On Monday, December 23, 2019, sgtech19  wrote:
 
> Hello team,
> I would like to add a new feature to the existing
>>> visor
> commands. Could you give me some direction on how to achieve this if
>>> its
> possible? Do we need a visor plugin ? if so,please provide any
>> example
>>> of
> this plugin .
> 
> Thanks
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> 
 
 
 --
 -
 Denis
>>> 
>>> 
>>> 
>>> --
>>> Best regards,
>>> Ivan Pavlukhin
>>> 
>> 

Re: Discovery-based services deployment guarantees question

[Николай Ижиков]

Alexey, are you sure, you are testing new service framework?

Is yes - you definitely should file a bug. 

> 23 дек. 2019 г., в 17:02, Alexey Goncharuk  
> написал(а):
> 
> Igniters,
> 
> I have a question based on one of my recent tests debugging.
> 
> The test is related to Ignite services. I noticed that sometimes a proxy
> invocation of a newly deployed service fails because the service cannot be
> found. I managed to reduce the test to a simple "start two nodes, deploy a
> service, create a proxy, invoke the proxy" scenario. The proxy invocation
> fails in about ~80% of runs.
> 
> As far as I remember, the new discovery-based service deployment was
> supposed to be synchronous, so not only non-proxy service instances should
> work, but the proxies as well. Was my understanding correct? Should I file
> a bug for the observed behavior?
> 
> --AG

Re: Cache operations performance metrics

[Николай Ижиков]

> And if we have two metrics that are triggered for the same then one of them 
> is useless.

I don't understand what is the two metrics you are talking about.
I wrote about a single metric for a single cache operation.

> Obviously if you want know how fast or slow your business operation you must 
> measure latency of your business operation. What could be easier?

A business transaction includes work with several data sources, sending network 
requests, executing some remote services.
If it becomes slower then we should know - what basic API operations become 
slower.
So we have to measure  `PutTime` from Ignite, `InsertTime` from RDBMS and other 
parts of a transaction.

Ignite will provide this kind of value out of the box.
I think it’s useful values.

> User saw "cache put time" metric becomes x2 bigger. Does it become slower or 
> faster? Or we just put into the cache values that 4x bigger in size?

Ignite cache operations obviously becomes 2 times slower. 
*Why* they become slower is the question of an ongoing investigation.

I tried to look at other open-source products.
Here is an example of metrics provided by Apache Kafka [1] [2]

`request-latency-avg` - The average request latency in ms.
`records-lag-max` - The maximum lag in terms of number of records for any 
partition in this window. An increasing value over time is your best indication 
that the consumer group is not keeping up with the producers.
`fetch-latency-avg` - The average time taken for a fetch request.

It seems, they implement a similar approach to what I proposed.


[1] https://docs.confluent.io/current/kafka/monitoring.html#producer-metrics
[2] https://docs.confluent.io/current/kafka/monitoring.html#new-consumer-metrics

> 20 дек. 2019 г., в 15:53, Andrey Gura  написал(а):
> 
>> For example, the user saw «checkpoint time» metric becomes x2 bigger.
> 
> I just quote your words: " this is a trigger to make a deeper
> investigation". And if we have two metrics that are triggered for the
> same then one of them is useless.
> 
>> How it relates to business operations?
> 
> Why it should be related with business operation? It is concrete
> metrics for concrete process which can slowdown all operations in the
> system. Obviously if you want know how fast or slow your business
> operation you must measure latency of your business operation. What
> could be easier?
> 
>> Is it become slower or faster?
> 
> Very correct question! User saw "cache put time" metric becomes x2
> bigger. Does it become slower or faster? Or we just put into the cache
> values that 4x bigger in size? Or all time before we put values
> locally and now we put values on remote nodes. Or our operations
> execute in transaction and then time will depend on transaction type,
> actions in transaction and other transaction and actually will nothing
> talk about real cache operation. We have more questions then answers.
> 
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
> 
> Of course not :) See above.
> 
> On Fri, Dec 20, 2019 at 3:20 PM Николай Ижиков  wrote:
>> 
>>> It also will be visible on other metrics
>> 
>> How will it be visible?
>> 
>> For example, the user saw «checkpoint time» metric becomes x2 bigger.
>> How it relates to business operations? Is it become slower or faster?
>> What does it mean for an application performance?
>> 
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
>> 
>> *Why* it’s become slower - is the essence of «go deeper» investigation.
>> 
>>> 20 дек. 2019 г., в 15:07, Andrey Gura  написал(а):
>>> 
>>>> If a cache has some percent of the relatively slow transaction this is a 
>>>> trigger to make a deeper investigation.
>>> 
>>> It also will be visible on other metrics. So cache operations metrics
>>> still useless because it transitive values.
>>> 
>>>>> 1. Measure some important internals (WAL operations, checkpoint time, 
>>>>> etc) because it can talk about real problems.
>>> 
>>>> We already implement it.
>>> 
>>> I don't talk that it isn't implemented. It is just example of things
>>> that should be measured. All other metrics depends on internals.
>>> 
>>>>> 2. Measure business operations in user context, not cache API operations.
>>> 
>>>> Why do you think these approaches should exclude one another?
>>> 
>>> Because one of them is useless.
>>> 
>>> On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
&g

Re: Cache operations performance metrics

[Николай Ижиков]

> It also will be visible on other metrics

How will it be visible?

For example, the user saw «checkpoint time» metric becomes x2 bigger.
How it relates to business operations? Is it become slower or faster?
What does it mean for an application performance?

On the other hand - if `PuTime` increased - then we know for sure, all 
operation executing `put` becomes slower.

*Why* it’s become slower - is the essence of «go deeper» investigation.

> 20 дек. 2019 г., в 15:07, Andrey Gura  написал(а):
> 
>> If a cache has some percent of the relatively slow transaction this is a 
>> trigger to make a deeper investigation.
> 
> It also will be visible on other metrics. So cache operations metrics
> still useless because it transitive values.
> 
>>> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
>>> because it can talk about real problems.
> 
>> We already implement it.
> 
> I don't talk that it isn't implemented. It is just example of things
> that should be measured. All other metrics depends on internals.
> 
>>> 2. Measure business operations in user context, not cache API operations.
> 
>> Why do you think these approaches should exclude one another?
> 
> Because one of them is useless.
> 
> On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
>> 
>> Hello, Andrey.
>> 
>>> Where the sense in this value? I explained why this metrics are relatively 
>>> useless.
>> 
>> I don’t agree with you.
>> I believe they are not useless for a user.
>> And I try to explain why I think so.
>> 
>>> But user can't distinguish one transaction from another, so his knowledge 
>>> doesn't make sense definitely.
>> 
>> Users shouldn’t distinguish.
>> If a cache has some percent of the relatively slow transaction this is a 
>> trigger to make a deeper investigation.
>> 
>>> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
>>> because it can talk about real problems.
>> 
>> We already implement it.
>> What metrics are missing for internal processes?
>> 
>>> 2. Measure business operations in user context, not cache API operations.
>> 
>> Why do you think these approaches should exclude one another?
>> Users definitely should measure whole business transaction performance.
>> 
>> I think we should provide a way to measure part of the business transaction 
>> that relates to the Ignite.
>> 
>> 
>>> 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
>>> 
>>>> The goal of the proposed metrics is to measure whole cache operations 
>>>> behavior.
>>>> It provides some kind of statistics(histograms) for it.
>>> 
>>> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
>>> 
>>>> Yes, metrics will evaluate API call performance
>>> 
>>> And what? Where the sense in this value? I explained why this metrics
>>> are relatively useless.
>>> 
>>>> These are metrics of client-side operation performance.
>>> 
>>> Again. It's just a number without any sense.
>>> 
>>>> I think a specific user has knowledge - what are his transactions.
>>> 
>>> May be. But user can't distinguish one transaction from another, so
>>> his knowledge doesn't make sense definitely.
>>> 
>>>> From these metrics it can answer on the question «If my transaction 
>>>> includes cacheXXX, how long it usually takes?»
>>> 
>>> Actually not. The same caches can be involved  in a dozen of
>>> transactions and there are no ways to understand what transactions are
>>> slow or fast. It is useless.
>>> 
>>>> I disagree here.
>>>> If you have a better approach to measure cache operations performance - 
>>>> please, share your vision.
>>> 
>>> I already wrote about better approach. Two main points:
>>> 
>>> 1. Measure some important internals (WAL operations, checkpoint time,
>>> etc) because it can talk about real problems.
>>> 2. Measure business operations in user context, not cache API operations.
>>> 
>>> So  what we have? We have useless metrics that are doubled by useless
>>> histograms.
>>> 
>>> We should reconsider approach to metrics and performance measuring. It
>>> is hard and long task. There are no need to commit tons of useless
>>> metrics that just decrease performance.
>>> 
>>> Sorry for some sarcasm but I really believe in my opinion. 

Re: Cache operations performance metrics

[Николай Ижиков]

Hello, Andrey.

> Where the sense in this value? I explained why this metrics are relatively 
> useless.

I don’t agree with you.
I believe they are not useless for a user.
And I try to explain why I think so.

> But user can't distinguish one transaction from another, so his knowledge 
> doesn't make sense definitely.

Users shouldn’t distinguish.
If a cache has some percent of the relatively slow transaction this is a 
trigger to make a deeper investigation.

> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
> because it can talk about real problems.

We already implement it.
What metrics are missing for internal processes?

> 2. Measure business operations in user context, not cache API operations.

Why do you think these approaches should exclude one another?
Users definitely should measure whole business transaction performance.

I think we should provide a way to measure part of the business transaction 
that relates to the Ignite.


> 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
> 
>> The goal of the proposed metrics is to measure whole cache operations 
>> behavior.
>> It provides some kind of statistics(histograms) for it.
> 
> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
> 
>> Yes, metrics will evaluate API call performance
> 
> And what? Where the sense in this value? I explained why this metrics
> are relatively useless.
> 
>> These are metrics of client-side operation performance.
> 
> Again. It's just a number without any sense.
> 
>> I think a specific user has knowledge - what are his transactions.
> 
> May be. But user can't distinguish one transaction from another, so
> his knowledge doesn't make sense definitely.
> 
>> From these metrics it can answer on the question «If my transaction includes 
>> cacheXXX, how long it usually takes?»
> 
> Actually not. The same caches can be involved  in a dozen of
> transactions and there are no ways to understand what transactions are
> slow or fast. It is useless.
> 
>> I disagree here.
>> If you have a better approach to measure cache operations performance - 
>> please, share your vision.
> 
> I already wrote about better approach. Two main points:
> 
> 1. Measure some important internals (WAL operations, checkpoint time,
> etc) because it can talk about real problems.
> 2. Measure business operations in user context, not cache API operations.
> 
> So  what we have? We have useless metrics that are doubled by useless
> histograms.
> 
> We should reconsider approach to metrics and performance measuring. It
> is hard and long task. There are no need to commit tons of useless
> metrics that just decrease performance.
> 
> Sorry for some sarcasm but I really believe in my opinion. Metrics
> problem exists very very long time and existing metrics discussed many
> times. No one can explain this metrics to users because it requires
> too many additional knowledge about internals. And metric  value
> itself depends on many aspects of internals. It leads to impossibility
> of interpretation. And it's good time to remove it (in AI 3.0 due to a
> backward compatibility).
> 
> On Thu, Dec 19, 2019 at 9:09 PM Николай Ижиков  wrote:
>> 
>> Hello, Andrey.
>> 
>> The goal of the proposed metrics is to measure whole cache operations 
>> behavior.
>> It provides some kind of statistics(histograms) for it.
>> For more fine-grained analysis one will be use tracing or other «go deeper» 
>> tools.
>> 
>>>> Measured for API calls on the caller node side
>>> Values will the same only for cases when node is remote relative to data
>> 
>> Yes, metrics will evaluate API call performance.
>> I think this is the most valuable information from a user's point of view.
>> 
>> Regular user wants to know how fast his cache operation performs.
>> And these metrics provide the answer.
>> 
>>> For regular data node (server node) timing will depend on answers for 
>>> question:
>> 
>> I think these answers are always available.
>> I barely can imagine a scenario when one monitor «black box» cluster and 
>> don’t know it.
>> Even so, all answers are provided through system view we brought to the 
>> Ignite :)
>> 
>>> What is transaction commit or rollback time?
>> 
>> These are metrics of client-side operation performance.
>> 
>> I think a specific user has knowledge - what are his transactions.
>> From these metrics it can answer on the question «If my transaction includes 
>> cacheXXX, how long it usually takes?»
>> I think it’s very valuable knowledge.
>> 
>>> 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Николай Ижиков]

Good news, Anton!

Thanks for your work on PME feature!


> 19 дек. 2019 г., в 21:00, Anton Vinogradov  написал(а):
> 
> Folks,
> "2.8 + cherrypicked pme-free switch" vs "2.8" check finished, no blockers
> found.
> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7165%2Fhead=Latest=pull%2F7102%2Fhead
> Any objections to merging?
> 
> On Thu, Dec 19, 2019 at 4:20 PM Вячеслав Коптилин 
> wrote:
> 
>> Hello Pavel,
>> 
>> Good to hear from you!
>> 
>>> I guess we should have a system property to have ability to turn off PME
>> free switch behavior if something goes wrong after release.
>>> After feature battle testing we can remove it in the next release.
>> Sounds good to me.
>> 
>> Thanks,
>> S.
>> 
>> чт, 19 дек. 2019 г. в 15:59, Pavel Kovalenko :
>> 
>>> Anton, Slava
>>> 
>>> I guess we should have a system property to have ability to turn off PME
>>> free switch behavior if something goes wrong after release.
>>> After feature battle testing we can remove it in the next release.
>>> 
>>> чт, 19 дек. 2019 г. в 15:26, Anton Vinogradov :
>>> 
 Slava,
 
>> It would be nice to cut off a new branch from the release and run
>> all
 the
>> tests with an integrated feature and after that,
>> if you don’t see new failures and the release engineer agrees with
>> the
>> result, then and only then this feature can be merged into the
>>> release.
 I fully agree and see no other way to perform this.
 PR already created and testing check scheduled.
 
 But, Maxim's proposal was to perform checks at the master branch
 
>> 1. Merge the issue to the master branch.
>> 2. Wait for two-three weeks of running tests.
>> 3. Check that there are not flaky failures and fix them all
>> otherwise.
>> 4. Cherry-pick commit to the ignite-2.8 branch.
 
 and that's the point of discussion.
 
 On Thu, Dec 19, 2019 at 3:14 PM Вячеслав Коптилин <
 slava.kopti...@gmail.com>
 wrote:
 
> Hi,
> 
>> We're releasing release branch, not master... why we're checking
>> the
> "wrong" branch? :)
>> Performing the release verification we're checking not only how
 features
> work, but also how they work together.
> Yep, I agree that we should do verification for both branches of
>> corse.
> 
>> Finally, my concerns mostly related to integration checks and fail
> hidings, not feature checks.
>> That's why I propose to start integration checks asap.
> IMHO, that is not the right way in order to handle such cases.
> Cherry-picking new features into the release branch directly without
>>> the
> integration branch is not about STABILITY (this is normal and
>>> acceptable
 to
> cherry-pick obvious/trivial fixes into the release branch, though).
> It would be nice to cut off a new branch from the release and run all
>>> the
> tests with an integrated feature and after that,
> if you don’t see new failures and the release engineer agrees with
>> the
> result, then and only then this feature can be merged into the
>> release.
> 
> This is my own view of the matter, and I do not insist that you
>> should
 act
> in this way, but it seems to me that it would be safer and more
>>> correct.
> 
> Thanks,
> S.
> 
> чт, 19 дек. 2019 г. в 14:20, Anton Vinogradov :
> 
>> Feature already tested at the feature branch properly.
>> Question is about master -> release merge.
>> 
>> So:
>> 1) Testing at master does not equal to testing at release.
>> Code may fail in the master while it works at the release branch
>> and
 vice
>> versa.
>> 
>> 2) Master is flakier that release branch, so we able to miss
>> problem
> hidden
>> by other failures.
>> 
>> Summarizing,
>> We're releasing release branch, not master... why we're checking
>> the
>> "wrong" branch? :)
>> Performing the release verification we're checking not only how
 features
>> work, but also how they work together.
>> We may have different race condition windows at different branched.
>> The bug may never happen at master, but happen each 50th check at
>>> 2.8,
> and
>> in case we'll delay feature merge and check 2.8 only 49 times
>> (while
>>> we
>> have 150+ successful checks at master) we gain false-negative case
>>> for
> 2.8.
>> 
>> Finally, my concerns mostly related to integration checks and fail
> hidings,
>> not feature checks.
>> That's why I propose to start integration checks asap.
>> 
>> On Thu, Dec 19, 2019 at 1:21 PM Вячеслав Коптилин <
>> slava.kopti...@gmail.com>
>> wrote:
>> 
>>> Hello Anton,
>>> 
 We always have to merge all release features asap to have as
>> much
> time
>> as
>>> possible to fix all bugs.
>>> Could you please clarify this? What is the reason for that asap
> 

Re: Cache operations performance metrics

[Николай Ижиков]

lay, could you take a look, please?
>> 
>> If you do not mind, I will try to add affinity-nodes cache metrics:
>>>> * histograms that measure the time of processing `get`, `put`, `remove`, 
>>>> `commit`, `rollback` messages on affinity nodes(primary and backups). 
>>>> Ticket doesn't exist for it.
>> 
>> I have filed a ticket for it. [3]
>> 
>> [1] https://github.com/apache/ignite/pull/7141
>> [2] https://issues.apache.org/jira/browse/IGNITE-12450
>> [3] https://issues.apache.org/jira/browse/IGNITE-12453
>> 
>> пн, 16 дек. 2019 г. в 11:07, Alexei Scherbakov 
>> :
>>> 
>>> I think they are very useful.
>>> 
>>> пн, 16 дек. 2019 г. в 10:51, Николай Ижиков :
>>> 
>>>> Hello, Alexei.
>>>> 
>>>> Thanks for the link on the ticket, lableled it with the IEP-35 label.
>>>> What do you think about proposed metrics set?
>>>> 
>>>>> 16 дек. 2019 г., в 10:29, Alexei Scherbakov <
>>>> alexey.scherbak...@gmail.com> написал(а):
>>>>> 
>>>>> Nikolay,
>>>>> 
>>>>> What about batch operations?
>>>>> 
>>>>> For messages processing the ticket does exist and even has an
>>>>> implementation from before new metrics API times [1]
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-10418
>>>>> 
>>>>> пн, 16 дек. 2019 г. в 10:12, Николай Ижиков :
>>>>> 
>>>>>> Hello, Igniters.
>>>>>> 
>>>>>> I want to provide the user answers to the following question: "How cache
>>>>>> API operations perform?"
>>>>>> It seems, we need to implements metrics for basic cache API operations
>>>>>> like get, put, remove for it.
>>>>>> 
>>>>>> I think we should provide the following metrics:
>>>>>> 
>>>>>> * `get`, `put`, `remove` time histograms. Measured for API calls on the
>>>>>> caller node side.
>>>>>>   Implemented in [1], commit [2].
>>>>>> 
>>>>>> * `commit`, `rollback` time histograms. Measured for API calls on the
>>>>>> caller node side [3].
>>>>>> 
>>>>>> * histograms that measure the time of processing `get`, `put`, `remove`,
>>>>>> `commit`, `rollback` messages on affinity nodes(primary and backups).
>>>>>>   Ticket doesn't exist for it.
>>>>>> 
>>>>>> What do you think?
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12219
>>>>>> [2]
>>>>>> 
>>>> https://github.com/apache/ignite/commit/e66bbef97b2cef73a533ce8a506ec479852cb364
>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-12450
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Best regards,
>>>>> Alexei Scherbakov
>>>> 
>>>> 
>>> 
>>> --
>>> 
>>> Best regards,
>>> Alexei Scherbakov
>> 
>> 
>> 
>> --
>> Best wishes,
>> Amelchev Nikita

Re: Cache operations performance metrics

[Николай Ижиков]

Hello, Alexei.

Thanks for the link on the ticket, lableled it with the IEP-35 label.
What do you think about proposed metrics set?

> 16 дек. 2019 г., в 10:29, Alexei Scherbakov  
> написал(а):
> 
> Nikolay,
> 
> What about batch operations?
> 
> For messages processing the ticket does exist and even has an
> implementation from before new metrics API times [1]
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-10418
> 
> пн, 16 дек. 2019 г. в 10:12, Николай Ижиков :
> 
>> Hello, Igniters.
>> 
>> I want to provide the user answers to the following question: "How cache
>> API operations perform?"
>> It seems, we need to implements metrics for basic cache API operations
>> like get, put, remove for it.
>> 
>> I think we should provide the following metrics:
>> 
>> * `get`, `put`, `remove` time histograms. Measured for API calls on the
>> caller node side.
>>Implemented in [1], commit [2].
>> 
>> * `commit`, `rollback` time histograms. Measured for API calls on the
>> caller node side [3].
>> 
>> * histograms that measure the time of processing `get`, `put`, `remove`,
>> `commit`, `rollback` messages on affinity nodes(primary and backups).
>>Ticket doesn't exist for it.
>> 
>> What do you think?
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-12219
>> [2]
>> https://github.com/apache/ignite/commit/e66bbef97b2cef73a533ce8a506ec479852cb364
>> [3] https://issues.apache.org/jira/browse/IGNITE-12450
>> 
> 
> 
> -- 
> 
> Best regards,
> Alexei Scherbakov

Cache operations performance metrics

[Николай Ижиков]

Hello, Igniters.

I want to provide the user answers to the following question: "How cache API 
operations perform?"
It seems, we need to implements metrics for basic cache API operations like 
get, put, remove for it. 

I think we should provide the following metrics:

* `get`, `put`, `remove` time histograms. Measured for API calls on the caller 
node side.
Implemented in [1], commit [2].

* `commit`, `rollback` time histograms. Measured for API calls on the caller 
node side [3].

* histograms that measure the time of processing `get`, `put`, `remove`, 
`commit`, `rollback` messages on affinity nodes(primary and backups).
Ticket doesn't exist for it. 

What do you think?

[1] https://issues.apache.org/jira/browse/IGNITE-12219
[2] 
https://github.com/apache/ignite/commit/e66bbef97b2cef73a533ce8a506ec479852cb364
[3] https://issues.apache.org/jira/browse/IGNITE-12450

Re: BinaryObject API confuses users

[Николай Ижиков]

I’m too, Alex.

But, this signature leads to the same error as I mentioned.



> 6 дек. 2019 г., в 15:53, Alexei Scherbakov  
> написал(а):
> 
> I wonder why signature is not
> 
> setField(String name, BinaryObject obj)
> 
> пт, 6 дек. 2019 г. в 15:00, Ilya Kasnacheev :
> 
>> Hello!
>> 
>> I think that repeating argument type name in method name is a code smell.
>> Rather, we should describe how this method is different from other methods
>> of its bunch.
>> 
>> In this case, it is different since it allows you to create nested binary
>> objects, i.e., ones with non-flat structure.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> пт, 6 дек. 2019 г. в 13:45, Николай Ижиков :
>> 
>>> Hello, Ilya.
>>> 
>>> I don’t get your point
>>> 
>>>> We don’t passing … binary, just a … BinaryObjeсtBuilder.
>>> 
>>> May be we should go with `setBinaryObjectBuilderField` or
>>> `setBinaryObjectField`?
>>> 
>>> 
>>>> 6 дек. 2019 г., в 13:38, Ilya Kasnacheev 
>>> написал(а):
>>>> 
>>>> Hello!
>>>> 
>>>> I can see where you are getting at, can we call it "setFieldNested"
>>> instead
>>>> or something like that?
>>>> 
>>>> setBinaryField is confusing because we're not passing anything
>> especially
>>>> binary, just a nested BinaryObjectBuilder.
>>>> 
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>> 
>>>> 
>>>> пт, 6 дек. 2019 г. в 13:17, Николай Ижиков :
>>>> 
>>>>> Hello, Igniters.
>>>>> 
>>>>> We have confusing API in `BinaryObjectBuilder` class.
>>>>> 
>>>>> The code below leads to the `ClassCastException`
>>>>> The cause is java method resolution rules [1]
>>>>> 
>>>>>> There may be more than one such method, in which case the most
>> specific
>>>>> one is chosen
>>>>> 
>>>>> I suggest to deprecate `setField(String name, @Nullable
>>>>> BinaryObjectBuilder builder);` method and introduce
>>>>> `setBinaryField(String name, @Nullable BinaryObjectBuilder builder);`
>>>>> instead of it.
>>>>> 
>>>>> What do you think?
>>>>> 
>>>>> ```
>>>>> public class BugTest extends GridCommonAbstractTest {
>>>>>   @Test public void testBinaryObject() throws Exception {
>>>>>   try (Ignite ignite = startGrid(0)) {
>>>>>   BinaryObjectBuilder builder =
>>>>> ignite.binary().builder("testVal")
>>>>>   .setField("name", "John Doe", String.class);
>>>>> 
>>>>>   builder.setField("name", builder.getField("name"));
>>>>>   }
>>>>>   }
>>>>> }
>>>>> ```
>>>>> 
>>>>> [1]
>>>>> 
>>> 
>> https://docs.oracle.com/javase/specs/jls/se11/html/jls-15.html#jls-15.12.2
>>> 
>>> 
>> 
> 
> 
> -- 
> 
> Best regards,
> Alexei Scherbakov

Re: BinaryObject API confuses users

[Николай Ижиков]

Hello, Ilya.

I don’t get your point

> We don’t passing … binary, just a … BinaryObjeсtBuilder.

May be we should go with `setBinaryObjectBuilderField` or 
`setBinaryObjectField`?


> 6 дек. 2019 г., в 13:38, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> I can see where you are getting at, can we call it "setFieldNested" instead
> or something like that?
> 
> setBinaryField is confusing because we're not passing anything especially
> binary, just a nested BinaryObjectBuilder.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пт, 6 дек. 2019 г. в 13:17, Николай Ижиков :
> 
>> Hello, Igniters.
>> 
>> We have confusing API in `BinaryObjectBuilder` class.
>> 
>> The code below leads to the `ClassCastException`
>> The cause is java method resolution rules [1]
>> 
>>> There may be more than one such method, in which case the most specific
>> one is chosen
>> 
>> I suggest to deprecate `setField(String name, @Nullable
>> BinaryObjectBuilder builder);` method and introduce
>> `setBinaryField(String name, @Nullable BinaryObjectBuilder builder);`
>> instead of it.
>> 
>> What do you think?
>> 
>> ```
>> public class BugTest extends GridCommonAbstractTest {
>>@Test public void testBinaryObject() throws Exception {
>>try (Ignite ignite = startGrid(0)) {
>>BinaryObjectBuilder builder =
>> ignite.binary().builder("testVal")
>>.setField("name", "John Doe", String.class);
>> 
>>builder.setField("name", builder.getField("name"));
>>}
>>}
>> }
>> ```
>> 
>> [1]
>> https://docs.oracle.com/javase/specs/jls/se11/html/jls-15.html#jls-15.12.2

BinaryObject API confuses users

[Николай Ижиков]

Hello, Igniters.

We have confusing API in `BinaryObjectBuilder` class.

The code below leads to the `ClassCastException`
The cause is java method resolution rules [1]

> There may be more than one such method, in which case the most specific one 
> is chosen

I suggest to deprecate `setField(String name, @Nullable BinaryObjectBuilder 
builder);` method and introduce 
`setBinaryField(String name, @Nullable BinaryObjectBuilder builder);` instead 
of it.

What do you think?

```
public class BugTest extends GridCommonAbstractTest {
@Test public void testBinaryObject() throws Exception {
try (Ignite ignite = startGrid(0)) {
BinaryObjectBuilder builder = ignite.binary().builder("testVal")
.setField("name", "John Doe", String.class);

builder.setField("name", builder.getField("name"));
}
}
}
```

[1] https://docs.oracle.com/javase/specs/jls/se11/html/jls-15.html#jls-15.12.2

Re: [ANNOUNCE] 4 December 2019, the 2.8 RELEASE brach creation

[Николай Ижиков]

Thank for the leading release, Maxim.

Keep going!

> 4 дек. 2019 г., в 20:29, Alexey Zinoviev  написал(а):
> 
> Great, many thanks to your job
> 
> ср, 4 дек. 2019 г., 19:05 Maxim Muzafarov :
> 
>> Igniters,
>> 
>> 
>> 1. The release branch created: ignite-2.8
>> 2. The list of unresolved issues [1] related to the 2.8 release has
>> been cleaned up (only Blocker and Critical issues left).
>> 3. Apache Ignite 2.8 wiki page has been updated [2].
>> 4. The issue [3] with Corrupted B+ tree exception has been reverted
>> from the master branch. Discussed [4] [5].
>> 
>> 
>> If you are working on issues related to 2.8 release, please, don't forget
>> to:
>> 1. Discuss them on dev-list (To make the release branch stable we need
>> to eliminate risks of any such issues)
>> 2. Merge them to both branches master and ignite-2.8.
>> 
>> 
>> [1]
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolvedissues(notrelatedtodocumentation)
>> [2] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8
>> [3] https://issues.apache.org/jira/browse/IGNITE-11704
>> [4]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-8-RELEASE-Time-Scope-Manager-tp43616p44539.html
>> [5]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-8-RELEASE-Time-Scope-Manager-tp43616p44551.html
>> 
>> On Fri, 29 Nov 2019 at 12:52, Alexey Zinoviev 
>> wrote:
>>> 
>>> Is it ok for me, but please move IGFS dropping ticket to the next
>> release,
>>> I have no capacity to finish in time the blocker ticket related to
>>> Tensorflow-Ignite integration and as a result we shouldn't remove the
>> IGFS
>>> yet but please mark it as a deprecated and ready for removal in 3.0
>>> 
>>> чт, 28 нояб. 2019 г. в 16:03, Maxim Muzafarov :
>>> 
 Denis,
 
 What kind of additional checks do we need? Maybe I'm missing something
 but all issues currently merged to the master branch will be present
 in the 2.8 release branch too.
 I see that this issue has been closed a long time ago, so it
 definitely comes in.
 
 On Thu, 28 Nov 2019 at 01:33, Denis Magda  wrote:
> 
> Hi Maxim,
> 
> Sounds good, thanks for branching. Could you double-check that the
> following issue was added to the release?
> https://issues.apache.org/jira/browse/IGNITE-10577
> 
> -
> Denis
> 
> 
> On Wed, Nov 27, 2019 at 7:50 AM Maxim Muzafarov 
 wrote:
> 
>> Igniters,
>> 
>> 
>> I think we are ready to create the 2.8 release branch since
>> features
>> [1] which we are waiting for are almost completed.
>> 
>> 4 December 2019, the 2.8 release branch will be created. All
>> related
>> issues with a priority less than `critical` will be excluded from
>> the
>> 2.8 release (fix version will be changed to 2.9).
>> 
>> 
>> Any objections?
>> 
>> 
>> [1]
>> 
 
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Awatingfeaturescompletion
>> 
 
>> 

[TeamCIty] Issues with the disc space

[Николай Ижиков]

Hello, Igniters.

Looks like we have an issues with the TC agents disc space [1]
Can someone take a look?

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=4809938=buildResultsDiv=IgniteTests24Java8_PlatformCPPLinuxClang

Re: Joining node validation failure event.

[Николай Ижиков]

Exception(s) from component(s) that don’t want node joined.

> 3 дек. 2019 г., в 12:39, Ivan Pavlukhin  написал(а):
> 
> How that reason will look like? Actually, I mostly thinking about
> general API here. What I would like to avoid is exposing something not
> general but needed only for a particular extension (plugin).
> 
> вт, 3 дек. 2019 г. в 12:31, Николай Ижиков :
>> 
>> I think we also should provide the reason why join failed.
>> 
>>> 3 дек. 2019 г., в 12:22, Ivan Pavlukhin  написал(а):
>>> 
>>> Mikhail,
>>> 
>>> So, I suppose there should be ordering guarantees that listener is
>>> registered before first validation failure can occur. Hope
>>> GridComponent#onKernalStart is the right place. Is it enough to pass
>>> only problematic node id (or ClusterNode) with an event? Actually such
>>> event seems to fit naturally node join/left/failed events.
>>> 
>>> вт, 3 дек. 2019 г. в 10:38, Mikhail Petrov :
>>>> 
>>>> Hi Ivan.
>>>> 
>>>> No other lifecycle events are needed in my case.
>>>> 
>>>> We can register a listener in the security component's
>>>> GridComponent#onKernalStart method and listen locally to every failed
>>>> joining attempt. Am I missing something?
>>>> 
>>>> Regards,
>>>> Mikhail.
>>>> 
>>>> On 03.12.2019 8:48, Ivan Pavlukhin wrote:
>>>>> Mikhail,
>>>>> 
>>>>> Do you need some ordering guarantees among node lifecycle events and
>>>>> listener notifications. For example, I can imagine that it is good to
>>>>> notify security component about every node failed validation. How can
>>>>> we achieve it with events (I assume dynamic listener registration)?
>>>>> 
>>>>> пн, 2 дек. 2019 г. в 18:09, Mikhail Petrov :
>>>>>> Hi, Andrey.
>>>>>> 
>>>>>> It doesn't influence on authentication or authorization process. There
>>>>>> is a security requirement that demands to notify some outer security
>>>>>> subsystems in a specific way when joining node validation failed in any
>>>>>> Ignite component (e.g. GridCacheProcessor) not only in
>>>>>> IgniteSecurityProcessor. So PluginProvider#validateNewNode is not
>>>>>> acceptable for me.
>>>>>> 
>>>>>> Regards,
>>>>>> Mikhail.
>>>>>> 
>>>>>> On 02.12.2019 16:35, Andrey Gura wrote:
>>>>>>> Mikhail,
>>>>>>> 
>>>>>>> I don't understand how node validation on join affects security. But
>>>>>>> it seems that you can use
>>>>>>> PluginProvider#validateNewNode(org.apache.ignite.cluster.ClusterNode,
>>>>>>> java.io.Serializable) method. Does it fit for your needs?
>>>>>>> 
>>>>>>> On Mon, Dec 2, 2019 at 12:54 PM Mikhail Petrov  
>>>>>>> wrote:
>>>>>>>> Hi, Ivan.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Unfortunately, we came to no decision yet. As I mentioned above this
>>>>>>>> event is disabled by default and no node will receive this event 
>>>>>>>> without
>>>>>>>> an explicit subscription. In my case, that event is assumed to be used
>>>>>>>> on node locally to share joining node info between security and
>>>>>>>> discovery components. I have no idea how to solve this problem without
>>>>>>>> publishing a new event too. But I'm concerned about the acceptance of
>>>>>>>> that solution. Maybe it can be solved some other way? I will appreciate
>>>>>>>> any suggestion or review PR [1] with event implementation.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [1] https://github.com/apache/ignite/pull/7057
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Mikhail.
>>>>>>>> 
>>>>>>>> On 02.12.2019 10:38, Ivan Pavlukhin wrote:
>>>>>>>>> Mikhail, Andrey,
>>>>>>>>> 
>>>>>>>>> Have you come to a common decision here? As for me, it sounds useful
>

Re: Joining node validation failure event.

[Николай Ижиков]

I think we also should provide the reason why join failed.

> 3 дек. 2019 г., в 12:22, Ivan Pavlukhin  написал(а):
> 
> Mikhail,
> 
> So, I suppose there should be ordering guarantees that listener is
> registered before first validation failure can occur. Hope
> GridComponent#onKernalStart is the right place. Is it enough to pass
> only problematic node id (or ClusterNode) with an event? Actually such
> event seems to fit naturally node join/left/failed events.
> 
> вт, 3 дек. 2019 г. в 10:38, Mikhail Petrov :
>> 
>> Hi Ivan.
>> 
>> No other lifecycle events are needed in my case.
>> 
>> We can register a listener in the security component's
>> GridComponent#onKernalStart method and listen locally to every failed
>> joining attempt. Am I missing something?
>> 
>> Regards,
>> Mikhail.
>> 
>> On 03.12.2019 8:48, Ivan Pavlukhin wrote:
>>> Mikhail,
>>> 
>>> Do you need some ordering guarantees among node lifecycle events and
>>> listener notifications. For example, I can imagine that it is good to
>>> notify security component about every node failed validation. How can
>>> we achieve it with events (I assume dynamic listener registration)?
>>> 
>>> пн, 2 дек. 2019 г. в 18:09, Mikhail Petrov :
 Hi, Andrey.
 
 It doesn't influence on authentication or authorization process. There
 is a security requirement that demands to notify some outer security
 subsystems in a specific way when joining node validation failed in any
 Ignite component (e.g. GridCacheProcessor) not only in
 IgniteSecurityProcessor. So PluginProvider#validateNewNode is not
 acceptable for me.
 
 Regards,
 Mikhail.
 
 On 02.12.2019 16:35, Andrey Gura wrote:
> Mikhail,
> 
> I don't understand how node validation on join affects security. But
> it seems that you can use
> PluginProvider#validateNewNode(org.apache.ignite.cluster.ClusterNode,
> java.io.Serializable) method. Does it fit for your needs?
> 
> On Mon, Dec 2, 2019 at 12:54 PM Mikhail Petrov  
> wrote:
>> Hi, Ivan.
>> 
>> 
>> Unfortunately, we came to no decision yet. As I mentioned above this
>> event is disabled by default and no node will receive this event without
>> an explicit subscription. In my case, that event is assumed to be used
>> on node locally to share joining node info between security and
>> discovery components. I have no idea how to solve this problem without
>> publishing a new event too. But I'm concerned about the acceptance of
>> that solution. Maybe it can be solved some other way? I will appreciate
>> any suggestion or review PR [1] with event implementation.
>> 
>> 
>> [1] https://github.com/apache/ignite/pull/7057
>> 
>> 
>> Regards,
>> 
>> Mikhail.
>> 
>> On 02.12.2019 10:38, Ivan Pavlukhin wrote:
>>> Mikhail, Andrey,
>>> 
>>> Have you come to a common decision here? As for me, it sounds useful
>>> to expose node join failure details somehow. The thing to decide on is
>>> a mechanism to expose it. Unfortunately, immediately have no idea
>>> better than using events.
>>> 
 What is purpose of the special cluster wide event? Node is not joined
 to the topology. Why topology nodes should know something about this
 node?
>>> Was this question answered? I suppose that at least coordinator will
>>> receive the event, will not it?
>>> 
>>> чт, 28 нояб. 2019 г. в 10:10, Mikhail Petrov :
 Hi, Ivan.
 
 I'm sorry that the discussion was moved in private channel. The problem
 I'm trying to solve is described below in the thread. The security
 plugin in my case stores and analyzesinfo about a node that failed to 
 join.
 
 
 Regards,
 
 Mikhail.
 
 
 
  Forwarded Message 
 Subject:Re: Joining node validation failure event.
 Date:   Thu, 21 Nov 2019 21:43:33 +0300
 From:   Mikhail Petrov 
 To: Andrey Gura 
 
 
 
 Hi, Andrey.
 
 In my task security plugin running on the coordinator must locally
 handle the situation when some node is trying to join the topology with
 the invalid configuration. I can't handle the response on a node that
 failed to connect because it's untrusted.
 
 Actually I'm only concerned about one validation -- when all Ignite
 components validate new node.
 
 In my case plugin must be able to obtain general and security subject
 information from joining TcpDiscoveryNode attributes. But I have no 
 idea
 how to share information between the security and discovery components
 without recording event and listening it locally.
 
 This event is assumed to be disable by default and in my 

JDK11. ML module. Compilation error.

[Николай Ижиков]

Hello, guys.

We have a compile error under Java11 in ML module.
Can someone take a look?

```
sbt-izhikov-nv:~/src/ignite:[master]$ mvn -version
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 
2019-08-27T18:06:16+03:00)
Maven home: /Users/sbt-izhikov-nv/bin/apache-maven-3.6.2
Java version: 11.0.5, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
Default locale: ru_RU, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.1", arch: "x86_64", family: "mac"
sbt-izhikov-nv:~/src/ignite:[master]$ java -version
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)
sbt-izhikov-nv:~/src/ignite:[master]$ mvn -U 
-Pall-java,all-scala,scala,licenses,lgpl,examples,tensorflow,checkstyle 
-DskipTests -Dmaven.javadoc.skip=true clean install
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.7.0:testCompile 
(default-testCompile) on project ignite-ml: Compilation failure: Compilation 
failure: 
[ERROR] 
/Users/sbt-izhikov-nv/src/ignite/modules/ml/src/test/java/org/apache/ignite/ml/dataset/primitive/DatasetWrapperTest.java:[64,45]
 incompatible types: java.lang.Object cannot be converted to java.lang.Integer
[ERROR] 
/Users/sbt-izhikov-nv/src/ignite/modules/ml/src/test/java/org/apache/ignite/ml/dataset/primitive/DatasetWrapperTest.java:[78,45]
 incompatible types: java.lang.Object cannot be converted to java.lang.Integer
[ERROR] 
/Users/sbt-izhikov-nv/src/ignite/modules/ml/src/test/java/org/apache/ignite/ml/dataset/primitive/DatasetWrapperTest.java:[105,38]
 incompatible types: java.lang.Object cannot be converted to java.lang.Integer
[ERROR] 
/Users/sbt-izhikov-nv/src/ignite/modules/ml/src/test/java/org/apache/ignite/ml/dataset/primitive/DatasetWrapperTest.java:[119,38]
 incompatible types: java.lang.Object cannot be converted to java.lang.Integer

```

Re: Data consistency essentials explained

[Николай Ижиков]

Hello, Alex.

I think we should improve article formatting - code highlight, fonts, etc.

For now, it’s very hard to read it.

Can you, please, do it?

> 28 нояб. 2019 г., в 09:25, Ivan Pavlukhin  написал(а):
> 
> Alexei,
> 
> Many thanks for that article! Really nice that now we have a document
> describing Ignite approach for data consistency. I think it is super
> important because it is not trivial and very Ignite specific. I
> believe lots of Ignite developers will learn something new from this
> document.
> 
> By the way, I noticed a link to GridGain documentation [1]. Is it fine?
> 
> [1] 
> https://www.gridgain.com/docs/latest/developers-guide/key-value-api/transactions#pessimistic-transactions
> 
> чт, 28 нояб. 2019 г. в 02:49, Denis Magda :
>> 
>> Alex,
>> 
>> Thanks a lot for putting your knowledge on paper. That's an invaluable 
>> contribution!
>> 
>> Looping in the user community (will be interesting for those who use native 
>> persistence) and has just spread the word via Ignite twitter handle:
>> https://twitter.com/ApacheIgnite/status/1199837055007612928
>> 
>> -
>> Denis
>> 
>> 
>> On Wed, Nov 27, 2019 at 12:29 AM Alexei Scherbakov 
>>  wrote:
>>> 
>>> Igniters,
>>> 
>>> As a final action in my almost year long effort in repairing data
>>> consistency related issues for transactional persistent caches I've
>>> prepared an article explaining the principles of achieving the consistency
>>> between partition copies for these scenarios [1]
>>> Comments and suggestions are welcome.
>>> I plan to keep the article in actual state if something will change in this
>>> area in the future.
>>> Fixes were mostly done under IGNITE-10780 and several follow-up fixes.
>>> 
>>> There is still work to be done. Currently I'm working on similar issue for
>>> atomic persistent caches [2]. I hope to give an update on that soon.
>>> 
>>> [1]  https://cwiki.apache.org/confluence/display/IGNITE/Data+consistency
>>> [2] https://issues.apache.org/jira/browse/IGNITE-11797
>>> --
>>> 
>>> Best regards,
>>> Alexei Scherbakov
> 
> 
> 
> -- 
> Best regards,
> Ivan Pavlukhin

[MEETUP] 03.12.2019 Moscow

[Николай Ижиков]

Hello, Igniters.

Join us on Ignite meetup!

There will be cool old fashioned geeky Ignite talks and a "round table" 
discussion about Ignite future.

It will happen on *December 3 in Moscow* and will be hosted by Sberbank near 
the subway Kutuzovskaya.

Topics:

* Ignite affinity function.
* upcoming security improvement in Ignite.
* brainstorm with the whole community on the topic
"What I like in Ignite, what I want in Ignite, what in Ignite are piss me off"

Please, note, the language of the meetup is Russian.

You should register here  - 
https://www.meetup.com/ru-RU/Moscow-Apache-Ignite-Meetup/events/266728326

Re: The Spark 2.4 support

[Николай Ижиков]

Hello, Alexey.

Can we somehow highlight changes in Spark-2.4 module comparing to 2.3 one?
For now the changes look too huge for me (+11,681 −1).

Are we sure we want to add those huge piece of code to support two versions?
Can we extract unchanged parts(based on spark public API) and keep them in one 
copy?

> 18 нояб. 2019 г., в 23:47, Denis Magda  написал(а):
> 
> Alexey, thanks for the details and for reaching out this milestone with the
> 2.4 support.
> 
> Generally, I would advise us to merge the changes to the master only after
> we confirm the failing tests are not regressions. We should either remove
> them or replace them with some others or just fix.
> 
> -
> Denis
> 
> 
> On Mon, Nov 18, 2019 at 10:06 AM Alexey Zinoviev 
> wrote:
> 
>> Right, a few tests from 200 are failed due to known issue and couldnt be
>> fixed immediately, related to rare cases. These tests are copies of 2.3
>> tests and part of them could have no meaning for 2.4 due to Spark changed
>> behaviour.
>> 
>> пн, 18 нояб. 2019 г., 19:42 Denis Magda :
>> 
>>> Alexey,
>>> 
>>> Please help to understand what it means that 2.4 integration supports
>> "95%
>>> of tests of 2.3". Does it mean that 5% of existing tests are failing and,
>>> basically, need to be fixed?
>>> 
>>> -
>>> Denis
>>> 
>>> 
>>> On Mon, Nov 18, 2019 at 6:52 AM Alexey Zinoviev 
>>> wrote:
>>> 
 Dear Nikolay Izhikov, I've recreated the PR for 2.4 initial support
 
 The last commit
 
 
>>> 
>> https://github.com/apache/ignite/pull/7058/commits/60386802299deedc6ed60bf4736e922201a67fb8
 contains
 real changes from Spark 2.3
 
 I suggest to merge to master this initial solution with 95% support of
 Spark 2.4 and continue work on known issues listed in JIRA
 
 This solution supports the new Spark version for all examples and 95%
>> of
 tests of 2.3.
 
 вт, 1 окт. 2019 г. в 08:48, Ivan Pavlukhin :
 
> Alexey, Nikolay,
> 
> Thank you for sharing details!
> 
> вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev >> :
>> 
>> Great talk and paper, I've learnt it last year
>> 
>> пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov :
>> 
>>> Yes, I can :)
>>> 
>>> В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет:
 Nikolay,
 
 Would you be able to review the changes? I'm not sure there is
>> a
> better
>>> candidate for now.
 
 -
 Denis
 
 
 On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov <
> nizhi...@apache.org>
>>> wrote:
> Hello, Ivan.
> 
> I had a talk about internals of Spark integration in Ignite.
> It answers on question why we should use Spark internals.
> 
> You can take a look at my meetup talk(in Russian) [1] or read
>>> an
>>> article if you prefer text [2].
> 
> [1] https://www.youtube.com/watch?v=CzbAweNKEVY
> [2] https://habr.com/ru/company/sberbank/blog/427297/
> 
> В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev пишет:
>> Yes, as I understand it uses Spark internals from the first
> commit)))
>> The reason - we take Spark SQL query execution plan and try
>>> to
>>> execute it
>> on Ignite cluster
>> Also we inherit a lot of Developer API related classes that
> could be
>> unstable. Spark has no good point for extension and this
>> is a
> reason
>>> why we
>> should go deeper
>> 
>> пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin <
> vololo...@gmail.com>:
>> 
>>> Hi Alexey,
>>> 
>>> As an external watcher very far from Ignite Spark
 integration I
>>> would
>>> like to ask a humble question for my understanding. Why
>>> this
>>> integration uses Spark internals? Is it a common approach
>>> for
>>> integrating with Spark?
>>> 
>>> пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev <
>>> zaleslaw@gmail.com>:
 
 Hi, Igniters
 I've started the work on the Spark 2.4 support
 
 We started the discussion here, in
 https://issues.apache.org/jira/browse/IGNITE-12054
 
 The Spark internals were totally refactored between 2.3
>>> and
> 2.4
>>> versions,
 main changes touches
 
   - External catalog and listeners refactoring
   - Changes of HAVING operator semantic support
   - Push-down NULL filters generation in JOIN plans
   - minor changes in Plan Generation that should be
 adopted
> in
>>> our
   integration module
 
 I propose the initial solution here via creation of new
> module
>>> spark-2.4
 here
>> https://issues.apache.org/jira/browse/IGNITE-12247
 and
>>> addition of

Re: Thin client: compute support

[Николай Ижиков]

Hello, Igniters.

I think we should support full compute API for the thin clients(at least for 
the java clients)

I think our roadmap should be the following:

1. Execution existing compute tasks by id (as Alex suggested).
2. Deploy and execution arbitrary compute tasks from the java thin client
3. Research possibility to create API for deployment and execution compute 
tasks in any language that implemented thin client protocol.(python as a first 
candidate).


> 22 нояб. 2019 г., в 10:19, Alex Plehanov  написал(а):
> 
> Denis, the primary motivation is to enable execution of deployed to server
> java tasks from thin clients (java and other languages).
> 
> пт, 22 нояб. 2019 г. в 00:03, Denis Magda :
> 
>> Alex, what is the primary motivation for this improvement? Are you looking
>> to enable the compute APIs for languages different from Java?
>> 
>> -
>> Denis
>> 
>> 
>> On Wed, Nov 20, 2019 at 11:58 PM Alex Plehanov 
>> wrote:
>> 
>>> Hello, Igniters!
>>> 
>>> I have plans to start implementation of Compute interface for Ignite thin
>>> client and want to discuss features that should be implemented.
>>> 
>>> We already have Compute implementation for binary-rest clients
>>> (GridClientCompute), which have the following functionality:
>>> - Filtering cluster nodes (projection) for compute
>>> - Executing task by the name
>>> 
>>> I think we can implement this functionality in a thin client as well.
>>> 
>>> First of all, we need some operation types to request a list of all
>>> available nodes and probably node attributes (by a list of nodes). Node
>>> attributes will be helpful if we will decide to implement analog of
>>> ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in the
>> thin
>>> client. Perhaps they can be requested lazily.
>>> 
>>> From the protocol point of view there will be two new operations:
>>> 
>>> OP_CLUSTER_GET_NODES
>>> Request: empty
>>> Response: long topologyVersion, int minorTopologyVersion, int nodesCount,
>>> for each node set of node fields (UUID nodeId, Object or String
>>> consistentId, long order, etc)
>>> 
>>> OP_CLUSTER_GET_NODE_ATTRIBUTES
>>> Request: int nodesCount, for each node: UUID nodeId
>>> Response: int nodesCount, for each node: int attributesCount, for each
>> node
>>> attribute: String name, Object value
>>> 
>>> To execute tasks we need something like these methods in the client API:
>>> Object execute(String task, Object arg)
>>> Future executeAsync(String task, Object arg)
>>> Object affinityExecute(String task, String cache, Object key, Object arg)
>>> Future affinityExecuteAsync(String task, String cache, Object
>> key,
>>> Object arg)
>>> 
>>> Which can be mapped to protocol operations:
>>> 
>>> OP_COMPUTE_EXECUTE_TASK
>>> Request: UUID nodeId, String taskName, Object arg
>>> Response: Object result
>>> 
>>> OP_COMPUTE_EXECUTE_TASK_AFFINITY
>>> Request: String cacheName, Object key, String taskName, Object arg
>>> Response: Object result
>>> 
>>> The second operation is needed because we sometimes can't calculate and
>>> connect to affinity node on the client-side (affinity awareness can be
>>> disabled, custom affinity function can be used or there can be no
>>> connection between client and affinity node), but we can make best effort
>>> to send request to target node if affinity awareness is enabled.
>>> 
>>> Currently, on the server-side requests always processed synchronously and
>>> responses are sent right after request was processed. To execute long
>> tasks
>>> async we should whether change this logic or introduce some kind two-way
>>> communication between client and server (now only one-way requests from
>>> client to server are allowed).
>>> 
>>> Two-way communication can also be useful in the future if we will send
>> some
>>> server-side generated events to clients.
>>> 
>>> In case of two-way communication there can be new operations introduced:
>>> 
>>> OP_COMPUTE_EXECUTE_TASK (from client to server)
>>> Request: UUID nodeId, String taskName, Object arg
>>> Response: long taskId
>>> 
>>> OP_COMPUTE_TASK_FINISHED (from server to client)
>>> Request: taskId, Object result
>>> Response: empty
>>> 
>>> The same for affinity requests.
>>> 
>>> Also, we can implement not only execute task operation, but some other
>>> operations from IgniteCompute (broadcast, run, call), but it will be
>> useful
>>> only for java thin client. And even with java thin client we should
>> whether
>>> implement peer-class-loading for thin clients (this also requires two-way
>>> client-server communication) or put classes with executed closures to the
>>> server locally.
>>> 
>>> What do you think about proposed protocol changes?
>>> Do we need two-way requests between client and server?
>>> Do we need support of compute methods other than "execute task"?
>>> What do you think about peer-class-loading for thin clients?
>>> 
>> 

Re: Integration of Spark and Ignite. Prototype.

[Николай Ижиков]

Val, Thank you.

I fixed issues and answered questions from comments.
Please, take a look.

2017-12-13 3:28 GMT+03:00 Valentin Kulichenko <valentin.kuliche...@gmail.com
>:

> Hi Nikolay,
>
> I reviewed the code and left several comments in the ticket [1]. Please
> take a look.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
>
> -Val
>
> On Mon, Dec 4, 2017 at 3:03 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Denis,
> >
> > Nikolay was doing final changes and TC stabilization. I'm planning to do
> > final review this week, so hopefully we will merge the code soon.
> >
> > -Val
> >
> > On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda <dma...@apache.org> wrote:
> >
> >> Nikolay, Val,
> >>
> >> Since we agreed to release the feature without the strategy support, can
> >> the current integration meet the world in 2.4 release? Please chime in
> this
> >> conversation:
> >> http://apache-ignite-developers.2346864.n4.nabble.com/Time-
> >> and-scope-for-Apache-Ignite-2-4-td24987.html
> >>
> >> —
> >> Denis
> >>
> >> > On Nov 28, 2017, at 5:42 PM, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >> >
> >> > Denis,
> >> >
> >> > Agree. I will do the final review in next few days and merge the code.
> >> >
> >> > -Val
> >> >
> >> > On Tue, Nov 28, 2017 at 5:28 PM, Denis Magda <dma...@apache.org>
> wrote:
> >> >
> >> >> Guys,
> >> >>
> >> >> Looking into the parallel discussion about the strategy support I
> would
> >> >> change my initial stance and support the idea of releasing the
> >> integration
> >> >> in its current state. Is the code ready to be merged into the master?
> >> Let’s
> >> >> concentrate on this first and handle the strategy support as a
> separate
> >> >> JIRA task. Agree?
> >> >>
> >> >> —
> >> >> Denis
> >> >>
> >> >>> On Nov 27, 2017, at 3:47 PM, Valentin Kulichenko <
> >> >> valentin.kuliche...@gmail.com> wrote:
> >> >>>
> >> >>> Nikolay,
> >> >>>
> >> >>> Let's estimate the strategy implementation work, and then decide
> >> weather
> >> >> to
> >> >>> merge the code in current state or not. If anything is unclear,
> please
> >> >>> start a separate discussion.
> >> >>>
> >> >>> -Val
> >> >>>
> >> >>> On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков <
> >> nizhikov@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Hello, Val, Denis.
> >> >>>>
> >> >>>>> Personally, I think that we should release the integration only
> >> after
> >> >>>> the strategy is fully supported.
> >> >>>>
> >> >>>> I see two major reason to propose merge of DataFrame API
> >> implementation
> >> >>>> without custom strategy:
> >> >>>>
> >> >>>> 1. My PR is relatively huge, already. From my experience of
> >> interaction
> >> >>>> with Ignite community - the bigger PR becomes, the more time of
> >> >> commiters
> >> >>>> required to review PR.
> >> >>>> So, I propose to move smaller, but complete steps here.
> >> >>>>
> >> >>>> 2. It is not clear for me what exactly includes "custom strategy
> and
> >> >>>> optimization".
> >> >>>> Seems, that additional discussion required.
> >> >>>> I think, I can put my thoughts on the paper and start discussion
> >> right
> >> >>>> after basic implementation is done.
> >> >>>>
> >> >>>>> Custom strategy implementation is actually very important for this
> >> >>>> integration.
> >> >>>>
> >> >>>> Understand and fully agreed.
> >> >>>> I'm ready to continue work in that area.
> >> >>>>
> >> >>>> 23.11.2017 02:15, Denis Magda пишет:
> >> >>>>
> >> >>>> Val, Nikolay,
&

Re: TC issues. IGNITE-3084. Spark Data Frame API

[Николай Ижиков]

Valentin,

Now it's run OK.

Thank you.

30.11.2017 23:41, Valentin Kulichenko пишет:

Nikolay,

Please try once again.

-Val

On Thu, Nov 30, 2017 at 11:43 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>> wrote:

Valentin,

Thank you, but your changes does not enough

I ran a build for my branch and still has "Unsupport major.minor version 
52.0" issue [1]

Build log:

`Starting: /usr/lib/jvm/java-7-oracle/bin/java 
-DJAVA_HOME=/usr/lib/jvm/java-8-oracle`

I look to build settings and found some variables that sill points to jdk7:

Environment variables:

env.JAVA_HOME   /usr/lib/jvm/java-7-oracle
env.JDK_HOME    /usr/lib/jvm/java-7-oracle

Can you, please, change this variables too?

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=970913=Ignite20Tests_IgniteRdd=buildLog

<https://ci.ignite.apache.org/viewLog.html?buildId=970913=Ignite20Tests_IgniteRdd=buildLog>


30.11.2017 22:16, Valentin Kulichenko пишет:

Nikolay,

Java 7 support will be dropped by Ignite soon, so let's do the upgrade 
now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on 
TC to use JDK 8. Can you try it out and let
me know if it works?

-Val

On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>> wrote:

     Valentin,

     > Oh, so this is because of upgrade to 2.2.0?

     Yes, we should upgrade spark module to jdk1.8 because switching to 
spark 2.2.0.

     > least we should consider not dropping previous version yet.

     Please, note, we can have IgniteRDD and Ignite Data Frame for a 
spark 2.1.

     The only thing we can't have is IgniteCatalog.

     Currently, in my PR I include IgniteRDD and Ignite Data Frame in 
spark_2.10 module
     So we doesn't have to drop spark 2.1 completely.

https://github.com/apache/ignite/pull/2742 
<https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 
<https://github.com/apache/ignite/pull/2742>>

     30.11.2017 02:09, Valentin Kulichenko пишет:

         Nikolay,

         Oh, so this is because of upgrade to 2.2.0? Then I'm not sure 
we should upgrade in the first place, or at least we should consider not 
dropping previous version yet. I sent a message
to the
         original thread about upgrade, let's decide there and then 
come back to this issue.

         -Val

         On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com> 
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com>>>> wrote:

              Valentin,

              For now `Ignite RDD` build runs on jdk1.7.
              We need to update it to jdk1.8.

              I wrote the whole versions numbers to be clear:

              1. Current master - Spark version is 2.1.0.
                       So both `Ignite RDD` and `Ignite RDD 2.10` runs 
OK on jdk1.7.

              2. My branch -
                       `Ignite RDD 2.10` - spark version is 2.1.2 - 
runs OK on jdk1.7.
                       `Ignite RDD` - spark version 2.2.0 - fails on 
jdk1.7, *has to be changed to run on jdk1.8*


              29.11.2017 03:27, Valentin Kulichenko пишет:

                  Nikolay,

                  If Spark requires Java 8, then I guess we have no 
choice. How TC is configured at the moment? My understanding is that Spark 
related suites are successfully executed there, so is
             there an issue?

                  -Val

                  On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com> 
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com>>>
         <mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com> 
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com>> 
<mailto:nizhikov@gmail.com
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>>>> wrote:

                       Hello, Valentin.

                           Added '-Dscala-2.10' to the build config. 
Let me know if i

Re: TC issues. IGNITE-3084. Spark Data Frame API

[Николай Ижиков]

Valentin,

Thank you, but your changes does not enough

I ran a build for my branch and still has "Unsupport major.minor version 52.0" 
issue [1]

Build log:

`Starting: /usr/lib/jvm/java-7-oracle/bin/java 
-DJAVA_HOME=/usr/lib/jvm/java-8-oracle`

I look to build settings and found some variables that sill points to jdk7:

Environment variables:

env.JAVA_HOME   /usr/lib/jvm/java-7-oracle
env.JDK_HOME/usr/lib/jvm/java-7-oracle

Can you, please, change this variables too?

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=970913=Ignite20Tests_IgniteRdd=buildLog


30.11.2017 22:16, Valentin Kulichenko пишет:

Nikolay,

Java 7 support will be dropped by Ignite soon, so let's do the upgrade now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on TC to use JDK 8. Can you try it out and let me know 
if it works?


-Val

On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>> wrote:

Valentin,

> Oh, so this is because of upgrade to 2.2.0?

Yes, we should upgrade spark module to jdk1.8 because switching to spark 
2.2.0.

> least we should consider not dropping previous version yet.

Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.

The only thing we can't have is IgniteCatalog.

Currently, in my PR I include IgniteRDD and Ignite Data Frame in spark_2.10 
module
So we doesn't have to drop spark 2.1 completely.

https://github.com/apache/ignite/pull/2742 
<https://github.com/apache/ignite/pull/2742>

30.11.2017 02:09, Valentin Kulichenko пишет:

Nikolay,

Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should 
upgrade in the first place, or at least we should consider not dropping 
previous version yet. I sent a message to the
original thread about upgrade, let's decide there and then come back to 
this issue.

-Val

    On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>> wrote:

     Valentin,

     For now `Ignite RDD` build runs on jdk1.7.
     We need to update it to jdk1.8.

     I wrote the whole versions numbers to be clear:

     1. Current master - Spark version is 2.1.0.
              So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on 
jdk1.7.

     2. My branch -
              `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on 
jdk1.7.
              `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, 
*has to be changed to run on jdk1.8*


     29.11.2017 03:27, Valentin Kulichenko пишет:

         Nikolay,

         If Spark requires Java 8, then I guess we have no choice. How 
TC is configured at the moment? My understanding is that Spark related suites 
are successfully executed there, so is
there an issue?

         -Val

             On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com> 
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com>>>> wrote:

              Hello, Valentin.

                  Added '-Dscala-2.10' to the build config. Let me know 
if it helps.


              Yes, it helps. Thank you!
              Now, 'Ignite RDD spark 2_10' succeed for my branch.


                  Do you mean that IgniteRDD does not compile on JDK7? 
If yes, do we know the reason? I don't think switching it to JDK8 is a solution 
as it should work with both.


              I mean that latest version of spark doesn't support jdk7.

http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/> 
<http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>> 
<http://spark.apache.org/docs/latest/
<http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ 
<http://spark.apache.org/docs/latest/>>>

              "Spark runs on Java 8+..."
              "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
              "Note that support for Java 7... were removed as of Spark 
2.2.0"
              "Note that support for Scala 2.10 is deprecated..."

              Moreover, We can't have IgniteCatalog for spark 2.1.
              Please, see my explanation in jira ticket -


https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpa

Re: Optimization of SQL queries from Spark Data Frame to Ignite

[Николай Ижиков]

Valentin,

> Can you please create a separate ticket for the strategy implementation then?

Done.

https://issues.apache.org/jira/browse/IGNITE-7077

> Any idea on how long will it take?

I think it will take 2-4 weeks to implement such a strategy.
I try my best to make a ready to review PR before the end of the year.


30.11.2017 02:13, Valentin Kulichenko пишет:

Nikolay,

Can you please create a separate ticket for the strategy implementation
then? Any idea on how long will it take?

As for querying a partition, both SqlQuery and SqlFieldQuery allow to
specify set of partitions to work with (see setPartitions method). I think
that should be enough.

-Val

On Wed, Nov 29, 2017 at 3:39 AM, Vladimir Ozerov <voze...@gridgain.com>
wrote:


Hi Nikolay,

No, it is not possible to get this info from public API, neither we planned
to expose it. See IGNITE-4509 and commit *fbf0e353* to get better
understanding on how this was implemented.

Vladimir.

On Wed, Nov 29, 2017 at 2:01 PM, Николай Ижиков <nizhikov@gmail.com>
wrote:


Hello, Vladimir.


partition pruning is already implemented in Ignite, so there is no need

to do this on your own.

Spark work with partitioned data set.
It is required to provide data partition information to Spark from custom
Data Source(Ignite).

Can I get information about pruned partitions throw some public API?
Is there a plan or ticket to implement such API?



2017-11-29 10:34 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:


Nikolay,

Regarding p3. - partition pruning is already implemented in Ignite, so
there is no need to do this on your own.

On Wed, Nov 29, 2017 at 3:23 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:


Nikolay,

Custom strategy allows to fully process the AST generated by Spark

and

convert it to Ignite SQL, so there will be no execution on Spark side

at

all. This is what we are trying to achieve here. Basically, one will

be

able to use DataFrame API to execute queries directly on Ignite. Does

it

make sense to you?

I would recommend you to take a look at MemSQL implementation which

does

similar stuff: https://github.com/memsql/memsql-spark-connector

Note that this approach will work only if all relations included in

AST

are

Ignite tables. Otherwise, strategy should return null so that Spark

falls

back to its regular mode. Ignite will be used as regular data source

in

this case, and probably it's possible to implement some optimizations

here

as well. However, I never investigated this and it seems like another
separate discussion.

-Val

On Tue, Nov 28, 2017 at 9:54 AM, Николай Ижиков <

nizhikov@gmail.com>

wrote:


Hello, guys.

I have implemented basic support of Spark Data Frame API [1], [2]

for

Ignite.
Spark provides API for a custom strategy to optimize queries from

spark

to

underlying data source(Ignite).

The goal of optimization(obvious, just to be on the same page):
Minimize data transfer between Spark and Ignite.
Speedup query execution.

I see 3 ways to optimize queries:

 1. *Join Reduce* If one make some query that join two or

more

Ignite tables, we have to pass all join info to Ignite and transfer

to

Spark only result of table join.
 To implement it we have to extend current implementation

with

new

RelationProvider that can generate all kind of joins for two or

more

tables.

 We should add some tests, also.
 The question is - how join result should be partitioned?


 2. *Order by* If one make some query to Ignite table with

order

by

clause we can execute sorting on Ignite side.
 But it seems that currently Spark doesn’t have any way to

tell

that partitions already sorted.


 3. *Key filter* If one make query with `WHERE key = XXX` or

`WHERE

key IN (X, Y, Z)`, we can reduce number of partitions.
 And query only partitions that store certain key values.
 Is this kind of optimization already built in Ignite or I

should

implement it by myself?

May be, there is any other way to make queries run faster?

[1] https://spark.apache.org/docs/latest/sql-programming-guide.

html

[2] https://github.com/apache/ignite/pull/2742









--
Nikolay Izhikov
nizhikov@gmail.com

Re: TC issues. IGNITE-3084. Spark Data Frame API

[Николай Ижиков]

Valentin,

> Oh, so this is because of upgrade to 2.2.0?

Yes, we should upgrade spark module to jdk1.8 because switching to spark 2.2.0.

> least we should consider not dropping previous version yet.

Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.

The only thing we can't have is IgniteCatalog.

Currently, in my PR I include IgniteRDD and Ignite Data Frame in spark_2.10 
module
So we doesn't have to drop spark 2.1 completely.

https://github.com/apache/ignite/pull/2742

30.11.2017 02:09, Valentin Kulichenko пишет:

Nikolay,

Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should upgrade in the first place, or at least we should consider not dropping previous version yet. I sent a message to the original 
thread about upgrade, let's decide there and then come back to this issue.


-Val

On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>> wrote:

Valentin,

For now `Ignite RDD` build runs on jdk1.7.
We need to update it to jdk1.8.

I wrote the whole versions numbers to be clear:

1. Current master - Spark version is 2.1.0.
         So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.

2. My branch -
         `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
         `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be 
changed to run on jdk1.8*


29.11.2017 03:27, Valentin Kulichenko пишет:

Nikolay,

If Spark requires Java 8, then I guess we have no choice. How TC is 
configured at the moment? My understanding is that Spark related suites are 
successfully executed there, so is there an issue?

-Val

On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>> wrote:

     Hello, Valentin.

         Added '-Dscala-2.10' to the build config. Let me know if it 
helps.


     Yes, it helps. Thank you!
     Now, 'Ignite RDD spark 2_10' succeed for my branch.


         Do you mean that IgniteRDD does not compile on JDK7? If yes, 
do we know the reason? I don't think switching it to JDK8 is a solution as it 
should work with both.


     I mean that latest version of spark doesn't support jdk7.

http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/> 
<http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>>

     "Spark runs on Java 8+..."
     "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
     "Note that support for Java 7... were removed as of Spark 2.2.0"
     "Note that support for Scala 2.10 is deprecated..."

     Moreover, We can't have IgniteCatalog for spark 2.1.
     Please, see my explanation in jira ticket -


https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523

<https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
     
<https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523

<https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>

     Do you see any options to support jdk7 for spark module?

     > I think all tests should be executed on TC. Can you check if 
they work and add them to corresponding suites

     OK, I file a ticket and try to fix it shortly.

https://issues.apache.org/jira/browse/IGNITE-7042 
<https://issues.apache.org/jira/browse/IGNITE-7042> 
<https://issues.apache.org/jira/browse/IGNITE-7042
<https://issues.apache.org/jira/browse/IGNITE-7042>>

     28.11.2017 03:33, Valentin Kulichenko пишет:

         Hi Nikolay,

         Please see my responses inline.

         -Val

         On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com> 
<mailto:nizhikov@gmail.com <mailto:nizhikov@gmail.com>>>> wrote:

              Hello, guys.

              I have some issues on TC with my PR [1] for 
IGNITE-3084(Spark Data Frame API).
              Can you, please, help me:


              1. `Ignite

Re: Optimization of SQL queries from Spark Data Frame to Ignite

[Николай Ижиков]

Valentin,

> process the AST generated by Spark and convert it to Ignite SQL...
> Does it make sense to you?

Yes.
I think it is a great approach.

Let's implement such feature as the second step of Data Frame integration.

2017-11-29 3:23 GMT+03:00 Valentin Kulichenko <valentin.kuliche...@gmail.com
>:

> Nikolay,
>
> Custom strategy allows to fully process the AST generated by Spark and
> convert it to Ignite SQL, so there will be no execution on Spark side at
> all. This is what we are trying to achieve here. Basically, one will be
> able to use DataFrame API to execute queries directly on Ignite. Does it
> make sense to you?
>
> I would recommend you to take a look at MemSQL implementation which does
> similar stuff: https://github.com/memsql/memsql-spark-connector
>
> Note that this approach will work only if all relations included in AST are
> Ignite tables. Otherwise, strategy should return null so that Spark falls
> back to its regular mode. Ignite will be used as regular data source in
> this case, and probably it's possible to implement some optimizations here
> as well. However, I never investigated this and it seems like another
> separate discussion.
>
> -Val
>
> On Tue, Nov 28, 2017 at 9:54 AM, Николай Ижиков <nizhikov@gmail.com>
> wrote:
>
> > Hello, guys.
> >
> > I have implemented basic support of Spark Data Frame API [1], [2] for
> > Ignite.
> > Spark provides API for a custom strategy to optimize queries from spark
> to
> > underlying data source(Ignite).
> >
> > The goal of optimization(obvious, just to be on the same page):
> > Minimize data transfer between Spark and Ignite.
> > Speedup query execution.
> >
> > I see 3 ways to optimize queries:
> >
> > 1. *Join Reduce* If one make some query that join two or more
> > Ignite tables, we have to pass all join info to Ignite and transfer to
> > Spark only result of table join.
> > To implement it we have to extend current implementation with new
> > RelationProvider that can generate all kind of joins for two or more
> tables.
> > We should add some tests, also.
> > The question is - how join result should be partitioned?
> >
> >
> > 2. *Order by* If one make some query to Ignite table with order
> by
> > clause we can execute sorting on Ignite side.
> > But it seems that currently Spark doesn’t have any way to tell
> > that partitions already sorted.
> >
> >
> > 3. *Key filter* If one make query with `WHERE key = XXX` or
> `WHERE
> > key IN (X, Y, Z)`, we can reduce number of partitions.
> > And query only partitions that store certain key values.
> > Is this kind of optimization already built in Ignite or I should
> > implement it by myself?
> >
> > May be, there is any other way to make queries run faster?
> >
> > [1] https://spark.apache.org/docs/latest/sql-programming-guide.html
> > [2] https://github.com/apache/ignite/pull/2742
> >
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

Re: Optimization of SQL queries from Spark Data Frame to Ignite

[Николай Ижиков]

Hello, Vladimir.

> partition pruning is already implemented in Ignite, so there is no need
to do this on your own.

Spark work with partitioned data set.
It is required to provide data partition information to Spark from custom
Data Source(Ignite).

Can I get information about pruned partitions throw some public API?
Is there a plan or ticket to implement such API?



2017-11-29 10:34 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:

> Nikolay,
>
> Regarding p3. - partition pruning is already implemented in Ignite, so
> there is no need to do this on your own.
>
> On Wed, Nov 29, 2017 at 3:23 AM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Nikolay,
> >
> > Custom strategy allows to fully process the AST generated by Spark and
> > convert it to Ignite SQL, so there will be no execution on Spark side at
> > all. This is what we are trying to achieve here. Basically, one will be
> > able to use DataFrame API to execute queries directly on Ignite. Does it
> > make sense to you?
> >
> > I would recommend you to take a look at MemSQL implementation which does
> > similar stuff: https://github.com/memsql/memsql-spark-connector
> >
> > Note that this approach will work only if all relations included in AST
> are
> > Ignite tables. Otherwise, strategy should return null so that Spark falls
> > back to its regular mode. Ignite will be used as regular data source in
> > this case, and probably it's possible to implement some optimizations
> here
> > as well. However, I never investigated this and it seems like another
> > separate discussion.
> >
> > -Val
> >
> > On Tue, Nov 28, 2017 at 9:54 AM, Николай Ижиков <nizhikov@gmail.com>
> > wrote:
> >
> > > Hello, guys.
> > >
> > > I have implemented basic support of Spark Data Frame API [1], [2] for
> > > Ignite.
> > > Spark provides API for a custom strategy to optimize queries from spark
> > to
> > > underlying data source(Ignite).
> > >
> > > The goal of optimization(obvious, just to be on the same page):
> > > Minimize data transfer between Spark and Ignite.
> > > Speedup query execution.
> > >
> > > I see 3 ways to optimize queries:
> > >
> > > 1. *Join Reduce* If one make some query that join two or more
> > > Ignite tables, we have to pass all join info to Ignite and transfer to
> > > Spark only result of table join.
> > > To implement it we have to extend current implementation with
> new
> > > RelationProvider that can generate all kind of joins for two or more
> > tables.
> > > We should add some tests, also.
> > > The question is - how join result should be partitioned?
> > >
> > >
> > > 2. *Order by* If one make some query to Ignite table with order
> > by
> > > clause we can execute sorting on Ignite side.
> > > But it seems that currently Spark doesn’t have any way to tell
> > > that partitions already sorted.
> > >
> > >
> > > 3. *Key filter* If one make query with `WHERE key = XXX` or
> > `WHERE
> > > key IN (X, Y, Z)`, we can reduce number of partitions.
> > > And query only partitions that store certain key values.
> > > Is this kind of optimization already built in Ignite or I
> should
> > > implement it by myself?
> > >
> > > May be, there is any other way to make queries run faster?
> > >
> > > [1] https://spark.apache.org/docs/latest/sql-programming-guide.html
> > > [2] https://github.com/apache/ignite/pull/2742
> > >
> >
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

Re: TC issues. IGNITE-3084. Spark Data Frame API

[Николай Ижиков]

Valentin,

For now `Ignite RDD` build runs on jdk1.7.
We need to update it to jdk1.8.

I wrote the whole versions numbers to be clear:

1. Current master - Spark version is 2.1.0.
So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.

2. My branch -
`Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
`Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be 
changed to run on jdk1.8*


29.11.2017 03:27, Valentin Kulichenko пишет:

Nikolay,

If Spark requires Java 8, then I guess we have no choice. How TC is configured 
at the moment? My understanding is that Spark related suites are successfully 
executed there, so is there an issue?

-Val

On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>> wrote:

Hello, Valentin.

Added '-Dscala-2.10' to the build config. Let me know if it helps.


Yes, it helps. Thank you!
Now, 'Ignite RDD spark 2_10' succeed for my branch.


Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know 
the reason? I don't think switching it to JDK8 is a solution as it should work 
with both.


I mean that latest version of spark doesn't support jdk7.

http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>

"Spark runs on Java 8+..."
"For the Scala API, Spark 2.2.0 uses Scala 2.11..."
"Note that support for Java 7... were removed as of Spark 2.2.0"
"Note that support for Scala 2.10 is deprecated..."

Moreover, We can't have IgniteCatalog for spark 2.1.
Please, see my explanation in jira ticket -


https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523

<https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>

Do you see any options to support jdk7 for spark module?

> I think all tests should be executed on TC. Can you check if they work 
and add them to corresponding suites

OK, I file a ticket and try to fix it shortly.

https://issues.apache.org/jira/browse/IGNITE-7042 
<https://issues.apache.org/jira/browse/IGNITE-7042>

28.11.2017 03:33, Valentin Kulichenko пишет:

Hi Nikolay,

Please see my responses inline.

-Val

On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov@gmail.com 
<mailto:nizhikov@gmail.com> <mailto:nizhikov@gmail.com 
<mailto:nizhikov@gmail.com>>> wrote:

     Hello, guys.

     I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data 
Frame API).
     Can you, please, help me:


     1. `Ignite RDD spark 2_10` -

     Currently this build runs with following profiles: 
`-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
     That means `scala` profile is activated too for `Ignite RDD spark 
2_10`
     Because `scala` activation is done like [3]:

     ```
                  
                      !scala-2.10
                  
     ```

     I think it a misconfiguration because scala(2.11) shouldn't be 
activated for 2.10 build.
     Am I miss something?

     Can someone edit build property?
              * Add `-scala` to profiles list
              * Or add `-Dscala-2.10` to jvm properties to turn off 
`scala` profile in this build.


Added '-Dscala-2.10' to the build config. Let me know if it helps.


     2. `Ignite RDD` -

     Currently this build run on jvm7 [4].
     As I wrote in my previous mail [5] current version of spark(2.2) 
runs only on jvm8.

     Can someone edit build property to run it on jvm8?


Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know 
the reason? I don't think switching it to JDK8 is a solution as it should work 
with both.


     3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java 
tests [6] existing in `spark` module.
     There are several existing tests written in scala(i.e. scala-test) 
ignored in TC. IgniteRDDSpec [7] for example.
     Is it turned off by a purpose or I miss something?
     Should we run scala-test for spark and spark_2.10 modules?

I think all tests should be executed on TC. Can you check if they work 
and add them to corresponding suites?


     [1] https://github.com/apache/ignite/pull/2742 
<https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 
<https://github.com/apache/ignite/pull/2742>>
     [2] 
https://ci.ignite.apache.org/viewLog.html?buildId=960220=Ignite20Tests_

Optimization of SQL queries from Spark Data Frame to Ignite

[Николай Ижиков]

Hello, guys.

I have implemented basic support of Spark Data Frame API [1], [2] for Ignite.
Spark provides API for a custom strategy to optimize queries from spark to 
underlying data source(Ignite).

The goal of optimization(obvious, just to be on the same page):
Minimize data transfer between Spark and Ignite.
Speedup query execution.

I see 3 ways to optimize queries:

1. *Join Reduce* If one make some query that join two or more Ignite 
tables, we have to pass all join info to Ignite and transfer to Spark only 
result of table join.
To implement it we have to extend current implementation with new 
RelationProvider that can generate all kind of joins for two or more tables.
We should add some tests, also.
The question is - how join result should be partitioned?


2. *Order by* If one make some query to Ignite table with order by 
clause we can execute sorting on Ignite side.
But it seems that currently Spark doesn’t have any way to tell that 
partitions already sorted.


3. *Key filter* If one make query with `WHERE key = XXX` or `WHERE key 
IN (X, Y, Z)`, we can reduce number of partitions.
And query only partitions that store certain key values.
Is this kind of optimization already built in Ignite or I should 
implement it by myself?

May be, there is any other way to make queries run faster?

[1] https://spark.apache.org/docs/latest/sql-programming-guide.html
[2] https://github.com/apache/ignite/pull/2742