New blog post on Apache Ignite in AWS

[Sergey Chugunov]

Hello community,

Recently I published a new blog post on getting started with Apache Ignite
in AWS [1]. I tried to make my example as simple as possible while keeping
it usable.

Let me know if this post is useful for you.

I plan to write several follow-up posts about AWS-specific things but based
on feedback may cover other topics in more detail.

Any feedback is welcome, thank you!

[1] https://www.gridgain.com/node/6247

Re: ML Model persist and reuse

[Alexey Zinoviev]

I totally agree with you, that model saving is missed element in this
framework.
In this release I've experimented with binary format and with a commonly
used format PMML (in scikit-learn and in Spark 1.0)
https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language

This is a link to the example with PMML model parsing (PMML was generated
in Spark)
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/inference/spark/LogRegFromSparkThroughPMMLExample.java

Apache Spark now uses its own project MLeap to save not only models but
whole pipeline with preprocessors.

Thank you for https://onnx.ai/ link (I never used this for model
persistence and it grows very fast), maybe we could create
onnx-ignite-model integration.

The main problem with independent format is that integration between Ignite
ML have very limited support of working with meta-information and it
depends on Ignite SQL internals and couldn't be resolved very fast.

But the our own binary format could be implemented more earlier, it's a
real feature for 2.9 and could be implemented by newbies too (after some
reference examples with the models).

Also, we could collect here (in this thread) our experience with the
different inference interchangeable formats to prepare experimental
examples for 2.9 release to taste them.

If you know any formats that are popular on cloud servers or in production,
please post them here, in this thread.


сб, 18 янв. 2020 г. в 03:47, kencottrell :

> Hello all Apache Ignite ML developers:
>
> I understand currently Ignite can't save a model after training, in such a
> way that the model can be re-imported by another Ignite cluster. Correct me
> if you can save and reload a model but I don't think you can.
>
> Anyway, I'd like to know if you have recommendations on how you can do
> either one of the following:
> 1. convert the Ignite model into an interchangeable format? For example
> there are some emerging standards (such as https://onnx.ai/  for one)  and
> others - have any of you worked with such
>
> 2. if not transform the Ignite model into some standard format, how about
> saving the model into Native persistence, binary serialized format,
> creating
> some kind of handle that can shared with other clusters, and then use this
> to reload the model into a new Ignite session?
>
>
> This question has been asked of me recently, and this would be a good way
> to
> let Apache Ignite ML/DL models participate in a broader enterprise model
> deployment process.
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Ignite to H20 integration

[Alexey Zinoviev]

The current integration could be used for model inference purposes only (on
Ignite data) as presented in h20-ml-inference example.
This features could be available in 2.8

Please have a look to
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/inference/h2o/H2OMojoModelParserExample.java

сб, 18 янв. 2020 г. в 03:53, kencottrell :

> Have any of you performed an H20 integration with Ignite to import an
> extracted feature data set directly as input into Ignite training engine?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Call for presentations for ApacheCon North America 2020 now open

[Saikat Maitra]

Hi Denis,

Thank you for your email. I have submitted a talk about Data Streaming
using Apache Ignite and Apache Flink.

I would also like to co-present on Ignite internals and usecases deep dive.

Regards,
Saikat

On Wed, Jan 22, 2020 at 6:22 PM Denis Magda  wrote:

> Igniters,
>
> I was submitting an Ignite talk to the conference and found out that this
> time ASF created a separate category for Ignite-specific proposals. You can
> see an abstract submitted by me:
> https://drive.google.com/file/d/1woaEOWaIFxN8UIJ7nvFbYoc53mYsUsSN/view?usp=sharing
>
> Who else is ready to submit? It can be a deep-dive about Ignite internals
> or Ignite use case overview. We can co-present. Also, GridGain is ready to
> sponsor your trip if the talk is accepted.
>
>
>
> -
> Denis
>
>
> On Tue, Jan 21, 2020 at 5:22 PM Denis Magda  wrote:
>
>> Ignite folks, just bringing to your attention and encourage anybody
>> interested to submit an Ignite-related session. Both Alex Zinoviev and I
>> presented at ApacheCons the previous year -- the events are worth attending
>> and speaking at.
>>
>> -
>> Denis
>>
>>
>> -- Forwarded message -
>> From: Rich Bowen 
>> Date: Tue, Jan 21, 2020 at 7:06 AM
>> Subject: Call for presentations for ApacheCon North America 2020 now open
>> To: plann...@apachecon.com 
>>
>>
>> Dear Apache enthusiast,
>>
>> (You’re receiving this message because you are subscribed to one or more
>> project mailing lists at the Apache Software Foundation.)
>>
>> The call for presentations for ApacheCon North America 2020 is now open
>> at https://apachecon.com/acna2020/cfp
>>
>> ApacheCon will be held at the Sheraton, New Orleans, September 28th
>> through October 2nd, 2020.
>>
>> As in past years, ApacheCon will feature tracks focusing on the various
>> technologies within the Apache ecosystem, and so the call for
>> presentations will ask you to select one of those tracks, or “General”
>> if the content falls outside of one of our already-organized tracks.
>> These tracks are:
>>
>> Karaf
>> Internet of Things
>> Fineract
>> Community
>> Content Delivery
>> Solr/Lucene (Search)
>> Gobblin/Big Data Integration
>> Ignite
>> Observability
>> Cloudstack
>> Geospatial
>> Graph
>> Camel/Integration
>> Flagon
>> Tomcat
>> Cassandra
>> Groovy
>> Web/httpd
>> General/Other
>>
>> The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).
>>
>> Submit early, submit often, at https://apachecon.com/acna2020/cfp
>>
>> Rich, for the ApacheCon Planners
>>
>

Re: Call for presentations for ApacheCon North America 2020 now open

[Denis Magda]

Igniters,

I was submitting an Ignite talk to the conference and found out that this
time ASF created a separate category for Ignite-specific proposals. You can
see an abstract submitted by me:
https://drive.google.com/file/d/1woaEOWaIFxN8UIJ7nvFbYoc53mYsUsSN/view?usp=sharing

Who else is ready to submit? It can be a deep-dive about Ignite internals
or Ignite use case overview. We can co-present. Also, GridGain is ready to
sponsor your trip if the talk is accepted.



-
Denis


On Tue, Jan 21, 2020 at 5:22 PM Denis Magda  wrote:

> Ignite folks, just bringing to your attention and encourage anybody
> interested to submit an Ignite-related session. Both Alex Zinoviev and I
> presented at ApacheCons the previous year -- the events are worth attending
> and speaking at.
>
> -
> Denis
>
>
> -- Forwarded message -
> From: Rich Bowen 
> Date: Tue, Jan 21, 2020 at 7:06 AM
> Subject: Call for presentations for ApacheCon North America 2020 now open
> To: plann...@apachecon.com 
>
>
> Dear Apache enthusiast,
>
> (You’re receiving this message because you are subscribed to one or more
> project mailing lists at the Apache Software Foundation.)
>
> The call for presentations for ApacheCon North America 2020 is now open
> at https://apachecon.com/acna2020/cfp
>
> ApacheCon will be held at the Sheraton, New Orleans, September 28th
> through October 2nd, 2020.
>
> As in past years, ApacheCon will feature tracks focusing on the various
> technologies within the Apache ecosystem, and so the call for
> presentations will ask you to select one of those tracks, or “General”
> if the content falls outside of one of our already-organized tracks.
> These tracks are:
>
> Karaf
> Internet of Things
> Fineract
> Community
> Content Delivery
> Solr/Lucene (Search)
> Gobblin/Big Data Integration
> Ignite
> Observability
> Cloudstack
> Geospatial
> Graph
> Camel/Integration
> Flagon
> Tomcat
> Cassandra
> Groovy
> Web/httpd
> General/Other
>
> The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).
>
> Submit early, submit often, at https://apachecon.com/acna2020/cfp
>
> Rich, for the ApacheCon Planners
>

Re: Ignite-spring-boot-autoconfigurer

[Николай Ижиков]

Hello, Saikat.

Thank you so much for the review.

I answered your questions and resolve all the comments.
Please, take a look, one more time.

> 22 янв. 2020 г., в 07:58, Saikat Maitra  написал(а):
> 
> Hi Nikolay,
> 
> I have reviewed the PR and shared comments.
> 
> Please let me know if you have any feedback.
> 
> Regards,
> Saikat
> 
> On Mon, Jan 20, 2020 at 2:42 PM Николай Ижиков  wrote:
> 
>> Hello, Saikat.
>> 
>> Thanks, for feedback.
>> 
>> I raised a PR [1] to `ignite-extensions`.
>> 
>> You can find description of the new module below(examples can be found at
>> [2]):
>> 
>> Module provides the ability to integrate `Ignite` into you spring-boot
>> application with zero(or minimal) configuration.
>> 
>> After you add this module as a dependency to your spring-boot application
>> `Ignite` node will be configured and injected into `BeanFactory`.
>> 
>> Algorithm to configure `Ignite` is the following:
>>  1. If `IgniteConfiguration` bean exists in the `BeanFactory` it will be
>> used.
>>  2. If `IgniteConfiguration` bean doesn't exist following rules are
>> applied:
>>2.1. Default `Ignite` configuration created.
>>2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will be
>> used to customize `IgniteConfiguration`.
>> If a user wants to set custom SPI instances or similar hardcoded
>> values
>> one should do it with `IgniteConfigurer` implementation.
>>2.3  Application properties applied to `IgniteConfiguration`. Prefix
>> for the properties is `ignite`.
>> 
>> 
>> [1] https://github.com/apache/ignite-extensions/pull/6
>> [2] https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example
>> 
>> 
>>> 18 янв. 2020 г., в 06:44, Saikat Maitra 
>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> Thank you for your email. As part of Ignite Extensions migration we are
>> migrating Ignite Extensions to following repo.
>>> 
>>> https://github.com/apache/ignite-extensions
>>> 
>>> We have added flink and pub-sub modules and few additional modules are
>> open in PR.
>>> 
>>> You can refer to this PR to see how we are migrating the modules
>> https://github.com/apache/ignite-extensions/pull/5
>>> 
>>> I wanted to connect and discuss the changes to understand the spring
>> boot auto configure feature. We currently have an ignite spring module that
>> allows resource injection capabilities and provides a parser for Spring
>> based xml configuration files. Can you please review and share if the
>> changes you are proposing can be added as part of Ignite spring module or
>> it make sense to make it a separate spring boot auto configure module.
>>> 
>>> https://github.com/apache/ignite/tree/master/modules/spring
>>> 
>>> Regards,
>>> Saikat
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков 
>> wrote:
>>> Tests added.
>>> Please, review.
>>> 
>>> Saikat, can you help with this PR [1]?
>>> 
>>> I think it should be added as a separate module as we do with the flink
>> integration.
>>> Can you help me with it?
>>> Do we have some how-to for it?
>>> 
>>> [1] https://github.com/apache/ignite/pull/7237
>>> 
 16 янв. 2020 г., в 16:51, Николай Ижиков 
>> написал(а):
 
 Hello, Denis.
 
 Thanks, for the feedback.
 
 Alexey, it seems, PR is ready to be reviewed, but I need some time(a
>> day or two) to write tests.
 You can start with the core code review if you wish.
 
 Here are autoconfigurer requirements:
 
 1. Start usage of Ignite with minimal(or zero) configuration.
 2. Configure Ignite configuration properties with the standard spring
>> boot application properties.
 3. Configure Ignite SPI implementation and so on that can’t be
>> configured via #2.
 
 After some consultation with the Spring experts from the
>> community(Maxim Stepachev thanks for the idea)
 I updated the PR with the logic described below:
 
 1. To enable Ignite auto-configuration user should add
>> `org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to dependencies.
   After it Ignite node will be started during spring-boot application
>> startup.
 
 2. IgniteConfiguration initialization logic:
 
 2.1 If {@link IgniteConfiguration} bean exists in {@link BeanFactory}
>> it will be used for the node start.
 2.2 If {@link IgniteConfiguration} bean doesn't exist following rules
>> are applied:
 * Newly introducer IgniteConfigurer bean will be used to customize an
>> empty IgniteConfiguration instance.
   If a user wants to set custom SPI instances or similar hardcoded
>> values one should do it IgniteConfigurer implementation.
 
 * Application properties will override config values. Prefix for
>> properties names is "ignite».
 
 PS. Similar logic applied for the second module -
>> `org.apache.ignite:spring-boot-ignite-client-autoconfigure:2.9.0`.
 It provides the same features but for the autoconfiguration of the
>> IgniteClient
 
 
> 15 

Re: Add user attributes to thin clients

[Pavel Tupitsyn]

I've looked through the PR more closely, trying to understand the use case,
and there are some Java-specific things going on (left a comment).
Please keep in mind that we have thin clients in Python, Node.js, C++, C#.
The protocol must be language-agnostic. If we add some features there,
let's make sure they are usable from anywhere.

On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn  wrote:

> The approach with UserAttributes map looks dirty to me and raises
> questions:
>
> - Why is UserAttributes property related to authentication?
> - UserAttributes name implies that users can put there anything they want,
> but what for? What are those additional use cases?
>
> I think we should focus on a specific problem at hand and avoid
> unnecessary future-proofing.
> What are current and potential future custom authenticators and what kind
> of data do they need?
>
> On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev 
> wrote:
>
>> I think we should add this. It will provide an extra level of security.
>> This approach is used in many products, for example in AWS (MFA). [1]
>>
>> [1]
>> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
>>
>> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
>> >
>> > Hi, Pavel!
>> >
>> > Sometimes single authentication factor is not enough. Attributes
>> proposed
>> > allow to add extra factors flexibly.
>> >
>> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
>> >
>> > > Token can be sent instead of a password (like git works with GitHub
>> > > tokens).
>> > >
>> > > For now I don't see a reason to include attributes into the handshake
>> > > message.
>> > >
>> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Hello!
>> > > >
>> > > > One does not send security certificate as attribute. The only way to
>> > > obtain
>> > > > peer security certificate is to ask SSL engine to provide it.
>> > > >
>> > > > Nevertheless, I can see how it can be useful with e.g. Kerberos,
>> which is
>> > > > token-based IIRC.
>> > > >
>> > > > Regards,
>> > > > --
>> > > > Ilya Kasnacheev
>> > > >
>> > > >
>> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov > >:
>> > > >
>> > > > > This map is something like user object from `SecurityCredentials`.
>> > > > > Sometimes login and password are not enough for security checks.
>> For
>> > > > > example, we can send security certificate and validate it inside
>> > > > > authenticator.
>> > > > >
>> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
>> > > > >
>> > > > > > Hi Dmitrii,
>> > > > > >
>> > > > > > Can you please explain your use case?
>> > > > > > I'm not sure I'm getting what is the motivation of this change.
>> > > > > >
>> > > > > > Best Regards,
>> > > > > > Igor
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn <
>> ptupit...@apache.org
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Dmitrii,
>> > > > > > >
>> > > > > > > Honestly, I could not grasp the problem, can you explain it
>> in more
>> > > > > > detail?
>> > > > > > > What do we solve by adding a map with arbitrary stuff to the
>> client
>> > > > > > > protocol handshake?
>> > > > > > >
>> > > > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
>> > > > somefire...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello, Igniters!
>> > > > > > > >
>> > > > > > > > I want to add the possibility of sending user defined
>> attributes
>> > > > from
>> > > > > > > thin
>> > > > > > > > clients. And check them inside custom authenticator during
>> > > > handshake
>> > > > > > [1].
>> > > > > > > >
>> > > > > > > > There is an issue in hardcoded binary writer for JDBC and
>> > > > > > `IgniteClient`.
>> > > > > > > > This writer searches for a classes in the JDK and
>> > > > > > > > META-INF/classnames.properties, and tries to sync
>> notdeclared
>> > > > classes
>> > > > > > > with
>> > > > > > > > cluster. But fails because current classloading uses
>> discovery.
>> > > > > > > >
>> > > > > > > > I'd like to keep this writer and allow only primitive types
>> and
>> > > > > > `String`
>> > > > > > > > for user attributes to prevent unexpected fails. I think it
>> is
>> > > > better
>> > > > > > > than
>> > > > > > > > changing writer to one with heavy classloading.
>> > > > > > > >
>> > > > > > > > Is it ok to restrict thin attributes to primitives and
>> 'String'?
>> > > > > > > >
>> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>
>

Re: Add user attributes to thin clients

[Pavel Tupitsyn]

The approach with UserAttributes map looks dirty to me and raises questions:

- Why is UserAttributes property related to authentication?
- UserAttributes name implies that users can put there anything they want,
but what for? What are those additional use cases?

I think we should focus on a specific problem at hand and avoid unnecessary
future-proofing.
What are current and potential future custom authenticators and what kind
of data do they need?

On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev 
wrote:

> I think we should add this. It will provide an extra level of security.
> This approach is used in many products, for example in AWS (MFA). [1]
>
> [1]
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
>
> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
> >
> > Hi, Pavel!
> >
> > Sometimes single authentication factor is not enough. Attributes proposed
> > allow to add extra factors flexibly.
> >
> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
> >
> > > Token can be sent instead of a password (like git works with GitHub
> > > tokens).
> > >
> > > For now I don't see a reason to include attributes into the handshake
> > > message.
> > >
> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > One does not send security certificate as attribute. The only way to
> > > obtain
> > > > peer security certificate is to ask SSL engine to provide it.
> > > >
> > > > Nevertheless, I can see how it can be useful with e.g. Kerberos,
> which is
> > > > token-based IIRC.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov :
> > > >
> > > > > This map is something like user object from `SecurityCredentials`.
> > > > > Sometimes login and password are not enough for security checks.
> For
> > > > > example, we can send security certificate and validate it inside
> > > > > authenticator.
> > > > >
> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> > > > >
> > > > > > Hi Dmitrii,
> > > > > >
> > > > > > Can you please explain your use case?
> > > > > > I'm not sure I'm getting what is the motivation of this change.
> > > > > >
> > > > > > Best Regards,
> > > > > > Igor
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn <
> ptupit...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Dmitrii,
> > > > > > >
> > > > > > > Honestly, I could not grasp the problem, can you explain it in
> more
> > > > > > detail?
> > > > > > > What do we solve by adding a map with arbitrary stuff to the
> client
> > > > > > > protocol handshake?
> > > > > > >
> > > > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> > > > somefire...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello, Igniters!
> > > > > > > >
> > > > > > > > I want to add the possibility of sending user defined
> attributes
> > > > from
> > > > > > > thin
> > > > > > > > clients. And check them inside custom authenticator during
> > > > handshake
> > > > > > [1].
> > > > > > > >
> > > > > > > > There is an issue in hardcoded binary writer for JDBC and
> > > > > > `IgniteClient`.
> > > > > > > > This writer searches for a classes in the JDK and
> > > > > > > > META-INF/classnames.properties, and tries to sync notdeclared
> > > > classes
> > > > > > > with
> > > > > > > > cluster. But fails because current classloading uses
> discovery.
> > > > > > > >
> > > > > > > > I'd like to keep this writer and allow only primitive types
> and
> > > > > > `String`
> > > > > > > > for user attributes to prevent unexpected fails. I think it
> is
> > > > better
> > > > > > > than
> > > > > > > > changing writer to one with heavy classloading.
> > > > > > > >
> > > > > > > > Is it ok to restrict thin attributes to primitives and
> 'String'?
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

Re: AWS EBS Discovery: Contributor Wanted

[Denis Magda]

Hi Emmanouil,

Thanks for preparing a pull-request for Application Load Balancer:
https://issues.apache.org/jira/browse/IGNITE-8617

Igniters, who is willing to step in as a primary reviewer?

As for automated testing on AWS, are you aware of any sponsorship program
of AWS for open source projects of our kind? It will be ideal to have real
testing on AWS but someone needs to pay.

-
Denis


On Sun, Jan 19, 2020 at 6:45 AM Emmanouil Gkatziouras 
wrote:

> Hi all!
>
> I have spinned up an Application Load Balancer and an autoscaling group on
> AWS and the Ignite discovery using TcpDiscoveryAlbIpFinder works as
> expected.
>
>- On startup nodes discover each other.
>- On ec2 node down, connection is lost and the cluster decreases.
>- On an extra node addition the cluster size increases
>
> This contribution is essential since the Previous ELB based discovery uses
> the Classic Load Balancer which is still available however
> AWS advices users to use the Application one. [1]
> While my pull request gets reviewed I will also have a look at
> the IGNITE-12398 [2] issue which has to do with the S3 discovery.
> Another idea would also be to implement a `TCP Load Balancer based`
> discovery.
>
> In order to test this issue and future ones I implemented some terraform
> scripts (which I shall use for other issues too) [3].
> If some automated e2e testing on AWS is being considered they might be of
> value.
> I can help on implementing those tests by provisioning the infrastructure
> in an automated way and validate the discovery.
>
> [1]
>
> https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/migrate-to-application-load-balancer.html
> [2] https://issues.apache.org/jira/browse/IGNITE-12398
> [3] https://github.com/gkatzioura/ignite-aws-deploy
>
> Kind regards,
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Tue, 14 Jan 2020 at 22:22, Denis Magda  wrote:
>
> > Hi Emmanouil,
> >
> > Agree, let's check that the IP finder functions normally in the cloud
> > environment and the mock tests can be used for regular testing on Team
> > City. That's the way we tested other environment-specific IP finders
> > including the Kubernetes one.
> >
> > Let us know once the IP finder is tested on AWS and then we can proceed
> > with the review.
> >
> > -
> > Denis
> >
> >
> > On Tue, Jan 14, 2020 at 2:47 AM Emmanouil Gkatziouras <
> > gkatzio...@gmail.com>
> > wrote:
> >
> > > Hi all!
> > >
> > > With regards to the `Node Discovery Using AWS Application ELB` issue
> [1]
> > > I made this pull request to fix the merge conflicts [2].
> > > Will also check if the concerns described on the mailing list topic
> > > `Volunteer needed: AWS Elastic Load Balancer IP Finders implemented`
> are
> > > addressed [3]
> > > Since I did not test it on an AWS infrastructure the next step would be
> > to
> > > spin up some AWS resources and test it.
> > > By doing so it will help me on checking the Amazon S3 Based Discovery
> > issue
> > > [4] .
> > > I will try to make the deployment as infrastructure as code, and see if
> > > this can be of value for the ignite project and testing.
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8617
> > > [2]  https://github.com/apache/ignite/pull/7247
> > > [3]
> > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Volunteer-needed-AWS-Elastic-Load-Balancer-IP-Finders-implemented-td33847.html#a39793
> > > [4] https://issues.apache.org/jira/browse/IGNITE-12398
> > >
> > > >
> > >
> >
>

Re: Ignite and Pauseless JVMs for Low-Latency Scenarious: Meetup with Azul

[Denis Magda]

Igniters,

Just a reminder for those who live in the San Francisco Bay Area. We're
meeting tomorrow at Azul HQ.

-
Denis


On Thu, Jan 9, 2020 at 12:01 PM Denis Magda  wrote:

> Igniters,
>
> Some of you should have come across an article by Simon Ritter describing
> how Ignite and Azul Zing JVM are used together to power low-latency use
> cases:
> https://www.azul.com/igniting-in-memory-performance-with-gridgain-and-zing/
>
> Several of our community members including Pavel Vinokurov and Stan
> Lukyanov were involved in that project, and this time we are gathering
> together with Silicon Valley Java Performance meetup group to results in
> details:
> https://www.meetup.com/Bay-Area-In-Memory-Computing/events/267751131
>
> The session is to be led by Gil Tene, Azul Systems CTO, Java RockStar and
> Java Champion, who will cover JVM aspects of the solution. While I will
> join Gil to explain Ignite internals.
>
> RSVP and come over!
>
> -
> Denis
>

Re: Add user attributes to thin clients

[Nikita Amelchev]

I think we should add this. It will provide an extra level of security.
This approach is used in many products, for example in AWS (MFA). [1]

[1] 
https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication

ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
>
> Hi, Pavel!
>
> Sometimes single authentication factor is not enough. Attributes proposed
> allow to add extra factors flexibly.
>
> ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
>
> > Token can be sent instead of a password (like git works with GitHub
> > tokens).
> >
> > For now I don't see a reason to include attributes into the handshake
> > message.
> >
> > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev  > >
> > wrote:
> >
> > > Hello!
> > >
> > > One does not send security certificate as attribute. The only way to
> > obtain
> > > peer security certificate is to ask SSL engine to provide it.
> > >
> > > Nevertheless, I can see how it can be useful with e.g. Kerberos, which is
> > > token-based IIRC.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov :
> > >
> > > > This map is something like user object from `SecurityCredentials`.
> > > > Sometimes login and password are not enough for security checks. For
> > > > example, we can send security certificate and validate it inside
> > > > authenticator.
> > > >
> > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> > > >
> > > > > Hi Dmitrii,
> > > > >
> > > > > Can you please explain your use case?
> > > > > I'm not sure I'm getting what is the motivation of this change.
> > > > >
> > > > > Best Regards,
> > > > > Igor
> > > > >
> > > > >
> > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn  > >
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitrii,
> > > > > >
> > > > > > Honestly, I could not grasp the problem, can you explain it in more
> > > > > detail?
> > > > > > What do we solve by adding a map with arbitrary stuff to the client
> > > > > > protocol handshake?
> > > > > >
> > > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> > > somefire...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, Igniters!
> > > > > > >
> > > > > > > I want to add the possibility of sending user defined attributes
> > > from
> > > > > > thin
> > > > > > > clients. And check them inside custom authenticator during
> > > handshake
> > > > > [1].
> > > > > > >
> > > > > > > There is an issue in hardcoded binary writer for JDBC and
> > > > > `IgniteClient`.
> > > > > > > This writer searches for a classes in the JDK and
> > > > > > > META-INF/classnames.properties, and tries to sync notdeclared
> > > classes
> > > > > > with
> > > > > > > cluster. But fails because current classloading uses discovery.
> > > > > > >
> > > > > > > I'd like to keep this writer and allow only primitive types and
> > > > > `String`
> > > > > > > for user attributes to prevent unexpected fails. I think it is
> > > better
> > > > > > than
> > > > > > > changing writer to one with heavy classloading.
> > > > > > >
> > > > > > > Is it ok to restrict thin attributes to primitives and 'String'?
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >



-- 
Best wishes,
Amelchev Nikita

Re: Add user attributes to thin clients

[Andrey Kuznetsov]

Hi, Pavel!

Sometimes single authentication factor is not enough. Attributes proposed
allow to add extra factors flexibly.

ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :

> Token can be sent instead of a password (like git works with GitHub
> tokens).
>
> For now I don't see a reason to include attributes into the handshake
> message.
>
> On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > One does not send security certificate as attribute. The only way to
> obtain
> > peer security certificate is to ask SSL engine to provide it.
> >
> > Nevertheless, I can see how it can be useful with e.g. Kerberos, which is
> > token-based IIRC.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov :
> >
> > > This map is something like user object from `SecurityCredentials`.
> > > Sometimes login and password are not enough for security checks. For
> > > example, we can send security certificate and validate it inside
> > > authenticator.
> > >
> > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> > >
> > > > Hi Dmitrii,
> > > >
> > > > Can you please explain your use case?
> > > > I'm not sure I'm getting what is the motivation of this change.
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > >
> > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > Hi Dmitrii,
> > > > >
> > > > > Honestly, I could not grasp the problem, can you explain it in more
> > > > detail?
> > > > > What do we solve by adding a map with arbitrary stuff to the client
> > > > > protocol handshake?
> > > > >
> > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> > somefire...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello, Igniters!
> > > > > >
> > > > > > I want to add the possibility of sending user defined attributes
> > from
> > > > > thin
> > > > > > clients. And check them inside custom authenticator during
> > handshake
> > > > [1].
> > > > > >
> > > > > > There is an issue in hardcoded binary writer for JDBC and
> > > > `IgniteClient`.
> > > > > > This writer searches for a classes in the JDK and
> > > > > > META-INF/classnames.properties, and tries to sync notdeclared
> > classes
> > > > > with
> > > > > > cluster. But fails because current classloading uses discovery.
> > > > > >
> > > > > > I'd like to keep this writer and allow only primitive types and
> > > > `String`
> > > > > > for user attributes to prevent unexpected fails. I think it is
> > better
> > > > > than
> > > > > > changing writer to one with heavy classloading.
> > > > > >
> > > > > > Is it ok to restrict thin attributes to primitives and 'String'?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-12569) Can't set serialized enum to a BinaryObject's field

[Konstantin Orlov (Jira)]

Konstantin Orlov created IGNITE-12569:
-

 Summary: Can't set serialized enum to a BinaryObject's field
 Key: IGNITE-12569
 URL: https://issues.apache.org/jira/browse/IGNITE-12569
 Project: Ignite
  Issue Type: Bug
  Components: binary
Reporter: Konstantin Orlov
Assignee: Konstantin Orlov


The deserialization of the BinaryObject fails since a serialized representation 
of the enum was set to a field instead of the enum itself. Because of this 
issue there is no way to work with enums without a corresponding class.

Error message thrown during the deserialization:

{noformat}[18:03:06,159][ERROR][main][BinaryContext] Failed to deserialize 
object [typeName=binary.BinaryEnumExample$TestClass]
org.apache.ignite.binary.BinaryObjectException: Failed to read field 
[name=theEnum]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:191)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
 [classes/:?]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
 [classes/:?]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
 [classes/:?]
at 
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:794)
 [classes/:?]
at 
org.apache.ignite.internal.binary.BinaryObjectImpl.deserialize(BinaryObjectImpl.java:636)
 [classes/:?]
at binary.BinaryEnumExample.main(BinaryEnumExample.java:20) [classes/:?]
Caused by: org.apache.ignite.binary.BinaryObjectException: Unexpected field 
type [pos=24, expected=Enum, actual=Enum]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.checkFlagNoHandles(BinaryReaderExImpl.java:1677)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum0(BinaryReaderExImpl.java:1403)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum(BinaryReaderExImpl.java:1387)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.readFixedType(BinaryFieldAccessor.java:885)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
 ~[classes/:?]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
 ~[classes/:?]
... 6 more
Exception in thread "main" class 
org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object 
[typeName=binary.BinaryEnumExample$TestClass]
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:926)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
at 
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:794)
at 
org.apache.ignite.internal.binary.BinaryObjectImpl.deserialize(BinaryObjectImpl.java:636)
at binary.BinaryEnumExample.main(BinaryEnumExample.java:20)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read 
field [name=theEnum]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:191)
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
... 5 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Unexpected 
field type [pos=24, expected=Enum, actual=Enum]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.checkFlagNoHandles(BinaryReaderExImpl.java:1677)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum0(BinaryReaderExImpl.java:1403)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum(BinaryReaderExImpl.java:1387)
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.readFixedType(BinaryFieldAccessor.java:885)
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
... 6 more{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Pavel Tupitsyn]

Good idea about pre-release build of ignite-2.8 branch.
However, I would not name it `rc`, since it is not really a release
candidate. Make it `pre0` or something like that.

For Ignite.NET I've uploaded pre-release NuGet packages built from current
ignite-2.8 branch:
https://www.nuget.org/packages/Apache.Ignite/2.8.0-alpha20200122


On Wed, Jan 22, 2020 at 3:09 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have committed the bumping of essential dependencies' versions:
> https://issues.apache.org/jira/browse/IGNITE-12540
>
> Would you mind including this change into the scope of 2.8? No point of
> shipping known problematic JARs in our deliverable.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 22 янв. 2020 г. в 14:00, Maxim Muzafarov :
>
> > Alexey,
> >
> > Sure, I've just thought about it too a few days ago.
> >
> > On Wed, 22 Jan 2020 at 12:09, Anton Vinogradov  wrote:
> > >
> > > Good Idea, this will also check that the release process is alive.
> > >
> > > On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > Folks, Maxim,
> > > >
> > > > Do you mind if I build the current state of ignite-2.8 branch and
> > upload a
> > > > maven staging as rc0 (step 4.3.2 of the release process)? I want run
> > some
> > > > tests for the fixes that are already included to the branch.
> > > >
> > > > вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
> > > >
> > > > > Folks,
> > > > >
> > > > >
> > > > > I think both of these issues [1] [2] are critical to 2.8 release
> and
> > > > > we must include them.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > Excessive AtomicLong instantiations lead to GC pressure.
> > > > >
> > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > > > > Pages list caching can cause IgniteOOME when the checkpoint is
> > > > > triggered by "too many dirty pages" reason.
> > > > >
> > > > >
> > > > > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov <
> plehanov.a...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Guys,
> > > > > >
> > > > > > There is an issue [1] caused by page list caching [2], which also
> > > > affects
> > > > > > 2.8 release. IgniteOutOfMemoryException can be thrown in some
> cases
> > > > (data
> > > > > > region is small, a checkpoint is triggered by "too many dirty
> > pages"
> > > > > reason
> > > > > > and pages list cache is rather big).
> > > > > > The fix is ready and merged to master, I suggest to include this
> > fix to
> > > > > 2.8
> > > > > > release. What do you think?
> > > > > >
> > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > > > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > > > > >
> > > > > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com>:
> > > > > >
> > > > > > > Maxim,
> > > > > > >
> > > > > > > I took a quick look at IGNITE-12456 and I am not sure it's
> about
> > data
> > > > > > > corruption. In the attached logs blocked system threads are
> > reported,
> > > > > > > however, there is no enough information to investigate the
> issue
> > (the
> > > > > full
> > > > > > > thread dump was not attached). I asked the ticket creator to
> > attach
> > > > > missing
> > > > > > > pieces.
> > > > > > >
> > > > > > > Should we consider moving this ticket to a next release?
> > > > > > >
> > > > > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> > > > >  > > > > > > >:
> > > > > > >
> > > > > > > >
> > > > > > > > Maxim, performance fix issue [1] already in master, if no
> > > > > objections, can
> > > > > > > > u merge it into 2.8 ? Thanks !
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > > > >
> > > > > > > > >Igniters,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >Here is the actual list of BLOCKER release issues:
> > > > > > > > >
> > > > > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load
> > test
> > > > > > > > >*[Unassigned]* OPEN
> > > > > > > > >IGNITE-12489 Error during purges by expiration: Unknown page
> > type*
> > > > > > > > >[Unassigned]* OPEN
> > > > > > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml
> > config
> > > > > > > > >*[Unassigned]* OPEN
> > > > > > > > >
> > > > > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based
> Discovery)
> > > > Nodes
> > > > > > > > getting
> > > > > > > > >down [Emmanouil Gkatziouras] OPEN
> > > > > > > > >IGNITE-9184 Cluster hangs during concurrent node client and
> > server
> > > > > nodes
> > > > > > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > > > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement
> > [Nikolay
> > > > > > > Izhikov]
> > > > > > > > >Blocker IN PROGRESS
> > > > > > > > >
> > > > > > > > >IGNITE-12227 Default auto-adjust baseline enabled flag
> > calculated
> > > > > > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > > > > > >IGNITE-12470 Pme-free switch feature should be deactivatable
> > > > 

Re: Add user attributes to thin clients

[Pavel Tupitsyn]

Token can be sent instead of a password (like git works with GitHub tokens).

For now I don't see a reason to include attributes into the handshake
message.

On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> One does not send security certificate as attribute. The only way to obtain
> peer security certificate is to ask SSL engine to provide it.
>
> Nevertheless, I can see how it can be useful with e.g. Kerberos, which is
> token-based IIRC.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov :
>
> > This map is something like user object from `SecurityCredentials`.
> > Sometimes login and password are not enough for security checks. For
> > example, we can send security certificate and validate it inside
> > authenticator.
> >
> > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> >
> > > Hi Dmitrii,
> > >
> > > Can you please explain your use case?
> > > I'm not sure I'm getting what is the motivation of this change.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Hi Dmitrii,
> > > >
> > > > Honestly, I could not grasp the problem, can you explain it in more
> > > detail?
> > > > What do we solve by adding a map with arbitrary stuff to the client
> > > > protocol handshake?
> > > >
> > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> somefire...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello, Igniters!
> > > > >
> > > > > I want to add the possibility of sending user defined attributes
> from
> > > > thin
> > > > > clients. And check them inside custom authenticator during
> handshake
> > > [1].
> > > > >
> > > > > There is an issue in hardcoded binary writer for JDBC and
> > > `IgniteClient`.
> > > > > This writer searches for a classes in the JDK and
> > > > > META-INF/classnames.properties, and tries to sync notdeclared
> classes
> > > > with
> > > > > cluster. But fails because current classloading uses discovery.
> > > > >
> > > > > I'd like to keep this writer and allow only primitive types and
> > > `String`
> > > > > for user attributes to prevent unexpected fails. I think it is
> better
> > > > than
> > > > > changing writer to one with heavy classloading.
> > > > >
> > > > > Is it ok to restrict thin attributes to primitives and 'String'?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-12568) MessageFactory implementations refactoring

[Andrey N. Gura (Jira)]

Andrey N. Gura created IGNITE-12568:
---

 Summary: MessageFactory implementations refactoring
 Key: IGNITE-12568
 URL: https://issues.apache.org/jira/browse/IGNITE-12568
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrey N. Gura
Assignee: Andrey N. Gura






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Add user attributes to thin clients

[Ilya Kasnacheev]

Hello!

One does not send security certificate as attribute. The only way to obtain
peer security certificate is to ask SSL engine to provide it.

Nevertheless, I can see how it can be useful with e.g. Kerberos, which is
token-based IIRC.

Regards,
-- 
Ilya Kasnacheev


ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov :

> This map is something like user object from `SecurityCredentials`.
> Sometimes login and password are not enough for security checks. For
> example, we can send security certificate and validate it inside
> authenticator.
>
> ср, 22 янв. 2020 г., 17:16 Igor Sapego :
>
> > Hi Dmitrii,
> >
> > Can you please explain your use case?
> > I'm not sure I'm getting what is the motivation of this change.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Hi Dmitrii,
> > >
> > > Honestly, I could not grasp the problem, can you explain it in more
> > detail?
> > > What do we solve by adding a map with arbitrary stuff to the client
> > > protocol handshake?
> > >
> > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov 
> > > wrote:
> > >
> > > > Hello, Igniters!
> > > >
> > > > I want to add the possibility of sending user defined attributes from
> > > thin
> > > > clients. And check them inside custom authenticator during handshake
> > [1].
> > > >
> > > > There is an issue in hardcoded binary writer for JDBC and
> > `IgniteClient`.
> > > > This writer searches for a classes in the JDK and
> > > > META-INF/classnames.properties, and tries to sync notdeclared classes
> > > with
> > > > cluster. But fails because current classloading uses discovery.
> > > >
> > > > I'd like to keep this writer and allow only primitive types and
> > `String`
> > > > for user attributes to prevent unexpected fails. I think it is better
> > > than
> > > > changing writer to one with heavy classloading.
> > > >
> > > > Is it ok to restrict thin attributes to primitives and 'String'?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > > >
> > >
> >
>

Re: Add user attributes to thin clients

[Dmitrii Ryabov]

This map is something like user object from `SecurityCredentials`.
Sometimes login and password are not enough for security checks. For
example, we can send security certificate and validate it inside
authenticator.

ср, 22 янв. 2020 г., 17:16 Igor Sapego :

> Hi Dmitrii,
>
> Can you please explain your use case?
> I'm not sure I'm getting what is the motivation of this change.
>
> Best Regards,
> Igor
>
>
> On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn 
> wrote:
>
> > Hi Dmitrii,
> >
> > Honestly, I could not grasp the problem, can you explain it in more
> detail?
> > What do we solve by adding a map with arbitrary stuff to the client
> > protocol handshake?
> >
> > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov 
> > wrote:
> >
> > > Hello, Igniters!
> > >
> > > I want to add the possibility of sending user defined attributes from
> > thin
> > > clients. And check them inside custom authenticator during handshake
> [1].
> > >
> > > There is an issue in hardcoded binary writer for JDBC and
> `IgniteClient`.
> > > This writer searches for a classes in the JDK and
> > > META-INF/classnames.properties, and tries to sync notdeclared classes
> > with
> > > cluster. But fails because current classloading uses discovery.
> > >
> > > I'd like to keep this writer and allow only primitive types and
> `String`
> > > for user attributes to prevent unexpected fails. I think it is better
> > than
> > > changing writer to one with heavy classloading.
> > >
> > > Is it ok to restrict thin attributes to primitives and 'String'?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> > >
> >
>

Re: Internal classes are exposed in public API

[Николай Ижиков]

Hello, Igniters.

* IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the master 
and cherry-picked to the 2.8.
So the main issue with the Metric API solved.

* I raised the PR [1] to fix second issue with the new Metric API: absence of 
the public Java API to get metrics.
This PR introduces the following changes:

1. IgniteMetric interface created: it provides Java API to access Ignite 
metrics created with the new Metric API.

```
public interface IgniteMetric extends Iterable {
@Nullable ReadOnlyMetricRegistry registry(String name);
}
```

2. All deprecation javadocs regarding metrics now reference to the public 
IgniteMetric instead of internal GridMetricManager:

  > @deprecated Use {@link IgniteMetric} instead.

3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
possible.

Please, review.

[1]  https://github.com/apache/ignite/pull/7283

> 21 янв. 2020 г., в 17:51, Николай Ижиков  написал(а):
> 
> Hello, Igniters.
> 
> Alexey approved my PR [1] regarding fixing public API for metric exporters.
> I’m waiting for a bot visa and merge this PR.
> 
> As we discussed, the metrics API will be marked with IgniteExperimental 
> annotation.
> 
> If anyone has any objection to this plan, please provide your feedback.
> 
> [1] https://github.com/apache/ignite/pull/7269
> 
>> 21 янв. 2020 г., в 13:45, Николай Ижиков  написал(а):
>> 
>> Thanks, for the review Alexey.
>> 
>> I will fix your comment and  try to implement Monitoring facade, shortly.
>> 
>>> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
>>> написал(а):
>>> 
>>> Nikolay,
>>> 
>>> I left a single comment in the PR about the histogram metric. I think the
>>> API looks much cleaner now.
>>> 
>>> I will take care of the @IgniteExperimental annotation.
>>> 
>>> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
>>> 
 Alexey.
 
 PR [1] is waiting for your review.
 Please, take a look.
 
 I think we should do the following before 2.8 release
 
 * Introduce new @IgniteExperimental annotation as discussed.
 * Mark Monitoring API with it.
 * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
 * merge «[IEP-35] public Java metric API» to 2.8
 
 [1] https://github.com/apache/ignite/pull/7269
 
> 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
 написал(а):
> 
> Nikolay,
> 
> Should we wait for both of the tickets given that we agreed that we are
> putting an experimental marker on the new APIs? I'm ok to fix only the
> first one and add the experimental marker so that we do not delay 2.8
> release.
> 
> пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
> 
>> Andrey.
>> 
>> Let’s move from the long letters to the code.
>> If you want to change API - please, propose this changes.
>> I think everybody wins if we make our API better.
>> 
>> Please, describe proposed changes.
>> It would be great if you have some examples or PR for it.
>> 
>> As for this release, I have plans to contribute tickets
>> «[IEP-35] Expose MetricRegistry to the public API» [1] and
>> «[IEP-35] public Java metric API» [2] for it.
>> 
>> Any objections on it?
>> 
>> [1] https://github.com/apache/ignite/pull/7269
>> [2] https://issues.apache.org/jira/browse/IGNITE-12553
>> 
>> 
>>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
>>> 
>>> It solves problem.
>>> 
>>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
>>>  wrote:
 
 After giving it some thought, I agree with Denis - there is nothing
>> wrong
 with exposing the new APIs, we just need to make it clear that we are
>> still
 going to change it.
 
 Should we Introduce something like @IgniteExperimental annotation (I
 believe it has been discussed a dozen of times on the dev-list)?
 
 пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
 
> +1 to mark feature or whole release as EA.
> 
> пт, 17 янв. 2020 г., 23:00 Denis Magda :
> 
>> Folks, if you don't mind I'll share some thoughts/suggestions as an
>> observer who was not involved in the feature development.
>> 
>> It's absolutely 'ok' to deprecate an API that is replaced with a
 much
>> better version. However, we should do this only when the new APIs
 are
>> production-ready. If there are still many limitations or open items
>> then
>> don't deprecate anything that exists and release the new APIs
>> labeling as
>> early access. What if release the improvements labeling as EA
 instead
>> of
>> hiding them completely?
>> 
>> I would also encourage us to put aside emotions, don't blame or
 point
>> fingers. This IEP is a great initiative and you both have already
>> done a
>> tremendous job by 

Re: Add user attributes to thin clients

[Igor Sapego]

Hi Dmitrii,

Can you please explain your use case?
I'm not sure I'm getting what is the motivation of this change.

Best Regards,
Igor


On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn  wrote:

> Hi Dmitrii,
>
> Honestly, I could not grasp the problem, can you explain it in more detail?
> What do we solve by adding a map with arbitrary stuff to the client
> protocol handshake?
>
> On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov 
> wrote:
>
> > Hello, Igniters!
> >
> > I want to add the possibility of sending user defined attributes from
> thin
> > clients. And check them inside custom authenticator during handshake [1].
> >
> > There is an issue in hardcoded binary writer for JDBC and `IgniteClient`.
> > This writer searches for a classes in the JDK and
> > META-INF/classnames.properties, and tries to sync notdeclared classes
> with
> > cluster. But fails because current classloading uses discovery.
> >
> > I'd like to keep this writer and allow only primitive types and `String`
> > for user attributes to prevent unexpected fails. I think it is better
> than
> > changing writer to one with heavy classloading.
> >
> > Is it ok to restrict thin attributes to primitives and 'String'?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12049
> >
>

Re: Add user attributes to thin clients

[Pavel Tupitsyn]

Hi Dmitrii,

Honestly, I could not grasp the problem, can you explain it in more detail?
What do we solve by adding a map with arbitrary stuff to the client
protocol handshake?

On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov 
wrote:

> Hello, Igniters!
>
> I want to add the possibility of sending user defined attributes from thin
> clients. And check them inside custom authenticator during handshake [1].
>
> There is an issue in hardcoded binary writer for JDBC and `IgniteClient`.
> This writer searches for a classes in the JDK and
> META-INF/classnames.properties, and tries to sync notdeclared classes with
> cluster. But fails because current classloading uses discovery.
>
> I'd like to keep this writer and allow only primitive types and `String`
> for user attributes to prevent unexpected fails. I think it is better than
> changing writer to one with heavy classloading.
>
> Is it ok to restrict thin attributes to primitives and 'String'?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12049
>

Add user attributes to thin clients

[Dmitrii Ryabov]

Hello, Igniters!

I want to add the possibility of sending user defined attributes from thin
clients. And check them inside custom authenticator during handshake [1].

There is an issue in hardcoded binary writer for JDBC and `IgniteClient`.
This writer searches for a classes in the JDK and
META-INF/classnames.properties, and tries to sync notdeclared classes with
cluster. But fails because current classloading uses discovery.

I'd like to keep this writer and allow only primitive types and `String`
for user attributes to prevent unexpected fails. I think it is better than
changing writer to one with heavy classloading.

Is it ok to restrict thin attributes to primitives and 'String'?

[1] https://issues.apache.org/jira/browse/IGNITE-12049

[jira] [Created] (IGNITE-12567) H2Tree goes into illegal state when non-indexed columns are dropped

[Konstantin Orlov (Jira)]

Konstantin Orlov created IGNITE-12567:
-

 Summary: H2Tree goes into illegal state when non-indexed columns 
are dropped
 Key: IGNITE-12567
 URL: https://issues.apache.org/jira/browse/IGNITE-12567
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Konstantin Orlov
Assignee: Konstantin Orlov


*How to reproduce:*
* {{CREATE TABLE tbl (f0 INT, f1 INT, f2 INT, f3 INT, f4 INT, PRIMARY KEY (f3, 
f4))}}
* populate table with some data
* {{ALTER TABLE tbl DROP COLUMN f1, f2}}
* try to execute query which will use pk index: {{SELECT * FROM tbl WHERE f3 = 
1 AND f4 = 1}} 

*Expected*:
* Query returns result set

*Actual*:
* Query fails with {{General error: "java.lang.ArrayIndexOutOfBoundsException: 
5";}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Ilya Kasnacheev]

Hello!

I have committed the bumping of essential dependencies' versions:
https://issues.apache.org/jira/browse/IGNITE-12540

Would you mind including this change into the scope of 2.8? No point of
shipping known problematic JARs in our deliverable.

Regards,
-- 
Ilya Kasnacheev


ср, 22 янв. 2020 г. в 14:00, Maxim Muzafarov :

> Alexey,
>
> Sure, I've just thought about it too a few days ago.
>
> On Wed, 22 Jan 2020 at 12:09, Anton Vinogradov  wrote:
> >
> > Good Idea, this will also check that the release process is alive.
> >
> > On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> > > Folks, Maxim,
> > >
> > > Do you mind if I build the current state of ignite-2.8 branch and
> upload a
> > > maven staging as rc0 (step 4.3.2 of the release process)? I want run
> some
> > > tests for the fixes that are already included to the branch.
> > >
> > > вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
> > >
> > > > Folks,
> > > >
> > > >
> > > > I think both of these issues [1] [2] are critical to 2.8 release and
> > > > we must include them.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > Excessive AtomicLong instantiations lead to GC pressure.
> > > >
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > > > Pages list caching can cause IgniteOOME when the checkpoint is
> > > > triggered by "too many dirty pages" reason.
> > > >
> > > >
> > > > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov  >
> > > > wrote:
> > > > >
> > > > > Guys,
> > > > >
> > > > > There is an issue [1] caused by page list caching [2], which also
> > > affects
> > > > > 2.8 release. IgniteOutOfMemoryException can be thrown in some cases
> > > (data
> > > > > region is small, a checkpoint is triggered by "too many dirty
> pages"
> > > > reason
> > > > > and pages list cache is rather big).
> > > > > The fix is ready and merged to master, I suggest to include this
> fix to
> > > > 2.8
> > > > > release. What do you think?
> > > > >
> > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > > > >
> > > > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com>:
> > > > >
> > > > > > Maxim,
> > > > > >
> > > > > > I took a quick look at IGNITE-12456 and I am not sure it's about
> data
> > > > > > corruption. In the attached logs blocked system threads are
> reported,
> > > > > > however, there is no enough information to investigate the issue
> (the
> > > > full
> > > > > > thread dump was not attached). I asked the ticket creator to
> attach
> > > > missing
> > > > > > pieces.
> > > > > >
> > > > > > Should we consider moving this ticket to a next release?
> > > > > >
> > > > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> > > >  > > > > > >:
> > > > > >
> > > > > > >
> > > > > > > Maxim, performance fix issue [1] already in master, if no
> > > > objections, can
> > > > > > > u merge it into 2.8 ? Thanks !
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > > >
> > > > > > > >Igniters,
> > > > > > > >
> > > > > > > >
> > > > > > > >Here is the actual list of BLOCKER release issues:
> > > > > > > >
> > > > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load
> test
> > > > > > > >*[Unassigned]* OPEN
> > > > > > > >IGNITE-12489 Error during purges by expiration: Unknown page
> type*
> > > > > > > >[Unassigned]* OPEN
> > > > > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml
> config
> > > > > > > >*[Unassigned]* OPEN
> > > > > > > >
> > > > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery)
> > > Nodes
> > > > > > > getting
> > > > > > > >down [Emmanouil Gkatziouras] OPEN
> > > > > > > >IGNITE-9184 Cluster hangs during concurrent node client and
> server
> > > > nodes
> > > > > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement
> [Nikolay
> > > > > > Izhikov]
> > > > > > > >Blocker IN PROGRESS
> > > > > > > >
> > > > > > > >IGNITE-12227 Default auto-adjust baseline enabled flag
> calculated
> > > > > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > > > > >IGNITE-12470 Pme-free switch feature should be deactivatable
> > > [Sergei
> > > > > > > >Ryzhov] PATCH AVAILABLE
> > > > > > > >IGNITE-12552 [IEP-35] Expose MetricRegistry to the public API
> > > > > > Improvement
> > > > > > > >[Nikolay Izhikov] PATCH AVAILABLE
> > > > > > > >
> > > > > > > >
> > > > > > > >[1]  https://issues.apache.org/jira/browse/IGNITE-12456
> > > > > > > >[2]  https://issues.apache.org/jira/browse/IGNITE-12489
> > > > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-8641
> > > > > > > >[8]  https://issues.apache.org/jira/browse/IGNITE-12398
> > > > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-9184
> > > > > > > >[6]  https://issues.apache.org/jira/browse/IGNITE-12553
> > > > > > > >[7]  

[jira] [Created] (IGNITE-12566) Calcite integration. Expressions evaluation.

[Igor Seliverstov (Jira)]

Igor Seliverstov created IGNITE-12566:
-

 Summary: Calcite integration. Expressions evaluation.
 Key: IGNITE-12566
 URL: https://issues.apache.org/jira/browse/IGNITE-12566
 Project: Ignite
  Issue Type: Task
Reporter: Igor Seliverstov


Currently we use a part of Calcite "Bindables" to evaluate expressions at the 
execution time. Using it we 
 * lose a control on how expressions are evaluated
 * cannot implement several important optimizations, like "keepBinary"
 * can use only Object[] as a row, which causes a lot of array copy operations 
and unnecessary allocations at the execution time
 * suffer from delays, which go from expressions compilation process when it's 
more efficient to start from interpretation and compile an expression on 
repeatable execution only

We need to implement expressions interpreter and our own expression compiler 
taking Ignite specifics in consideration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12565) Extend test coverage [IGNITE-9279] Support custom default SQL schema name

[Ivan Pavlukhin (Jira)]

Ivan Pavlukhin created IGNITE-12565:
---

 Summary: Extend test coverage [IGNITE-9279] Support custom default 
SQL schema name
 Key: IGNITE-12565
 URL: https://issues.apache.org/jira/browse/IGNITE-12565
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Ivan Pavlukhin
Assignee: Ivan Pavlukhin
 Fix For: 2.9


Add more tests to cover schemas support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12564) Calcite integration. Exceptions handling.

[Igor Seliverstov (Jira)]

Igor Seliverstov created IGNITE-12564:
-

 Summary: Calcite integration. Exceptions handling.
 Key: IGNITE-12564
 URL: https://issues.apache.org/jira/browse/IGNITE-12564
 Project: Ignite
  Issue Type: Task
Reporter: Igor Seliverstov


There are several exceptions types that may be thrown at
 * parsing and validation time
 * planning time
 * mapping time
 * execution time

All of them should be handled properly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Request for contribution permission.

[Ivan Pavlukhin]

Artem,

Welcome to Apache Ignite Community!

I added a contributor role to your Jira account. Now you can assign
tickets to yourself. Please get familiar with the contribution process
[1].

[1] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

ср, 22 янв. 2020 г. в 14:20, Artem Demchenko :
>
> Jira:
> Username:
> Mincer2412
> Full name:
> Artem D
>
>
>


-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-12563) Calcite integration. Reset and remap queries on topology change.

[Igor Seliverstov (Jira)]

Igor Seliverstov created IGNITE-12563:
-

 Summary: Calcite integration. Reset and remap queries on topology 
change.
 Key: IGNITE-12563
 URL: https://issues.apache.org/jira/browse/IGNITE-12563
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Igor Seliverstov


A query should:
 * be silently remapped in case an actual topology version at query plan 
materialization time differs from a topology version at the time locations was 
calculated
 * be cancelled with appropriate exception in case one of responsible nodes 
left a cluster after execution is started.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Request for contribution permission.

[Artem Demchenko]

Jira:
Username:
Mincer2412
Full name:
Artem D

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Maxim Muzafarov]

Alexey,

Sure, I've just thought about it too a few days ago.

On Wed, 22 Jan 2020 at 12:09, Anton Vinogradov  wrote:
>
> Good Idea, this will also check that the release process is alive.
>
> On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Folks, Maxim,
> >
> > Do you mind if I build the current state of ignite-2.8 branch and upload a
> > maven staging as rc0 (step 4.3.2 of the release process)? I want run some
> > tests for the fixes that are already included to the branch.
> >
> > вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
> >
> > > Folks,
> > >
> > >
> > > I think both of these issues [1] [2] are critical to 2.8 release and
> > > we must include them.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > Excessive AtomicLong instantiations lead to GC pressure.
> > >
> > > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > > Pages list caching can cause IgniteOOME when the checkpoint is
> > > triggered by "too many dirty pages" reason.
> > >
> > >
> > > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov 
> > > wrote:
> > > >
> > > > Guys,
> > > >
> > > > There is an issue [1] caused by page list caching [2], which also
> > affects
> > > > 2.8 release. IgniteOutOfMemoryException can be thrown in some cases
> > (data
> > > > region is small, a checkpoint is triggered by "too many dirty pages"
> > > reason
> > > > and pages list cache is rather big).
> > > > The fix is ready and merged to master, I suggest to include this fix to
> > > 2.8
> > > > release. What do you think?
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > > >
> > > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com>:
> > > >
> > > > > Maxim,
> > > > >
> > > > > I took a quick look at IGNITE-12456 and I am not sure it's about data
> > > > > corruption. In the attached logs blocked system threads are reported,
> > > > > however, there is no enough information to investigate the issue (the
> > > full
> > > > > thread dump was not attached). I asked the ticket creator to attach
> > > missing
> > > > > pieces.
> > > > >
> > > > > Should we consider moving this ticket to a next release?
> > > > >
> > > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> > >  > > > > >:
> > > > >
> > > > > >
> > > > > > Maxim, performance fix issue [1] already in master, if no
> > > objections, can
> > > > > > u merge it into 2.8 ? Thanks !
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > >
> > > > > > >Igniters,
> > > > > > >
> > > > > > >
> > > > > > >Here is the actual list of BLOCKER release issues:
> > > > > > >
> > > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load test
> > > > > > >*[Unassigned]* OPEN
> > > > > > >IGNITE-12489 Error during purges by expiration: Unknown page type*
> > > > > > >[Unassigned]* OPEN
> > > > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml config
> > > > > > >*[Unassigned]* OPEN
> > > > > > >
> > > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery)
> > Nodes
> > > > > > getting
> > > > > > >down [Emmanouil Gkatziouras] OPEN
> > > > > > >IGNITE-9184 Cluster hangs during concurrent node client and server
> > > nodes
> > > > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement [Nikolay
> > > > > Izhikov]
> > > > > > >Blocker IN PROGRESS
> > > > > > >
> > > > > > >IGNITE-12227 Default auto-adjust baseline enabled flag calculated
> > > > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > > > >IGNITE-12470 Pme-free switch feature should be deactivatable
> > [Sergei
> > > > > > >Ryzhov] PATCH AVAILABLE
> > > > > > >IGNITE-12552 [IEP-35] Expose MetricRegistry to the public API
> > > > > Improvement
> > > > > > >[Nikolay Izhikov] PATCH AVAILABLE
> > > > > > >
> > > > > > >
> > > > > > >[1]  https://issues.apache.org/jira/browse/IGNITE-12456
> > > > > > >[2]  https://issues.apache.org/jira/browse/IGNITE-12489
> > > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-8641
> > > > > > >[8]  https://issues.apache.org/jira/browse/IGNITE-12398
> > > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-9184
> > > > > > >[6]  https://issues.apache.org/jira/browse/IGNITE-12553
> > > > > > >[7]  https://issues.apache.org/jira/browse/IGNITE-12227
> > > > > > >[9]  https://issues.apache.org/jira/browse/IGNITE-12470
> > > > > > >[5]  https://issues.apache.org/jira/browse/IGNITE-12552
> > > > > > >
> > > > > > >
> > > > > > >On Sat, 18 Jan 2020 at 19:11, Sergey Antonov <
> > > > > antonovserge...@gmail.com
> > > > > > >
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Maxim,
> > > > > > >>
> > > > > > >> Conflicts in pr [1] are resolved. TC Run all is started.
> > > > > > >>
> > > > > > >> [1]  https://github.com/apache/ignite/pull/7238
> > > > > > >>
> > > > > > >> пт, 17 янв. 2020 г. в 16:04, Sergey 

Re: New SQL engine, review/feedback request

[Alexey Goncharuk]

Igor,

Thanks for driving this activity! I hope someone more prominent in SQL
subsystem will take a look at the execution logic. From my side, I left a
comment regarding the code structure in the ticket.

ср, 22 янв. 2020 г. в 12:56, Seliverstov Igor :

> Hi guys,
>
> Some of you know, I'm working on a brand new Calcite based SQL engine.
>
> So, now I have some intermediate results and need a feedback from
> community.
>
> A feature branch with a concept of a module is ignite-12248 <
> https://github.com/apache/ignite/tree/ignite-12248>. The module is going
> to be continuously enhanced with new functionality.
>
> I’ll appreciate If you share your thoughts.
>
> Additionally a next patch required a review:
>
> https://issues.apache.org/jira/browse/IGNITE-12448 <
> https://issues.apache.org/jira/browse/IGNITE-12448>
> https://github.com/apache/ignite/pull/7272 <
> https://github.com/apache/ignite/pull/7272>
>
> There are quite a lot of changes because I'm still trying to find a best
> way of how to organize the engine parts cross dependencies and code in
> general.
>
> Regards,
> Igor

New SQL engine, review/feedback request

[Seliverstov Igor]

Hi guys,

Some of you know, I'm working on a brand new Calcite based SQL engine.

So, now I have some intermediate results and need a feedback from community.

A feature branch with a concept of a module is ignite-12248 
. The module is going to be 
continuously enhanced with new functionality.

I’ll appreciate If you share your thoughts.

Additionally a next patch required a review:

https://issues.apache.org/jira/browse/IGNITE-12448 

https://github.com/apache/ignite/pull/7272 


There are quite a lot of changes because I'm still trying to find a best way of 
how to organize the engine parts cross dependencies and code in general.

Regards,
Igor

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Anton Vinogradov]

Good Idea, this will also check that the release process is alive.

On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Folks, Maxim,
>
> Do you mind if I build the current state of ignite-2.8 branch and upload a
> maven staging as rc0 (step 4.3.2 of the release process)? I want run some
> tests for the fixes that are already included to the branch.
>
> вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
>
> > Folks,
> >
> >
> > I think both of these issues [1] [2] are critical to 2.8 release and
> > we must include them.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > Excessive AtomicLong instantiations lead to GC pressure.
> >
> > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > Pages list caching can cause IgniteOOME when the checkpoint is
> > triggered by "too many dirty pages" reason.
> >
> >
> > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov 
> > wrote:
> > >
> > > Guys,
> > >
> > > There is an issue [1] caused by page list caching [2], which also
> affects
> > > 2.8 release. IgniteOutOfMemoryException can be thrown in some cases
> (data
> > > region is small, a checkpoint is triggered by "too many dirty pages"
> > reason
> > > and pages list cache is rather big).
> > > The fix is ready and merged to master, I suggest to include this fix to
> > 2.8
> > > release. What do you think?
> > >
> > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > >
> > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > alexey.goncha...@gmail.com>:
> > >
> > > > Maxim,
> > > >
> > > > I took a quick look at IGNITE-12456 and I am not sure it's about data
> > > > corruption. In the attached logs blocked system threads are reported,
> > > > however, there is no enough information to investigate the issue (the
> > full
> > > > thread dump was not attached). I asked the ticket creator to attach
> > missing
> > > > pieces.
> > > >
> > > > Should we consider moving this ticket to a next release?
> > > >
> > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> >  > > > >:
> > > >
> > > > >
> > > > > Maxim, performance fix issue [1] already in master, if no
> > objections, can
> > > > > u merge it into 2.8 ? Thanks !
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > >
> > > > > >Igniters,
> > > > > >
> > > > > >
> > > > > >Here is the actual list of BLOCKER release issues:
> > > > > >
> > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load test
> > > > > >*[Unassigned]* OPEN
> > > > > >IGNITE-12489 Error during purges by expiration: Unknown page type*
> > > > > >[Unassigned]* OPEN
> > > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml config
> > > > > >*[Unassigned]* OPEN
> > > > > >
> > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery)
> Nodes
> > > > > getting
> > > > > >down [Emmanouil Gkatziouras] OPEN
> > > > > >IGNITE-9184 Cluster hangs during concurrent node client and server
> > nodes
> > > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement [Nikolay
> > > > Izhikov]
> > > > > >Blocker IN PROGRESS
> > > > > >
> > > > > >IGNITE-12227 Default auto-adjust baseline enabled flag calculated
> > > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > > >IGNITE-12470 Pme-free switch feature should be deactivatable
> [Sergei
> > > > > >Ryzhov] PATCH AVAILABLE
> > > > > >IGNITE-12552 [IEP-35] Expose MetricRegistry to the public API
> > > > Improvement
> > > > > >[Nikolay Izhikov] PATCH AVAILABLE
> > > > > >
> > > > > >
> > > > > >[1]  https://issues.apache.org/jira/browse/IGNITE-12456
> > > > > >[2]  https://issues.apache.org/jira/browse/IGNITE-12489
> > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-8641
> > > > > >[8]  https://issues.apache.org/jira/browse/IGNITE-12398
> > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-9184
> > > > > >[6]  https://issues.apache.org/jira/browse/IGNITE-12553
> > > > > >[7]  https://issues.apache.org/jira/browse/IGNITE-12227
> > > > > >[9]  https://issues.apache.org/jira/browse/IGNITE-12470
> > > > > >[5]  https://issues.apache.org/jira/browse/IGNITE-12552
> > > > > >
> > > > > >
> > > > > >On Sat, 18 Jan 2020 at 19:11, Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Maxim,
> > > > > >>
> > > > > >> Conflicts in pr [1] are resolved. TC Run all is started.
> > > > > >>
> > > > > >> [1]  https://github.com/apache/ignite/pull/7238
> > > > > >>
> > > > > >> пт, 17 янв. 2020 г. в 16:04, Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >:
> > > > > >>
> > > > > >>> Maxim,
> > > > > >>>
> > > > > >>> I will do that on monday (20/01).
> > > > > >>>
> > > > > >>> пт, 17 янв. 2020 г. в 13:08, Maxim Muzafarov <
> mmu...@apache.org
> > >:
> > > > > >>>
> > > > >  Sergey,
> > > > > 
> > > > > 
> > > > >  Can you, please, resolve the PR 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Alexey Goncharuk]

Folks, Maxim,

Do you mind if I build the current state of ignite-2.8 branch and upload a
maven staging as rc0 (step 4.3.2 of the release process)? I want run some
tests for the fixes that are already included to the branch.

вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :

> Folks,
>
>
> I think both of these issues [1] [2] are critical to 2.8 release and
> we must include them.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12547
> Excessive AtomicLong instantiations lead to GC pressure.
>
> [2] https://issues.apache.org/jira/browse/IGNITE-12530
> Pages list caching can cause IgniteOOME when the checkpoint is
> triggered by "too many dirty pages" reason.
>
>
> On Mon, 20 Jan 2020 at 19:00, Alex Plehanov 
> wrote:
> >
> > Guys,
> >
> > There is an issue [1] caused by page list caching [2], which also affects
> > 2.8 release. IgniteOutOfMemoryException can be thrown in some cases (data
> > region is small, a checkpoint is triggered by "too many dirty pages"
> reason
> > and pages list cache is rather big).
> > The fix is ready and merged to master, I suggest to include this fix to
> 2.8
> > release. What do you think?
> >
> > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> >
> > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> alexey.goncha...@gmail.com>:
> >
> > > Maxim,
> > >
> > > I took a quick look at IGNITE-12456 and I am not sure it's about data
> > > corruption. In the attached logs blocked system threads are reported,
> > > however, there is no enough information to investigate the issue (the
> full
> > > thread dump was not attached). I asked the ticket creator to attach
> missing
> > > pieces.
> > >
> > > Should we consider moving this ticket to a next release?
> > >
> > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
>  > > >:
> > >
> > > >
> > > > Maxim, performance fix issue [1] already in master, if no
> objections, can
> > > > u merge it into 2.8 ? Thanks !
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > >
> > > > >Igniters,
> > > > >
> > > > >
> > > > >Here is the actual list of BLOCKER release issues:
> > > > >
> > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load test
> > > > >*[Unassigned]* OPEN
> > > > >IGNITE-12489 Error during purges by expiration: Unknown page type*
> > > > >[Unassigned]* OPEN
> > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml config
> > > > >*[Unassigned]* OPEN
> > > > >
> > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes
> > > > getting
> > > > >down [Emmanouil Gkatziouras] OPEN
> > > > >IGNITE-9184 Cluster hangs during concurrent node client and server
> nodes
> > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement [Nikolay
> > > Izhikov]
> > > > >Blocker IN PROGRESS
> > > > >
> > > > >IGNITE-12227 Default auto-adjust baseline enabled flag calculated
> > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > >IGNITE-12470 Pme-free switch feature should be deactivatable [Sergei
> > > > >Ryzhov] PATCH AVAILABLE
> > > > >IGNITE-12552 [IEP-35] Expose MetricRegistry to the public API
> > > Improvement
> > > > >[Nikolay Izhikov] PATCH AVAILABLE
> > > > >
> > > > >
> > > > >[1]  https://issues.apache.org/jira/browse/IGNITE-12456
> > > > >[2]  https://issues.apache.org/jira/browse/IGNITE-12489
> > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-8641
> > > > >[8]  https://issues.apache.org/jira/browse/IGNITE-12398
> > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-9184
> > > > >[6]  https://issues.apache.org/jira/browse/IGNITE-12553
> > > > >[7]  https://issues.apache.org/jira/browse/IGNITE-12227
> > > > >[9]  https://issues.apache.org/jira/browse/IGNITE-12470
> > > > >[5]  https://issues.apache.org/jira/browse/IGNITE-12552
> > > > >
> > > > >
> > > > >On Sat, 18 Jan 2020 at 19:11, Sergey Antonov <
> > > antonovserge...@gmail.com
> > > > >
> > > > >wrote:
> > > > >
> > > > >> Maxim,
> > > > >>
> > > > >> Conflicts in pr [1] are resolved. TC Run all is started.
> > > > >>
> > > > >> [1]  https://github.com/apache/ignite/pull/7238
> > > > >>
> > > > >> пт, 17 янв. 2020 г. в 16:04, Sergey Antonov <
> > > antonovserge...@gmail.com
> > > > >:
> > > > >>
> > > > >>> Maxim,
> > > > >>>
> > > > >>> I will do that on monday (20/01).
> > > > >>>
> > > > >>> пт, 17 янв. 2020 г. в 13:08, Maxim Muzafarov < mmu...@apache.org
> >:
> > > > >>>
> > > >  Sergey,
> > > > 
> > > > 
> > > >  Can you, please, resolve the PR conflicts [1] [2]?
> > > > 
> > > >  [1]  https://github.com/apache/ignite/pull/7238
> > > >  [2]  https://issues.apache.org/jira/browse/IGNITE-11256
> > > > 
> > > >  On Thu, 16 Jan 2020 at 16:59, Ilya Kasnacheev <
> > > > ilya.kasnach...@gmail.com >
> > > >  wrote:
> > > >  >
> > > >  > Hello!
> > > >  >
> > > >  > I have bumped beanutils and re-ran Cassandra Store tests. Can
> you
> > 

[jira] [Created] (IGNITE-12562) FreeListCachingTest.testPageListCacheLimit failing in Disk Page Compressions suite

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-12562:
--

 Summary: FreeListCachingTest.testPageListCacheLimit failing in 
Disk Page Compressions suite
 Key: IGNITE-12562
 URL: https://issues.apache.org/jira/browse/IGNITE-12562
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov


Test failing with IgniteOutOfMemory exception because Disk Page Compressions 
changes default page size to 8192, we need at least 2*1024 pages to store 
freelist metadata and there is no such amount of not dirty pages after 
checkpoint is triggered by "too many dirty page" reason.

Explicitly defined data region max size should take into account page size to 
avoid such errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)