New channel about integration with Spring Framework
Hi igniters! I have created new channel "#ignite-spring-framework" for discussion about integration with Spring Framework. -- - With best regards, Sergey Chernolyas
Re: Task to implement: IGNITE-12581
Ivan, thanks. It's ok. I'm waiting for the test results, researching the
code, doing next simple ticket.
ср, 29 янв. 2020 г., 10:38 Ivan Pavlukhin :
> Vladimir,
>
> Do no hesitate to write back if you need any assistance, e.g. troubles
> with code or a code review.
>
> пн, 27 янв. 2020 г. в 13:28, Vladimir Steshin :
> >
> > Ok. Forgot to explain the ticket. I'm going to bring some refactoring to
> > the test, replace inheritance with parametrization. Many tests use
> > extending to handle params like
> >
> > public class JdbcThinBulkLoadAtomicReplicatedSelfTest extends
> > JdbcThinBulkLoadAbstractSelfTest {
> > /** {@inheritDoc} */
> > @Override protected CacheMode cacheMode() {
> > return CacheMode.REPLICATED;
> > }
> >
> > /** {@inheritDoc} */
> > @Override protected CacheAtomicityMode atomicityMode() {
> > return CacheAtomicityMode.ATOMIC;
> > }
> >
> > /** {@inheritDoc} */
> > @Override protected boolean nearCache() {
> > return false;
> > }
> > }
> >
> >
> > We might involve JUnit's params reducing code amount.
> >
> >
> >
> >
> > We might involve
> >
> >
> > пн, 27 янв. 2020 г. в 11:38, Maksim Stepachev <
> maksim.stepac...@gmail.com>:
> >
> > > Hi,
> > >
> > > I suppose you can.
> > > Nikolay possibly may help with it.
> > > If you have the telegram, join to Russian channel:
> https://t.me/RU_Ignite
> > >
> > > вс, 26 янв. 2020 г. в 19:21, Vladimir Steshin :
> > >
> > > > Hi everyone.
> > > >
> > > > How do you think, can I start this task:
> > > > https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12581 ?
> > > >
> > > > It looks useful and seems good for a newbie to begin researching the
> > > > code and test techniques.
> > > >
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Task to implement: IGNITE-12581
Vladimir,
Do no hesitate to write back if you need any assistance, e.g. troubles
with code or a code review.
пн, 27 янв. 2020 г. в 13:28, Vladimir Steshin :
>
> Ok. Forgot to explain the ticket. I'm going to bring some refactoring to
> the test, replace inheritance with parametrization. Many tests use
> extending to handle params like
>
> public class JdbcThinBulkLoadAtomicReplicatedSelfTest extends
> JdbcThinBulkLoadAbstractSelfTest {
> /** {@inheritDoc} */
> @Override protected CacheMode cacheMode() {
> return CacheMode.REPLICATED;
> }
>
> /** {@inheritDoc} */
> @Override protected CacheAtomicityMode atomicityMode() {
> return CacheAtomicityMode.ATOMIC;
> }
>
> /** {@inheritDoc} */
> @Override protected boolean nearCache() {
> return false;
> }
> }
>
>
> We might involve JUnit's params reducing code amount.
>
>
>
>
> We might involve
>
>
> пн, 27 янв. 2020 г. в 11:38, Maksim Stepachev :
>
> > Hi,
> >
> > I suppose you can.
> > Nikolay possibly may help with it.
> > If you have the telegram, join to Russian channel: https://t.me/RU_Ignite
> >
> > вс, 26 янв. 2020 г. в 19:21, Vladimir Steshin :
> >
> > > Hi everyone.
> > >
> > > How do you think, can I start this task:
> > > https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12581 ?
> > >
> > > It looks useful and seems good for a newbie to begin researching the
> > > code and test techniques.
> > >
> > >
> >
--
Best regards,
Ivan Pavlukhin
Re: Client nodes are unable to discover server nodes using kubernetes IpFinder
Manu,
Looks like it is currently impossible to use custom discovery port
with TcpDiscoveryKubernetesIpFinder. This limitation is covered in
code of a method
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#parseAddresses
Feel free to create a Jira issue for an improvement.
сб, 25 янв. 2020 г. в 10:31, Manu :
>
> Hi!
>
> We are trying to deploy an ignite cluster(s) into kubernetes and found some
> issues.
>
> Sample deployed topology into kubernetes (we need each ignite server cluster
> has different discovery port)
>
> Ignite Server(s) cluster to store main data (discoPort 49500)
> \
> Client App with 2 ignite clients (discoPort 49500, discoPort 48500) to
> talk with both clusters
> /
> Ignite Server(s) cluster to store cached data (discoPort 48500)
>
> Apparently, client nodes are unable to discover server nodes using
> kubernetes IpFinder whether non default discovery port, for example 49500,
> is used on server side ignite configuration.
>
> TcpDiscoverySpi local port configuration is ignored on client mode, so port
> 47500 is always used to connect to ignite servers (we don't want to use
> forceServerMode=true as it is a deprecated feature and may have negative
> performance impact)
>
> In general seems that any IpFinder implementation that has this behaviour
> (not provides server port when registered addresses were returned) will have
> same issue.
>
> Fix suggestion, modify org.apache.ignite.spi.discovery.tcp.ClientImpl
> spiStart, apparently this change has no collateral effects:
>
> Original
>@Override public void spiStart(@Nullable String igniteInstanceName)
> throws IgniteSpiException {
> spi.initLocalNode(
> 0,
> true);
>
> Fix
>@Override public void spiStart(@Nullable String igniteInstanceName)
> throws IgniteSpiException {
> // FIX ignored TcpDiscoverySpi local port configuration on client
> mode, takes configured one
> spi.initLocalNode(
> spi.locPort,
> true);
>
>
> How to reproduce issue:
>
> 1. Deploy a server node with tcp discovery kubernetes IpFinder and non
> default discovery port 49500 into kubernetes
>
> 2. Deploy client nodes with tcp discovery kubernetes IpFinder and non
> default discovery port 49500 into kubernetes
>
> 3. You will see on client nodes logs that will try to connect to servers on
> port 47500, instead of configured 49500.
>
> Ignite configuration for both server and client node(s)
> [...]
>
>
>
>
>
>
>class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
>
>
>
> value="https://kubernetes.default.svc.cluster.local:443"/>
> value="/var/run/secrets/kubernetes.io/serviceaccount/token"/>
>
>
>
>
> [...]
>
> Regards!
>
> Manu.
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
--
Best regards,
Ivan Pavlukhin
Re: Hello
Hi Sergey, Welcome to the Apache Ignite Community. Do you need some assistance? Or might be you have something to tell the Community. Do not hesitate to write back. I see that your Jira account was already added to the contributors list. You can find some contribution guidelines at https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute вт, 28 янв. 2020 г. в 22:26, Sergey Chernolyas : > > My jiraId is schernolyas > > -- > - > > With best regards, Sergey Chernolyas -- Best regards, Ivan Pavlukhin
Re: Feature masks for thin clients
> A protocol for one feature can change from release to release. I think about it in a following way. Imagine that we reworked somehow affinity awareness and changes are needed on a protocol level. From a features point of view old feature is AffinityAwarenessV1 and new one is AffinityAwarenessV2. > Probably the "feature" word is confusing here. Maybe we should use term > "Changes Mask" or something like that. Bitmask layout seems to be an implementation detail. From a high level the subject sounds to me more like "Feature set". Client and server can support different subsets of all features. With feature versioning term "feature" might be confusing sometimes. But I am not sure that it is a big problem. Feel free to suggest better naming. Regarding the necessity of the subject. The main driver for me (it was already mentioned it the thread) is a simplification of clients for different languages development. I suppose the community do not have enough resources to support all features for all languages, consequently some workaround is needed. пт, 24 янв. 2020 г. в 19:32, Igor Sapego : > > Taras, > > I believe we should have the same mechanism but different > set of flags. > > Alexey, > > The idea is to add new flag when you make changes instead of > increment protocol version. This way client maintainer can introduce > the same change (probably fix) in they client without the necessity to > implement all other features that were introduced in between. > > Probably the "feature" word is confusing here. Maybe we should use > term "Changes Mask" or something like that. > > Best Regards, > Igor > > > On Thu, Jan 23, 2020 at 5:27 PM Pavel Tupitsyn wrote: > > > Igor, great idea, I think this should be our priority for all thin clients. > > > > Alexei, protocol version bump can be still required for major changes (e.g. > > if we want to change handshake format). > > But for simpler things like new features that don't affect existing ones, > > feature masks allow us to keep current protocol version. > > > > On Thu, Jan 23, 2020 at 3:55 PM Taras Ledkov wrote: > > > > > Alexei, > > > > > > After the flags is introduced we can change the flag set instead of > > > change protocol version. > > > Of course, we will need to up the protocol version for introducing > > flags. > > > > > > On 23.01.2020 15:47, Alexei Scherbakov wrote: > > > > Igor Sapego, > > > > > > > > I do not understand how feature masks can remove the necessity of > > having > > > > protocol versioning. > > > > A protocol for one feature can change from release to release. > > > > > > > > чт, 23 янв. 2020 г. в 15:36, Igor Sapego : > > > > > > > >> Hi Igniters, > > > >> > > > >> As we have a lot of different thin clients now, maintained by > > different > > > >> people, the issues with our backward compatibility mechanism becomes > > > >> more and more prominent. > > > >> > > > >> Currently, we use protocol versioning as the only approach to provide > > > >> backward compatibility. The main issue of this approach is that we can > > > >> not skip some change in protocol and implement i.e. protocol of > > version > > > >> 1.5 without implementation of 1.4. There are many cases when one may > > > >> want to do so: e.g. when feature provided in 1.4 is not relevant for a > > > >> specific client, or when protocol version 1.5 contains urgent fix or > > > >> feature > > > >> which is easy to implement, but its blocked by not-so-urgent and > > > >> hard-to-implement feature introduced in 1.4. > > > >> > > > >> So to fix this issue I propose to introduce another backward > > > compatibility > > > >> mechanism. The idea is to send "supported features" mask by a client > > to > > > >> a server, which should be answered with the same mask by the server. > > > >> The resulting set of enabled features is acquired with a simple > > logical > > > >> "AND" > > > >> operation on these two masks. > > > >> > > > >> This change has many other positive effects: > > > >> 1. It improves readability and also potentially simplifies debugging. > > > >> 2. It gives users the ability to enable or disable features of thin > > > clients > > > >> on both > > > >> server and client as they desire. > > > >> > > > >> What are your thoughts guys? > > > >> > > > >> Best Regards, > > > >> Igor > > > >> > > > > > > > -- > > > Taras Ledkov > > > Mail-To: tled...@gridgain.com > > > > > > > > -- Best regards, Ivan Pavlukhin
Re: Ignite-spring-boot-autoconfigurer
Hi Nikolay, Here is the release process docs for Ignite [1] We do not have docs specific to Ignite Extensions, we will need to create docs for Ignite Extensions. Please find discussion on release process below [2] Regards, Saikat [1] https://cwiki.apache.org/confluence/display/IGNITE/Release+Process [2] http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-dependencies-and-release-process-for-Ignite-Extensions-td44478.html On Tue, Jan 28, 2020 at 8:13 AM Nikolay Izhikov wrote: > Merged to master [1] > > Igniters, I want to release both modules I added. > Do we have documentation, how it should be done for ignite-extensions? > > [1] > > https://github.com/apache/ignite-extensions/commit/575ec510e4d48b1e0c2ec7e4cf8d596e7de82f70 > > пн, 27 янв. 2020 г. в 13:06, Nikolay Izhikov : > > > Hello, Alexey Kuznetsov. > > > > I have two approvals from Saikat Maitra and Maxim Stepachov. > > I have plans to merge spring-boot autoconfigure modules to > > ignite-extensions. [1] > > Do you want to perform additional review? > > > > [1] https://github.com/apache/ignite-extensions/pull/6 > > > > пт, 24 янв. 2020 г. в 07:41, Saikat Maitra : > > > >> Hi Nikolay, > >> > >> Thank you for updating the PR, the changes looks good. > >> > >> Regards, > >> Saikat > >> > >> On Wed, Jan 22, 2020 at 1:33 PM Николай Ижиков > >> wrote: > >> > >> > Hello, Saikat. > >> > > >> > Thank you so much for the review. > >> > > >> > I answered your questions and resolve all the comments. > >> > Please, take a look, one more time. > >> > > >> > > 22 янв. 2020 г., в 07:58, Saikat Maitra > >> > написал(а): > >> > > > >> > > Hi Nikolay, > >> > > > >> > > I have reviewed the PR and shared comments. > >> > > > >> > > Please let me know if you have any feedback. > >> > > > >> > > Regards, > >> > > Saikat > >> > > > >> > > On Mon, Jan 20, 2020 at 2:42 PM Николай Ижиков > > >> > wrote: > >> > > > >> > >> Hello, Saikat. > >> > >> > >> > >> Thanks, for feedback. > >> > >> > >> > >> I raised a PR [1] to `ignite-extensions`. > >> > >> > >> > >> You can find description of the new module below(examples can be > >> found > >> > at > >> > >> [2]): > >> > >> > >> > >> Module provides the ability to integrate `Ignite` into you > >> spring-boot > >> > >> application with zero(or minimal) configuration. > >> > >> > >> > >> After you add this module as a dependency to your spring-boot > >> > application > >> > >> `Ignite` node will be configured and injected into `BeanFactory`. > >> > >> > >> > >> Algorithm to configure `Ignite` is the following: > >> > >> 1. If `IgniteConfiguration` bean exists in the `BeanFactory` it > >> will be > >> > >> used. > >> > >> 2. If `IgniteConfiguration` bean doesn't exist following rules are > >> > >> applied: > >> > >>2.1. Default `Ignite` configuration created. > >> > >>2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will > be > >> > >> used to customize `IgniteConfiguration`. > >> > >> If a user wants to set custom SPI instances or similar > >> hardcoded > >> > >> values > >> > >> one should do it with `IgniteConfigurer` implementation. > >> > >>2.3 Application properties applied to `IgniteConfiguration`. > >> Prefix > >> > >> for the properties is `ignite`. > >> > >> > >> > >> > >> > >> [1] https://github.com/apache/ignite-extensions/pull/6 > >> > >> [2] > >> > https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example > >> > >> > >> > >> > >> > >>> 18 янв. 2020 г., в 06:44, Saikat Maitra > >> > >> написал(а): > >> > >>> > >> > >>> Hi Nikolay, > >> > >>> > >> > >>> Thank you for your email. As part of Ignite Extensions migration > we > >> are > >> > >> migrating Ignite Extensions to following repo. > >> > >>> > >> > >>> https://github.com/apache/ignite-extensions > >> > >>> > >> > >>> We have added flink and pub-sub modules and few additional modules > >> are > >> > >> open in PR. > >> > >>> > >> > >>> You can refer to this PR to see how we are migrating the modules > >> > >> https://github.com/apache/ignite-extensions/pull/5 > >> > >>> > >> > >>> I wanted to connect and discuss the changes to understand the > spring > >> > >> boot auto configure feature. We currently have an ignite spring > >> module > >> > that > >> > >> allows resource injection capabilities and provides a parser for > >> Spring > >> > >> based xml configuration files. Can you please review and share if > the > >> > >> changes you are proposing can be added as part of Ignite spring > >> module > >> > or > >> > >> it make sense to make it a separate spring boot auto configure > >> module. > >> > >>> > >> > >>> https://github.com/apache/ignite/tree/master/modules/spring > >> > >>> > >> > >>> Regards, > >> > >>> Saikat > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков < > nizhi...@apache.org > >> > > >> > >> wrote: > >> > >>> Tests added. > >> > >>> Please, review. > >> > >>> > >> > >>> Saikat, can you help
Hello
My jiraId is schernolyas -- - With best regards, Sergey Chernolyas
Hello
-- - With best regards, Sergey Chernolyas
New Blog: Why my in-memory Ignite cluster underperforms? Network and co-located compute
Igniters, I've just published an article that describes one of the reasons for why an Ignite cluster can underperform. It's reviewed how a selection of proper Ignite APIs can put less pressure on the network and proliferate the performance: https://www.gridgain.com/resources/blog/why-my-in-memory-cluster-underperforms-negating-network-impact Let me know what you think. Suggest any other themes you would want to be laid out. Btw, our community member Denis Mekhanikov is doing a webinar about co-located computations in Ignite tomorrow, join him to learn more about this area: https://www.gridgain.com/resources/webinars/distributed-computing-apache-ignite - Denis
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Igniters, Here is the list of actual release BLOCKER issues: [1] Keep in mind unfinished discussion about internal classes IGNITE-12456 [2] Cluster Data Store grid gets Corrupted for Load test *[Unassigned]* OPEN IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes getting down [Emmanouil Gkatziouras] IN PROGRESS IGNITE-12580 NPE in GridMetricManager [Nikolay Izhikov] PATCH AVAILABLE IGNITE-12489 Error during purges by expiration: Unknown page type [Anton Kalashnikov] OPEN [1] http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html [2] https://issues.apache.org/jira/browse/IGNITE-12456 [3] https://issues.apache.org/jira/browse/IGNITE-12398 [4] https://issues.apache.org/jira/browse/IGNITE-12580 [5] https://issues.apache.org/jira/browse/IGNITE-12489 On Tue, 28 Jan 2020 at 19:25, Maxim Muzafarov wrote: > Andrey, > > I've looked through those changes [1] and now they look good to me. > Let's do the following: > > 1. Get a fresh TC.Bot visa > 2. Merge these changes to the master branch. > 3. After that and 3-day stabilization cherry-pick to 2.8 > > Should we wait for benchmarks? I think at this release stage any > additional benchmarks can eliminate our risks with extending scope. > We've already had one - [2] (2.7.6 compared to 2.8). > > > [1] https://issues.apache.org/jira/browse/IGNITE-12576 > [2] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks > > On Mon, 27 Jan 2020 at 23:58, Nikolay Izhikov wrote: > > > > Andrey. > > > > > My choice: correctness over performance > > > > I don’t think we should select performance OR correctness here. > > It seems we can got both. > > > > > May be we should rollback all metrics related changes because we don't > have benchmark results > > > > I perform benchmarking for initial refactoring of > TcpCommunicationMetricsListener. > > Initial refactoring of TcpCommunicationMetricsListener doesn’t bring any > performance drop according to the results of the tests I performed. > > > > I want to perform benchmarking just to be sure everything OK. > > Please, wait while I gather benchmark results for this PR. > > > > > 27 янв. 2020 г., в 22:33, Andrey Gura написал(а): > > > > > >> We still can’t accept patches that badly affects the performance of > TcpCommuncationMetricsListener. > > >> So we should perform yardstick tests before the merge. > > > > > > Absolutely all metrics are on the hot path. They inevitably affect > > > performance and this case is the same. May be we should rollback all > > > metrics related changes because we don't have benchmark results& > > > > > >> I can help to run yardstick benchmarks if you don’t have free servers > to do it. > > > > > > I don't need help in benchmarking. Once again, еhe current behavior is > > > incorrect and should be fixed regardless of performance. > > > > > > Or... this functionality should be removed if performance is more > > > important. In case of incorrect behavior it is the best option. > > > > > > My choice: correctness over performance. > > > > > > On Mon, Jan 27, 2020 at 10:02 PM Nikolay Izhikov > wrote: > > >> > > >>> I think it could be fixed easily by adding metricsEnabled flag to > TcpCommunicationSpi. > > >> > > >> We still can’t accept patches that badly affects the performance of > TcpCommuncationMetricsListener. > > >> So we should perform yardstick tests before the merge. > > >> > > >> I can help to run yardstick benchmarks if you don’t have free servers > to do it. > > >> > > >> > > >>> 27 янв. 2020 г., в 21:47, Andrey Gura написал(а): > > >>> > > > "If it doesn’t work, it doesn’t matter how fast it doesn’t work." > (c) > > Please, clarify, what do you mean by «doesn’t work»? > > Are there any unresolved bugs? > > >>> > > >>> Obviously some communication metrics can't be monitored or analyzed > > >>> retrospectively due to changing node ID during node restart. It's > bug. > > >>> > > > User can disable metrics if it will affect performance. > > Users can’t disable TcpCommunicationListener nor in any release nor > in current master so we should change this code carefully > > >>> > > >>> This is another bug. I think it could be fixed easily by adding > > >>> metricsEnabled flag to TcpCommunicationSpi. > > >>> > > >>> On Mon, Jan 27, 2020 at 9:17 PM Nikolay Izhikov > wrote: > > > > Andrey. > > > > > "If it doesn’t work, it doesn’t matter how fast it doesn’t work." > (c) > > > > Please, clarify, what do you mean by «doesn’t work»? > > Are there any unresolved bugs? > > > > > IGINTE-12576 affects it minimally > > > > All I asking for is to confirm this statement with the benchmark > results. > > > > > User can disable metrics if it will affect performance. > > > > Users can’t disable TcpCommunicationListener nor in any release nor > in current master so we should change this code carefully > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Maxim. > Should we wait for benchmarks? After review, these changes looks much safer for me - no additional metrics added. I performed benchmarking for initial refactoring of `TcpCommunicationMetricsListener` on the new Metric API. It seems, there is no need for benchmarking anymore. > 28 янв. 2020 г., в 19:25, Maxim Muzafarov написал(а): > > Andrey, > > I've looked through those changes [1] and now they look good to me. > Let's do the following: > > 1. Get a fresh TC.Bot visa > 2. Merge these changes to the master branch. > 3. After that and 3-day stabilization cherry-pick to 2.8 > > Should we wait for benchmarks? I think at this release stage any > additional benchmarks can eliminate our risks with extending scope. > We've already had one - [2] (2.7.6 compared to 2.8). > > > [1] https://issues.apache.org/jira/browse/IGNITE-12576 > [2] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks > > On Mon, 27 Jan 2020 at 23:58, Nikolay Izhikov wrote: >> >> Andrey. >> >>> My choice: correctness over performance >> >> I don’t think we should select performance OR correctness here. >> It seems we can got both. >> >>> May be we should rollback all metrics related changes because we don't have >>> benchmark results >> >> I perform benchmarking for initial refactoring of >> TcpCommunicationMetricsListener. >> Initial refactoring of TcpCommunicationMetricsListener doesn’t bring any >> performance drop according to the results of the tests I performed. >> >> I want to perform benchmarking just to be sure everything OK. >> Please, wait while I gather benchmark results for this PR. >> >>> 27 янв. 2020 г., в 22:33, Andrey Gura написал(а): >>> We still can’t accept patches that badly affects the performance of TcpCommuncationMetricsListener. So we should perform yardstick tests before the merge. >>> >>> Absolutely all metrics are on the hot path. They inevitably affect >>> performance and this case is the same. May be we should rollback all >>> metrics related changes because we don't have benchmark results& >>> I can help to run yardstick benchmarks if you don’t have free servers to do it. >>> >>> I don't need help in benchmarking. Once again, еhe current behavior is >>> incorrect and should be fixed regardless of performance. >>> >>> Or... this functionality should be removed if performance is more >>> important. In case of incorrect behavior it is the best option. >>> >>> My choice: correctness over performance. >>> >>> On Mon, Jan 27, 2020 at 10:02 PM Nikolay Izhikov >>> wrote: > I think it could be fixed easily by adding metricsEnabled flag to > TcpCommunicationSpi. We still can’t accept patches that badly affects the performance of TcpCommuncationMetricsListener. So we should perform yardstick tests before the merge. I can help to run yardstick benchmarks if you don’t have free servers to do it. > 27 янв. 2020 г., в 21:47, Andrey Gura написал(а): > >>> "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) >> Please, clarify, what do you mean by «doesn’t work»? >> Are there any unresolved bugs? > > Obviously some communication metrics can't be monitored or analyzed > retrospectively due to changing node ID during node restart. It's bug. > >>> User can disable metrics if it will affect performance. >> Users can’t disable TcpCommunicationListener nor in any release nor in >> current master so we should change this code carefully > > This is another bug. I think it could be fixed easily by adding > metricsEnabled flag to TcpCommunicationSpi. > > On Mon, Jan 27, 2020 at 9:17 PM Nikolay Izhikov > wrote: >> >> Andrey. >> >>> "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) >> >> Please, clarify, what do you mean by «doesn’t work»? >> Are there any unresolved bugs? >> >>> IGINTE-12576 affects it minimally >> >> All I asking for is to confirm this statement with the benchmark results. >> >>> User can disable metrics if it will affect performance. >> >> Users can’t disable TcpCommunicationListener nor in any release nor in >> current master so we should change this code carefully >> >> https://github.com/apache/ignite/blob/ignite-2.7.6/modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpi.java#L1178 >> >>> 27 янв. 2020 г., в 20:40, Andrey Gura написал(а): >>> >>> Nikolay, >>> But, we must gather yardstick benchmark results for PR(comparing to current master) before merge to ensure there is no performance drop. >>> >>> "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) >>> >>> I believe that benchmarks ignite-2.7.6 vs ignite-2.8 will show >>>
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Andrey, I've looked through those changes [1] and now they look good to me. Let's do the following: 1. Get a fresh TC.Bot visa 2. Merge these changes to the master branch. 3. After that and 3-day stabilization cherry-pick to 2.8 Should we wait for benchmarks? I think at this release stage any additional benchmarks can eliminate our risks with extending scope. We've already had one - [2] (2.7.6 compared to 2.8). [1] https://issues.apache.org/jira/browse/IGNITE-12576 [2] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks On Mon, 27 Jan 2020 at 23:58, Nikolay Izhikov wrote: > > Andrey. > > > My choice: correctness over performance > > I don’t think we should select performance OR correctness here. > It seems we can got both. > > > May be we should rollback all metrics related changes because we don't have > > benchmark results > > I perform benchmarking for initial refactoring of > TcpCommunicationMetricsListener. > Initial refactoring of TcpCommunicationMetricsListener doesn’t bring any > performance drop according to the results of the tests I performed. > > I want to perform benchmarking just to be sure everything OK. > Please, wait while I gather benchmark results for this PR. > > > 27 янв. 2020 г., в 22:33, Andrey Gura написал(а): > > > >> We still can’t accept patches that badly affects the performance of > >> TcpCommuncationMetricsListener. > >> So we should perform yardstick tests before the merge. > > > > Absolutely all metrics are on the hot path. They inevitably affect > > performance and this case is the same. May be we should rollback all > > metrics related changes because we don't have benchmark results& > > > >> I can help to run yardstick benchmarks if you don’t have free servers to > >> do it. > > > > I don't need help in benchmarking. Once again, еhe current behavior is > > incorrect and should be fixed regardless of performance. > > > > Or... this functionality should be removed if performance is more > > important. In case of incorrect behavior it is the best option. > > > > My choice: correctness over performance. > > > > On Mon, Jan 27, 2020 at 10:02 PM Nikolay Izhikov > > wrote: > >> > >>> I think it could be fixed easily by adding metricsEnabled flag to > >>> TcpCommunicationSpi. > >> > >> We still can’t accept patches that badly affects the performance of > >> TcpCommuncationMetricsListener. > >> So we should perform yardstick tests before the merge. > >> > >> I can help to run yardstick benchmarks if you don’t have free servers to > >> do it. > >> > >> > >>> 27 янв. 2020 г., в 21:47, Andrey Gura написал(а): > >>> > > "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) > Please, clarify, what do you mean by «doesn’t work»? > Are there any unresolved bugs? > >>> > >>> Obviously some communication metrics can't be monitored or analyzed > >>> retrospectively due to changing node ID during node restart. It's bug. > >>> > > User can disable metrics if it will affect performance. > Users can’t disable TcpCommunicationListener nor in any release nor in > current master so we should change this code carefully > >>> > >>> This is another bug. I think it could be fixed easily by adding > >>> metricsEnabled flag to TcpCommunicationSpi. > >>> > >>> On Mon, Jan 27, 2020 at 9:17 PM Nikolay Izhikov > >>> wrote: > > Andrey. > > > "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) > > Please, clarify, what do you mean by «doesn’t work»? > Are there any unresolved bugs? > > > IGINTE-12576 affects it minimally > > All I asking for is to confirm this statement with the benchmark results. > > > User can disable metrics if it will affect performance. > > Users can’t disable TcpCommunicationListener nor in any release nor in > current master so we should change this code carefully > > https://github.com/apache/ignite/blob/ignite-2.7.6/modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpi.java#L1178 > > > 27 янв. 2020 г., в 20:40, Andrey Gura написал(а): > > > > Nikolay, > > > >> But, we must gather yardstick benchmark results for PR(comparing to > >> current master) before merge to ensure there is no performance drop. > > > > "If it doesn’t work, it doesn’t matter how fast it doesn’t work." (c) > > > > I believe that benchmarks ignite-2.7.6 vs ignite-2.8 will show > > noticeable drop in performance for ignite-2.8. But it is cumulative > > effect and IGINTE-12576 affects it minimally. > > > >> Note, that these metrics updated on each communication message. > > > > Metrics are not free at all. User can disable metrics if it will > > affect performance. > > > > On Mon, Jan 27, 2020 at 8:23 PM Nikolay Izhikov > > wrote: > >> > >> Hello, Andrey. > >> > >>
Re: Internal classes are exposed in public API
Maxim. I’m +1 to do it. > 1. Let's keep @deprecated annotation to notify users earlier. > 2. Let's fix javadoc comments for this annotation (remove internal > references). We also have a consensus to add IgniteExperimental annotation to the new metrics API. > 28 янв. 2020 г., в 19:05, Maxim Muzafarov написал(а): > > Folks, > > > Let me share my humble opinion over this thread. The first priority > here is making Ignite better and taking the next step with releasing > Ignite's 2.8 version for our users (we've already done a great job > with metrics). > > According to this point of view and from the user perspective I'd like > to be notified about changing Ignite's architecture and concepts of > using metrics as early as possible not only through the public > documentation but through the code I write also. > > So, I see nothing strange by keeping @deprecated annotation (Javadoc > comments must be fixed, true): > 1. it's just a hint for users (we will notify them as earlier as > possible but they can still write a code using deprecated methods) > 2. users will be notified about new JmxMetricExporterSpi through the > code and documentation. > 3. it's a recommendation for developers that they should not add new > methods to the old facade. > > I also see nothing strange by removing @deprecated annotation > currently added to DataRegionMetrics interface. It's also OK if we are > still developing new facade and do not want to share some public API > changes with users because all the 2.8 release looks for me like an > experimental. > > > So, summarizing: > 1. Let's keep @deprecated annotation to notify users earlier. > 2. Let's fix javadoc comments for this annotation (remove internal > references). > 3. Let's do not too much debate here. This question isn't too > important compared to 23 critical release issues [1]. > > > Alexey, Andrey, Nikolay, > > Can we proceed this way? > > > P.S. > From the release managers view, it's sad to see such a discussions > about API changes and compatibility because we've discussed API at > July 2019 [3] and all issues with monitoring activities at October > 2019 [2]. > > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolvedissues(notrelatedtodocumentation) > [2] > http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-8-RELEASE-Time-Scope-Manager-tp43616p43807.html > [3] https://issues.apache.org/jira/browse/IGNITE-11848 > > On Mon, 27 Jan 2020 at 22:23, Andrey Gura wrote: >> I just want to remove the deprecations on the classes for which there is no a well-defined replacement. >>> Well-defined replacement for these classes exists. >> >> Nikolay, statements like this are not arguments. At least because >> there are people who disagree with this. >> >> Current implementation is not well-defined due to API is not >> well-defined enough. Also there are lack of functionality >> (enabling/disabling metrics, requirement for end user to configure new >> exporter in order to get a couple of new metrics while it is still >> possible to use old MBeans but without this metrics) and potential >> performance problems (that couldn't be solved because metrics couldn't >> be switched off). >> >> We are walking in a circle instead of moving forward. On comments >> about API you provide new PRs that change API while denying problems >> with the API. This is illogical and process doesn't converge because >> it is attempt to close gaps in a hurry. >> >> The best we can do now is stop, take a deep breath and follow the >> offers of Alexey about deprecation, my proposal and Denis' idea about >> one additional iteration. >> >> On Mon, Jan 27, 2020 at 4:37 PM Nikolay Izhikov wrote: >>> it's better to remove deprecation for the old APIs in AI 2.8 >>> >>> Are you suggest to have 2 concurrent metrics implementations(both not >>> deprecated)? >>> Let's say I am writing a custom exporter in binary format and I need to write the number of exported metrics in a packet beforehand. >>> >>> This an `MetricExporterSPI` implementation task. >>> Exporter has the listeners that can be used to track all registry events - >>> creation, removal >>> There is no separation on public and internal metrics >>> >>> I think we should write in the documentation that ANY metric can be changed. >>> We will try to keep them, but if we found that some metric doesn’t required >>> any more or implemented in the wrong way - we can remove it. >>> Will we allow users to register their own metrics? >>> >>> No. Why we should considering this feature? >>> It's still not clear how a user will map old interfaces and methods to the new metric names. >>> >>> With the documentation. >>> We can write separate deprecation comment for each metric providing method. >>> I just want to remove the deprecations on the classes for which there is no a well-defined replacement. >>>
Re: Internal classes are exposed in public API
Folks, Let me share my humble opinion over this thread. The first priority here is making Ignite better and taking the next step with releasing Ignite's 2.8 version for our users (we've already done a great job with metrics). According to this point of view and from the user perspective I'd like to be notified about changing Ignite's architecture and concepts of using metrics as early as possible not only through the public documentation but through the code I write also. So, I see nothing strange by keeping @deprecated annotation (Javadoc comments must be fixed, true): 1. it's just a hint for users (we will notify them as earlier as possible but they can still write a code using deprecated methods) 2. users will be notified about new JmxMetricExporterSpi through the code and documentation. 3. it's a recommendation for developers that they should not add new methods to the old facade. I also see nothing strange by removing @deprecated annotation currently added to DataRegionMetrics interface. It's also OK if we are still developing new facade and do not want to share some public API changes with users because all the 2.8 release looks for me like an experimental. So, summarizing: 1. Let's keep @deprecated annotation to notify users earlier. 2. Let's fix javadoc comments for this annotation (remove internal references). 3. Let's do not too much debate here. This question isn't too important compared to 23 critical release issues [1]. Alexey, Andrey, Nikolay, Can we proceed this way? P.S. >From the release managers view, it's sad to see such a discussions about API changes and compatibility because we've discussed API at July 2019 [3] and all issues with monitoring activities at October 2019 [2]. [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolvedissues(notrelatedtodocumentation) [2] http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-8-RELEASE-Time-Scope-Manager-tp43616p43807.html [3] https://issues.apache.org/jira/browse/IGNITE-11848 On Mon, 27 Jan 2020 at 22:23, Andrey Gura wrote: > > >> I just want to remove the deprecations on the classes for which there is > >> no a well-defined replacement. > > Well-defined replacement for these classes exists. > > Nikolay, statements like this are not arguments. At least because > there are people who disagree with this. > > Current implementation is not well-defined due to API is not > well-defined enough. Also there are lack of functionality > (enabling/disabling metrics, requirement for end user to configure new > exporter in order to get a couple of new metrics while it is still > possible to use old MBeans but without this metrics) and potential > performance problems (that couldn't be solved because metrics couldn't > be switched off). > > We are walking in a circle instead of moving forward. On comments > about API you provide new PRs that change API while denying problems > with the API. This is illogical and process doesn't converge because > it is attempt to close gaps in a hurry. > > The best we can do now is stop, take a deep breath and follow the > offers of Alexey about deprecation, my proposal and Denis' idea about > one additional iteration. > > On Mon, Jan 27, 2020 at 4:37 PM Nikolay Izhikov wrote: > > > > > it's better to remove deprecation for the old APIs in AI 2.8 > > > > Are you suggest to have 2 concurrent metrics implementations(both not > > deprecated)? > > > > > Let's say I am writing a custom exporter in binary format and I need to > > > write the number of exported metrics in a packet beforehand. > > > > This an `MetricExporterSPI` implementation task. > > Exporter has the listeners that can be used to track all registry events - > > creation, removal > > > > > There is no separation on public and internal metrics > > > > I think we should write in the documentation that ANY metric can be changed. > > We will try to keep them, but if we found that some metric doesn’t required > > any more or implemented in the wrong way - we can remove it. > > > > > Will we allow users to register their own metrics? > > > > No. Why we should considering this feature? > > > > > It's still not clear how a user will map old interfaces and methods to > > > the new metric names. > > > > With the documentation. > > We can write separate deprecation comment for each metric providing method. > > > > > I just want to remove the deprecations on the classes for which there is > > > no a well-defined replacement. > > > > Well-defined replacement for these classes exists. > > > > > 27 янв. 2020 г., в 16:21, Alexey Goncharuk > > > написал(а): > > > > > > Nikolay, > > > > > > I've reviewed your changes and I tend to agree with Andrey, it's better to > > > remove deprecation for the old APIs in AI 2.8 and discuss and merge the > > > new > > > facade (IGNITE-12553) in a more calm and structured way. My observations > > > on > > > the changes: > > > > > > - I am not sure if
Cache 6 suite is broken
Looks like i found problem root cause, first run completed on TC, hope it would be ok with other builds. > > >--- Forwarded message --- >From: "Ivan Pavlukhin" < vololo...@gmail.com > >To: dev < dev@ignite.apache.org > >Cc: >Subject: Re: Cache 6 suite is broken >Date: Fri, 20 Dec 2019 10:04:07 +0300 > >FYI > >I applied the patch. > >ср, 18 дек. 2019 г. в 14:46, Ivan Pavlukhin < vololo...@gmail.com >: >> >> So, if nobody objects, I will merge PR [1] on Friday. >> >> [1] https://github.com/apache/ignite/pull/7142 >> >> ср, 18 дек. 2019 г. в 13:13, Dmitriy Pavlov < dpav...@apache.org >: >> > >> > Hi Ivan, Igniters, >> > >> > My proposal is to apply PR and take a look to wider statistics in the >> > master branch. WDYT? >> > >> > Sincerely, >> > Dmitriy Pavlov >> > >> > ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin < vololo...@gmail.com >: >> > >> > > Igniters, >> > > >> > > I run tests several times for my PR [1] and results [2] looks somehow >> > > better than in master [3]. But the problematic test seems too flaky. >> > > So, I cannot decide between 2 options: >> > > 1. Apply PR [1]. >> > > 2. Ignore the problematic test. >> > > (Unfortunately do not have a chance to investigate and fix) >> > > >> > > Someone's opinion would be helpful here. >> > > >> > > [1] https://github.com/apache/ignite/pull/7142 >> > > [2] >> > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead=overview >> > > [3] >> > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E=overview >> > > >> > > пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin < vololo...@gmail.com >: >> > > > >> > > > An update. After increasing NetworkTimeout [1] the test passes >> rarely >> > > [2]. >> > > > >> > > > [1] https://github.com/apache/ignite/pull/7142/files >> > > > [2] >> > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead=overview >> > > > >> > > > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin < vololo...@gmail.com >: >> > > > > >> > > > > Hi, >> > > > > >> > > > > I noticied that Cache 6 suite finishes badly (exit code 137) >> almost >> > > > > everytime [1] on master. It started recently. A problematic test >> > > > > inside is IgniteCache150ClientsTest. I made some attempts to fix >> the >> > > > > test (including reverting recent commits), but had no luck. And >> a hard >> > > > > part here is that I do not know whether the issue was caused by >> code >> > > > > changes or it is infrastructural one. My best attempt was >> increasing >> > > > > NetworkTimeout in test IgniteConfiguration [2]. After that the >> suite >> > > > > was able to finish, but IgniteCache150ClientsTest failed. Also, >> the >> > > > > problematic test seems so flaky, there are many failures in >> history. >> > > > > >> > > > > Do not have any really good solution in my mind. I see following >> > > options: >> > > > > 1. Use workaround in PR [2]. >> > > > > 2. Ignore test and dig deeper in scope of a separate ticket. >> > > > > >> > > > > What do you think? >> > > > > >> > > > > [1] >> > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E=overview >> > > > > [2] https://github.com/apache/ignite/pull/7142/files >> > > > > -- >> > > > > Best regards, >> > > > > Ivan Pavlukhin >> > > > >> > > > >> > > > >> > > > -- >> > > > Best regards, >> > > > Ivan Pavlukhin >> > > >> > > >> > > >> > > -- >> > > Best regards, >> > > Ivan Pavlukhin >> > > >> >> >> >> -- >> Best regards, >> Ivan Pavlukhin
[DISCUSS] Relevance of CacheConfiguration.DefaultLockTimeout
Hi, I failed to find where CacheConfiguration.DefaultLockTimeout property is used. Does anyone know how is it used? If it is not used I suppose we can mark this property deprecated. Please share your thoughts! -- Best regards, Ivan Pavlukhin
[jira] [Created] (IGNITE-12597) IgniteTxStoreExceptionAbstractSelfTest
Vladimir Steshin created IGNITE-12597: - Summary: IgniteTxStoreExceptionAbstractSelfTest Key: IGNITE-12597 URL: https://issues.apache.org/jira/browse/IGNITE-12597 Project: Ignite Issue Type: Sub-task Reporter: Vladimir Steshin Assignee: Vladimir Steshin org.apache.ignite.internal.processors.cache.GridCacheColocatedTxStoreExceptionSelfTest might be parametrized. Extending classes wear only params and are executed in a row -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12596) Parametrization of IgniteCacheAbstractExecutionContextTest
Vladimir Steshin created IGNITE-12596: - Summary: Parametrization of IgniteCacheAbstractExecutionContextTest Key: IGNITE-12596 URL: https://issues.apache.org/jira/browse/IGNITE-12596 Project: Ignite Issue Type: Sub-task Environment: org.apache.ignite.internal.processors.cache.context.IgniteCacheAbstractExecutionContextTest is activated 3 times with just various params via inheritance. The problem is that the extending classes are included in the target test suits not always with entire combinations of params. Sometimes only 2 extendins classes are involved within tests, sometimes 3. I think of using subclasses of IgniteCacheAbstractExecutionContextTest as set of params. Reporter: Vladimir Steshin Assignee: Vladimir Steshin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12595) Parametrization of GridCacheSetAbstractSelfTest
Vladimir Steshin created IGNITE-12595: - Summary: Parametrization of GridCacheSetAbstractSelfTest Key: IGNITE-12595 URL: https://issues.apache.org/jira/browse/IGNITE-12595 Project: Ignite Issue Type: Sub-task Reporter: Vladimir Steshin Assignee: Vladimir Steshin org.apache.ignite.internal.processors.cache.datastructures.GridCacheSetAbstractSelfTest might be used with params. Not the best candidate, but is still able to reduce tests code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Ignite-spring-boot-autoconfigurer
Merged to master [1] Igniters, I want to release both modules I added. Do we have documentation, how it should be done for ignite-extensions? [1] https://github.com/apache/ignite-extensions/commit/575ec510e4d48b1e0c2ec7e4cf8d596e7de82f70 пн, 27 янв. 2020 г. в 13:06, Nikolay Izhikov : > Hello, Alexey Kuznetsov. > > I have two approvals from Saikat Maitra and Maxim Stepachov. > I have plans to merge spring-boot autoconfigure modules to > ignite-extensions. [1] > Do you want to perform additional review? > > [1] https://github.com/apache/ignite-extensions/pull/6 > > пт, 24 янв. 2020 г. в 07:41, Saikat Maitra : > >> Hi Nikolay, >> >> Thank you for updating the PR, the changes looks good. >> >> Regards, >> Saikat >> >> On Wed, Jan 22, 2020 at 1:33 PM Николай Ижиков >> wrote: >> >> > Hello, Saikat. >> > >> > Thank you so much for the review. >> > >> > I answered your questions and resolve all the comments. >> > Please, take a look, one more time. >> > >> > > 22 янв. 2020 г., в 07:58, Saikat Maitra >> > написал(а): >> > > >> > > Hi Nikolay, >> > > >> > > I have reviewed the PR and shared comments. >> > > >> > > Please let me know if you have any feedback. >> > > >> > > Regards, >> > > Saikat >> > > >> > > On Mon, Jan 20, 2020 at 2:42 PM Николай Ижиков >> > wrote: >> > > >> > >> Hello, Saikat. >> > >> >> > >> Thanks, for feedback. >> > >> >> > >> I raised a PR [1] to `ignite-extensions`. >> > >> >> > >> You can find description of the new module below(examples can be >> found >> > at >> > >> [2]): >> > >> >> > >> Module provides the ability to integrate `Ignite` into you >> spring-boot >> > >> application with zero(or minimal) configuration. >> > >> >> > >> After you add this module as a dependency to your spring-boot >> > application >> > >> `Ignite` node will be configured and injected into `BeanFactory`. >> > >> >> > >> Algorithm to configure `Ignite` is the following: >> > >> 1. If `IgniteConfiguration` bean exists in the `BeanFactory` it >> will be >> > >> used. >> > >> 2. If `IgniteConfiguration` bean doesn't exist following rules are >> > >> applied: >> > >>2.1. Default `Ignite` configuration created. >> > >>2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will be >> > >> used to customize `IgniteConfiguration`. >> > >> If a user wants to set custom SPI instances or similar >> hardcoded >> > >> values >> > >> one should do it with `IgniteConfigurer` implementation. >> > >>2.3 Application properties applied to `IgniteConfiguration`. >> Prefix >> > >> for the properties is `ignite`. >> > >> >> > >> >> > >> [1] https://github.com/apache/ignite-extensions/pull/6 >> > >> [2] >> > https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example >> > >> >> > >> >> > >>> 18 янв. 2020 г., в 06:44, Saikat Maitra >> > >> написал(а): >> > >>> >> > >>> Hi Nikolay, >> > >>> >> > >>> Thank you for your email. As part of Ignite Extensions migration we >> are >> > >> migrating Ignite Extensions to following repo. >> > >>> >> > >>> https://github.com/apache/ignite-extensions >> > >>> >> > >>> We have added flink and pub-sub modules and few additional modules >> are >> > >> open in PR. >> > >>> >> > >>> You can refer to this PR to see how we are migrating the modules >> > >> https://github.com/apache/ignite-extensions/pull/5 >> > >>> >> > >>> I wanted to connect and discuss the changes to understand the spring >> > >> boot auto configure feature. We currently have an ignite spring >> module >> > that >> > >> allows resource injection capabilities and provides a parser for >> Spring >> > >> based xml configuration files. Can you please review and share if the >> > >> changes you are proposing can be added as part of Ignite spring >> module >> > or >> > >> it make sense to make it a separate spring boot auto configure >> module. >> > >>> >> > >>> https://github.com/apache/ignite/tree/master/modules/spring >> > >>> >> > >>> Regards, >> > >>> Saikat >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков > > >> > >> wrote: >> > >>> Tests added. >> > >>> Please, review. >> > >>> >> > >>> Saikat, can you help with this PR [1]? >> > >>> >> > >>> I think it should be added as a separate module as we do with the >> flink >> > >> integration. >> > >>> Can you help me with it? >> > >>> Do we have some how-to for it? >> > >>> >> > >>> [1] https://github.com/apache/ignite/pull/7237 >> > >>> >> > 16 янв. 2020 г., в 16:51, Николай Ижиков >> > >> написал(а): >> > >> > Hello, Denis. >> > >> > Thanks, for the feedback. >> > >> > Alexey, it seems, PR is ready to be reviewed, but I need some >> time(a >> > >> day or two) to write tests. >> > You can start with the core code review if you wish. >> > >> > Here are autoconfigurer requirements: >> > >> > 1. Start usage of Ignite with minimal(or zero) configuration. >> > 2. Configure Ignite configuration properties with
[jira] [Created] (IGNITE-12594) Deadlock between GridCacheDataStore#purgeExpiredInternal and GridNearTxLocal#enlistWriteEntry
Anton Kalashnikov created IGNITE-12594:
--
Summary: Deadlock between GridCacheDataStore#purgeExpiredInternal
and GridNearTxLocal#enlistWriteEntry
Key: IGNITE-12594
URL: https://issues.apache.org/jira/browse/IGNITE-12594
Project: Ignite
Issue Type: Bug
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov
The deadlock is reproduced occasionally in PDS3 suite and can be seen in the
thread dump below.
One thread attempts to unwind evicts, acquires checkpoint read lock and then
locks {{GridCacheMapEntry}}. Another thread does {{GridCacheMapEntry#unswap}},
determines that the entry is expired and acquires checkpoint read lock to
remove the entry from the store.
We should not acquire checkpoint read lock inside of a locked
{{GridCacheMapEntry}}.
{code:java}Thread [name="updater-1", id=29900, state=WAITING, blockCnt=2,
waitCnt=4450]
Lock
[object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2fc51685,
ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1632)
<- CP read lock
at
o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:4081)
at
o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:559)
at
o.a.i.i.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:519)
<- locked entry
at
o.a.i.i.processors.cache.distributed.near.GridNearTxLocal.enlistWriteEntry(GridNearTxLocal.java:1437)
at
o.a.i.i.processors.cache.distributed.near.GridNearTxLocal.enlistWrite(GridNearTxLocal.java:1303)
at
o.a.i.i.processors.cache.distributed.near.GridNearTxLocal.putAllAsync0(GridNearTxLocal.java:957)
at
o.a.i.i.processors.cache.distributed.near.GridNearTxLocal.putAllAsync(GridNearTxLocal.java:491)
at
o.a.i.i.processors.cache.GridCacheAdapter$29.inOp(GridCacheAdapter.java:2526)
at
o.a.i.i.processors.cache.GridCacheAdapter$SyncInOp.op(GridCacheAdapter.java:4727)
at
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3740)
at
o.a.i.i.processors.cache.GridCacheAdapter.putAll0(GridCacheAdapter.java:2524)
at
o.a.i.i.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2513)
at
o.a.i.i.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1264)
at
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:863)
at
o.a.i.i.processors.cache.persistence.IgnitePdsContinuousRestartTest$1.call(IgnitePdsContinuousRestartTest.java:291)
at o.a.i.testframework.GridTestThread.run(GridTestThread.java:83)
Locked synchronizers:
java.util.concurrent.locks.ReentrantLock$NonfairSync@762613f7
Thread
[name="sys-stripe-0-#24086%persistence.IgnitePdsContinuousRestartTestWithExpiryPolicy0%",
id=29617, state=WAITING, blockCnt=2, waitCnt=65381]
Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@762613f7,
ownerName=updater-1, ownerId=29900]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
<- lock entry
at
o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:5017)
at
o.a.i.i.processors.cache.GridCacheMapEntry.markObsoleteVersion(GridCacheMapEntry.java:2799)
at
o.a.i.i.processors.cache.distributed.dht.topology.GridDhtLocalPartition.removeVersionedEntry(GridDhtLocalPartition.java:392)
at
[jira] [Created] (IGNITE-12593) Corruption of B+Tree caused by byte array values and TTL
Anton Kalashnikov created IGNITE-12593:
--
Summary: Corruption of B+Tree caused by byte array values and TTL
Key: IGNITE-12593
URL: https://issues.apache.org/jira/browse/IGNITE-12593
Project: Ignite
Issue Type: Bug
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov
It seems that the following set of parameters may lead to a corruption of
B+Tree:
- persistence is enabled
- TTL is enabled
- Expiry policy - AccessedExpiryPolicy 1 sec.
- cache value type is byte[]
- all caches belong to the same cache group
Example of the stack trace:
{code:java}
[2019-07-16
21:13:19,288][ERROR][sys-stripe-2-#46%db.IgnitePdsWithTtlDeactivateOnHighloadTest1%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1237460590,
val2=281586645860358]], msg=Runtime failure on search row: SearchRow
[key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], hash=378,
cacheId=-1806498247
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1237460590,
val2=281586645860358]], msg=Runtime failure on search row: SearchRow
[key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], hash=378,
cacheId=-1806498247]]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5910)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1859)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1662)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1645)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2410)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:445)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2309)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2570)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2030)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1848)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3235)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:139)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:273)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:268)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1141)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1558)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1186)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
[jira] [Created] (IGNITE-12592) Calcite integration. Query load balancing.
Igor Seliverstov created IGNITE-12592:
-
Summary: Calcite integration. Query load balancing.
Key: IGNITE-12592
URL: https://issues.apache.org/jira/browse/IGNITE-12592
Project: Ignite
Issue Type: Task
Reporter: Igor Seliverstov
Currently all query tasks execute in a query pool. The pool contains a limited
number of threads, so, in case of long running scan operation other query tasks
will starve.
We need to limit time a thread spends inside {{ScanNode#requestInternal
method}}.
See appropriate TODO
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12591) Scan query scans invalid partitions on unstable topology
Ivan Pavlukhin created IGNITE-12591:
---
Summary: Scan query scans invalid partitions on unstable topology
Key: IGNITE-12591
URL: https://issues.apache.org/jira/browse/IGNITE-12591
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.8
Reporter: Ivan Pavlukhin
Apparently scan query {{ScanQuery}} (without specifying explicit partition)
does not reserve partitions. Consequently when there is a partition movement
(unstable topology, e.g. new node joins the cluster) query can read from an
invalid (e.g. already destroyed) partition.
It is easy to catch "java.lang.IllegalStateException: Tree is being
concurrently destroyed". As it is a consequence of a race condition other
exceptions might be possible. See attached reproducer.
One possible solution here is to reserve partitions properly. Other approach is
to consider some optimistic strategy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Add user attributes to thin clients
Igor,
I see nothing criminal in attribute visibility to internal Ignite
components. It could be a hole if attributes were available through public
API.
вт, 28 янв. 2020 г., 13:07 Igor Sapego :
> Ilya,
>
> The fact that we have some security holes does not sound to me
> like a good reason to create a new one.
>
> Best Regards,
> Igor
>
>
> On Tue, Jan 28, 2020 at 12:22 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> wrote:
>
> > Hello!
> >
> > We already have several "security contexts" and we even have some code
> > which tries to clear sensitive data from it.
> >
> > However, it is no good for authenticating nodes anyway, since discovery
> > message have to travel via ring, so all server nodes will have access to
> > sensitive information.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 28 янв. 2020 г. в 11:41, Igor Sapego :
> >
> > > Dmitrii,
> > >
> > > I'm not a security expert, but I have a concern about the usage
> > > of common purpose mechanism for authentication purposes.
> > >
> > > I mean any other component seem to be able to get this data which
> > > should be private. Looks like a dangerous hack to me, to be honest.
> > >
> > > Maybe we should create and use some kind of "security context" for
> > > this purpose with limited access.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Mon, Jan 27, 2020 at 3:29 PM Dmitrii Ryabov
> > > wrote:
> > >
> > > > Ok, if no one have objections, I will restrict a map by strings only.
> > > >
> > > > чт, 23 янв. 2020 г., 17:20 Pavel Tupitsyn :
> > > >
> > > > > > Well, my understanding was a binary object with compact footer =
> > > false
> > > > > is totally standalone entity and can be read without any external
> > > > metadata
> > > > > Not exactly: type name and field names are unknown, you only have
> > > typeId,
> > > > > field ids and positions.
> > > > > There is one more Marshaller "mode": UNREGISTERED_TYPE_ID. In this
> > > mode,
> > > > > full type name is included in the binary object.
> > > > > However, field names are still missing and the class needs to be
> > > present
> > > > to
> > > > > deserialize.
> > > > > That's why arbitrary objects should not be allowed in the
> handshake:
> > in
> > > > > most cases they can't be decoded.
> > > > >
> > > > > On Thu, Jan 23, 2020 at 2:27 PM Dmitrii Ryabov <
> > somefire...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Alexei, yes, compactFooter is used only in 1 place.
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > BinaryWriterExImpl.marshal0() {
> > > > > >
> > > > > > BinaryClassDescriptor desc = ctx.descriptorForClass(cls);
> > > > > >
> > > > > > // descriptor transportation fails here
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > desc.write(obj, this); // compactFooter here
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn >:
> > > > > >
> > > > > > > > If we support only strings it will be necessary to encode
> > binary
> > > > > values
> > > > > > > to
> > > > > > > > something like BASE64 which is not sounds good from usability
> > > side
> > > > > > >
> > > > > > > There should be no need to put binary values to attributes.
> > What's
> > > > the
> > > > > > use
> > > > > > > case?
> > > > > > >
> > > > > > > On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> > > > > > > alexey.scherbak...@gmail.com> wrote:
> > > > > > >
> > > > > > > > > Footer is checked in postWrite - much later class
> descriptor
> > > > check.
> > > > > > > >
> > > > > > > > Well, my understanding was a binary object with compact
> footer
> > =
> > > > > false
> > > > > > is
> > > > > > > > totally standalone entity and can be read without any
> external
> > > > > > metadata.
> > > > > > > > Dmitrii Ryabov can you double check ?
> > > > > > > >
> > > > > > > > If we support only strings it will be necessary to encode
> > binary
> > > > > values
> > > > > > > to
> > > > > > > > something like BASE64 which is not sounds good from usability
> > > side.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev <
> > > nsamelc...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > +1 for the hardcoded String type only
> > > > > > > > >
> > > > > > > > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn <
> > > > ptupit...@apache.org
> > > > > >:
> > > > > > > > > >
> > > > > > > > > > - Cross-platform binary objects are totally possible, all
> > > those
> > > > > > thin
> > > > > > > > > > clients support them.
> > > > > > > > > > - User attributes can be useful, no objections here
> > > > > > > > > >
> > > > > > > > > > However, I don't think we should allow arbitrary objects
> in
> > > > user
> > > > > > > > > attributes.
> > > > > > > > > > Let's make them string only, much less to worry about.
> > > > > > > > > >
> > > > > > > > > > And using attributes for authentication still seems weird
> > and
> > > > > > dirty.
> > > > > > > > > >
> > > > > >
[jira] [Created] (IGNITE-12590) MERGE INTO query is failing on Ignite client node
Taras Ledkov created IGNITE-12590:
-
Summary: MERGE INTO query is failing on Ignite client node
Key: IGNITE-12590
URL: https://issues.apache.org/jira/browse/IGNITE-12590
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
Fix For: 2.9
Precondition: server nodes (any amount), webconsole is running.
1. Create the table as following:
CREATE TABLE USERPUBSTATICDATA (BOOK VARCHAR, DESK VARCHAR, TRADERS VARCHAR,
REGION VARCHAR, LOB VARCHAR, EXCLUDE VARCHAR, TRANSIT VARCHAR,
MAPBOOKTOTHISBOOK VARCHAR, CONSTRAINT USERPUBSTATICDATA_PK PRIMARY KEY
(BOOK,DESK)) WITH "template=replicated"
2. Execute merge into query:
MERGE INTO USERPUBSTATICDATA(BOOK, DESK, TRADERS, REGION, LOB, EXCLUDE,
TRANSIT, MAPBOOKTOTHISBOOK) VALUES('CADOIS', 'FRT TOR', 'Robin Das/Dave
Carlson', 'Toronto', 'FRT', null, null, 'CADOIS');
Step 2 is successfull on Web Console and called on IgniteCache from the server
node, but fails when called from the Ignite client node with exception:
{code}
javax.cache.CacheException: Invalid column name in KEYS clause of MERGE - it
may include only key and/or affinity columns: BOOK
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Ignite PMC member: Igor Sapego
Thanks guys, It's a pleasure to be a part of this great community Best Regards, Igor On Mon, Jan 27, 2020 at 9:26 PM Denis Magda wrote: > Igor, thanks for being an invaluable member of the community all this time! > Congrats! > > - > Denis > > > On Sun, Jan 26, 2020 at 9:58 PM Ivan Pavlukhin > wrote: > > > Hello Ignite Community, > > > > The Project Management Committee (PMC) for Apache Ignite has invited > > Igor Sapego to join the PMC as a new member and we are pleased to > > announce that he has accepted. > > > > Igor is an active Community member for a long time. He made a > > worthless contribution in Ignite C++ development and continues to do > > it today. Moreover he drives thin clients development for other > > programming languages. Additionally, always helpful on users mailing > > list. Wrote blog posts and spoke publicly about Ignite. > > > > Being a PMC member enables assistance with the management and to guide > > the direction of the project. > > > > Please join me in congratulating Igor! > > > > Best regards, > > Ivan Pavlukhin > > on behalf of Apache Ignite PMC > > >
Re: Add user attributes to thin clients
Ilya,
The fact that we have some security holes does not sound to me
like a good reason to create a new one.
Best Regards,
Igor
On Tue, Jan 28, 2020 at 12:22 PM Ilya Kasnacheev
wrote:
> Hello!
>
> We already have several "security contexts" and we even have some code
> which tries to clear sensitive data from it.
>
> However, it is no good for authenticating nodes anyway, since discovery
> message have to travel via ring, so all server nodes will have access to
> sensitive information.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 28 янв. 2020 г. в 11:41, Igor Sapego :
>
> > Dmitrii,
> >
> > I'm not a security expert, but I have a concern about the usage
> > of common purpose mechanism for authentication purposes.
> >
> > I mean any other component seem to be able to get this data which
> > should be private. Looks like a dangerous hack to me, to be honest.
> >
> > Maybe we should create and use some kind of "security context" for
> > this purpose with limited access.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Mon, Jan 27, 2020 at 3:29 PM Dmitrii Ryabov
> > wrote:
> >
> > > Ok, if no one have objections, I will restrict a map by strings only.
> > >
> > > чт, 23 янв. 2020 г., 17:20 Pavel Tupitsyn :
> > >
> > > > > Well, my understanding was a binary object with compact footer =
> > false
> > > > is totally standalone entity and can be read without any external
> > > metadata
> > > > Not exactly: type name and field names are unknown, you only have
> > typeId,
> > > > field ids and positions.
> > > > There is one more Marshaller "mode": UNREGISTERED_TYPE_ID. In this
> > mode,
> > > > full type name is included in the binary object.
> > > > However, field names are still missing and the class needs to be
> > present
> > > to
> > > > deserialize.
> > > > That's why arbitrary objects should not be allowed in the handshake:
> in
> > > > most cases they can't be decoded.
> > > >
> > > > On Thu, Jan 23, 2020 at 2:27 PM Dmitrii Ryabov <
> somefire...@gmail.com>
> > > > wrote:
> > > >
> > > > > Alexei, yes, compactFooter is used only in 1 place.
> > > > >
> > > > > ```
> > > > >
> > > > > BinaryWriterExImpl.marshal0() {
> > > > >
> > > > > BinaryClassDescriptor desc = ctx.descriptorForClass(cls);
> > > > >
> > > > > // descriptor transportation fails here
> > > > >
> > > > > ...
> > > > >
> > > > > desc.write(obj, this); // compactFooter here
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > > чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn :
> > > > >
> > > > > > > If we support only strings it will be necessary to encode
> binary
> > > > values
> > > > > > to
> > > > > > > something like BASE64 which is not sounds good from usability
> > side
> > > > > >
> > > > > > There should be no need to put binary values to attributes.
> What's
> > > the
> > > > > use
> > > > > > case?
> > > > > >
> > > > > > On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> > > > > > alexey.scherbak...@gmail.com> wrote:
> > > > > >
> > > > > > > > Footer is checked in postWrite - much later class descriptor
> > > check.
> > > > > > >
> > > > > > > Well, my understanding was a binary object with compact footer
> =
> > > > false
> > > > > is
> > > > > > > totally standalone entity and can be read without any external
> > > > > metadata.
> > > > > > > Dmitrii Ryabov can you double check ?
> > > > > > >
> > > > > > > If we support only strings it will be necessary to encode
> binary
> > > > values
> > > > > > to
> > > > > > > something like BASE64 which is not sounds good from usability
> > side.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev <
> > nsamelc...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > +1 for the hardcoded String type only
> > > > > > > >
> > > > > > > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > > >:
> > > > > > > > >
> > > > > > > > > - Cross-platform binary objects are totally possible, all
> > those
> > > > > thin
> > > > > > > > > clients support them.
> > > > > > > > > - User attributes can be useful, no objections here
> > > > > > > > >
> > > > > > > > > However, I don't think we should allow arbitrary objects in
> > > user
> > > > > > > > attributes.
> > > > > > > > > Let's make them string only, much less to worry about.
> > > > > > > > >
> > > > > > > > > And using attributes for authentication still seems weird
> and
> > > > > dirty.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov <
> > > > > > somefire...@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > Even if compact footer is disabled ?
> > > > > > > > > > Footer is checked in postWrite - much later class
> > descriptor
> > > > > check.
> > > > > > > > > >
> > > > > > > > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > > > > > > > alexey.scherbak...@gmail.com
> > > > > > > > > > >:
> > > > > > > > > >
Re: Add user attributes to thin clients
Hello!
We already have several "security contexts" and we even have some code
which tries to clear sensitive data from it.
However, it is no good for authenticating nodes anyway, since discovery
message have to travel via ring, so all server nodes will have access to
sensitive information.
Regards,
--
Ilya Kasnacheev
вт, 28 янв. 2020 г. в 11:41, Igor Sapego :
> Dmitrii,
>
> I'm not a security expert, but I have a concern about the usage
> of common purpose mechanism for authentication purposes.
>
> I mean any other component seem to be able to get this data which
> should be private. Looks like a dangerous hack to me, to be honest.
>
> Maybe we should create and use some kind of "security context" for
> this purpose with limited access.
>
> Best Regards,
> Igor
>
>
> On Mon, Jan 27, 2020 at 3:29 PM Dmitrii Ryabov
> wrote:
>
> > Ok, if no one have objections, I will restrict a map by strings only.
> >
> > чт, 23 янв. 2020 г., 17:20 Pavel Tupitsyn :
> >
> > > > Well, my understanding was a binary object with compact footer =
> false
> > > is totally standalone entity and can be read without any external
> > metadata
> > > Not exactly: type name and field names are unknown, you only have
> typeId,
> > > field ids and positions.
> > > There is one more Marshaller "mode": UNREGISTERED_TYPE_ID. In this
> mode,
> > > full type name is included in the binary object.
> > > However, field names are still missing and the class needs to be
> present
> > to
> > > deserialize.
> > > That's why arbitrary objects should not be allowed in the handshake: in
> > > most cases they can't be decoded.
> > >
> > > On Thu, Jan 23, 2020 at 2:27 PM Dmitrii Ryabov
> > > wrote:
> > >
> > > > Alexei, yes, compactFooter is used only in 1 place.
> > > >
> > > > ```
> > > >
> > > > BinaryWriterExImpl.marshal0() {
> > > >
> > > > BinaryClassDescriptor desc = ctx.descriptorForClass(cls);
> > > >
> > > > // descriptor transportation fails here
> > > >
> > > > ...
> > > >
> > > > desc.write(obj, this); // compactFooter here
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > > чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn :
> > > >
> > > > > > If we support only strings it will be necessary to encode binary
> > > values
> > > > > to
> > > > > > something like BASE64 which is not sounds good from usability
> side
> > > > >
> > > > > There should be no need to put binary values to attributes. What's
> > the
> > > > use
> > > > > case?
> > > > >
> > > > > On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> > > > > alexey.scherbak...@gmail.com> wrote:
> > > > >
> > > > > > > Footer is checked in postWrite - much later class descriptor
> > check.
> > > > > >
> > > > > > Well, my understanding was a binary object with compact footer =
> > > false
> > > > is
> > > > > > totally standalone entity and can be read without any external
> > > > metadata.
> > > > > > Dmitrii Ryabov can you double check ?
> > > > > >
> > > > > > If we support only strings it will be necessary to encode binary
> > > values
> > > > > to
> > > > > > something like BASE64 which is not sounds good from usability
> side.
> > > > > >
> > > > > >
> > > > > >
> > > > > > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev <
> nsamelc...@gmail.com
> > >:
> > > > > >
> > > > > > > +1 for the hardcoded String type only
> > > > > > >
> > > > > > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >:
> > > > > > > >
> > > > > > > > - Cross-platform binary objects are totally possible, all
> those
> > > > thin
> > > > > > > > clients support them.
> > > > > > > > - User attributes can be useful, no objections here
> > > > > > > >
> > > > > > > > However, I don't think we should allow arbitrary objects in
> > user
> > > > > > > attributes.
> > > > > > > > Let's make them string only, much less to worry about.
> > > > > > > >
> > > > > > > > And using attributes for authentication still seems weird and
> > > > dirty.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov <
> > > > > somefire...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > Even if compact footer is disabled ?
> > > > > > > > > Footer is checked in postWrite - much later class
> descriptor
> > > > check.
> > > > > > > > >
> > > > > > > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > > > > > > alexey.scherbak...@gmail.com
> > > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov <
> > > > > somefire...@gmail.com
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > > The protocol must be language-agnostic. If we add
> some
> > > > > features
> > > > > > > > > there,
> > > > > > > > > > > let's make sure they are usable from anywhere.
> > > > > > > > > > >
> > > > > > > > > > > That's why I want to allow primitives only. Any
> language
> > > can
> > > > > send
> > > > > > > > > numbers
> > > > > > > > > > > and strings.
> > > >
[jira] [Created] (IGNITE-12589) Remote thin client operations are not authorized correctly.
PetrovMikhail created IGNITE-12589: -- Summary: Remote thin client operations are not authorized correctly. Key: IGNITE-12589 URL: https://issues.apache.org/jira/browse/IGNITE-12589 Project: Ignite Issue Type: Bug Affects Versions: 2.7.6 Reporter: PetrovMikhail In the current Ignite security approach security subject id is considered to be a node id (see IgniteSecurityProcessor:107). In the case of thin clients, this approach doesn't work correctly. If some operation is executed on behalf of a thin client on a remote node (node that is different from one to which thin client connection was established), it's impossible in the same way as for a node obtain a thin client security subject information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Add user attributes to thin clients
Dmitrii,
I'm not a security expert, but I have a concern about the usage
of common purpose mechanism for authentication purposes.
I mean any other component seem to be able to get this data which
should be private. Looks like a dangerous hack to me, to be honest.
Maybe we should create and use some kind of "security context" for
this purpose with limited access.
Best Regards,
Igor
On Mon, Jan 27, 2020 at 3:29 PM Dmitrii Ryabov
wrote:
> Ok, if no one have objections, I will restrict a map by strings only.
>
> чт, 23 янв. 2020 г., 17:20 Pavel Tupitsyn :
>
> > > Well, my understanding was a binary object with compact footer = false
> > is totally standalone entity and can be read without any external
> metadata
> > Not exactly: type name and field names are unknown, you only have typeId,
> > field ids and positions.
> > There is one more Marshaller "mode": UNREGISTERED_TYPE_ID. In this mode,
> > full type name is included in the binary object.
> > However, field names are still missing and the class needs to be present
> to
> > deserialize.
> > That's why arbitrary objects should not be allowed in the handshake: in
> > most cases they can't be decoded.
> >
> > On Thu, Jan 23, 2020 at 2:27 PM Dmitrii Ryabov
> > wrote:
> >
> > > Alexei, yes, compactFooter is used only in 1 place.
> > >
> > > ```
> > >
> > > BinaryWriterExImpl.marshal0() {
> > >
> > > BinaryClassDescriptor desc = ctx.descriptorForClass(cls);
> > >
> > > // descriptor transportation fails here
> > >
> > > ...
> > >
> > > desc.write(obj, this); // compactFooter here
> > >
> > > }
> > >
> > > ```
> > >
> > > чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn :
> > >
> > > > > If we support only strings it will be necessary to encode binary
> > values
> > > > to
> > > > > something like BASE64 which is not sounds good from usability side
> > > >
> > > > There should be no need to put binary values to attributes. What's
> the
> > > use
> > > > case?
> > > >
> > > > On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> > > > alexey.scherbak...@gmail.com> wrote:
> > > >
> > > > > > Footer is checked in postWrite - much later class descriptor
> check.
> > > > >
> > > > > Well, my understanding was a binary object with compact footer =
> > false
> > > is
> > > > > totally standalone entity and can be read without any external
> > > metadata.
> > > > > Dmitrii Ryabov can you double check ?
> > > > >
> > > > > If we support only strings it will be necessary to encode binary
> > values
> > > > to
> > > > > something like BASE64 which is not sounds good from usability side.
> > > > >
> > > > >
> > > > >
> > > > > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev >:
> > > > >
> > > > > > +1 for the hardcoded String type only
> > > > > >
> > > > > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > > >
> > > > > > > - Cross-platform binary objects are totally possible, all those
> > > thin
> > > > > > > clients support them.
> > > > > > > - User attributes can be useful, no objections here
> > > > > > >
> > > > > > > However, I don't think we should allow arbitrary objects in
> user
> > > > > > attributes.
> > > > > > > Let's make them string only, much less to worry about.
> > > > > > >
> > > > > > > And using attributes for authentication still seems weird and
> > > dirty.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov <
> > > > somefire...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > Even if compact footer is disabled ?
> > > > > > > > Footer is checked in postWrite - much later class descriptor
> > > check.
> > > > > > > >
> > > > > > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > > > > > alexey.scherbak...@gmail.com
> > > > > > > > >:
> > > > > > > >
> > > > > > > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov <
> > > > somefire...@gmail.com
> > > > > >:
> > > > > > > > >
> > > > > > > > > > > The protocol must be language-agnostic. If we add some
> > > > features
> > > > > > > > there,
> > > > > > > > > > let's make sure they are usable from anywhere.
> > > > > > > > > >
> > > > > > > > > > That's why I want to allow primitives only. Any language
> > can
> > > > send
> > > > > > > > numbers
> > > > > > > > > > and strings.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > In general it's possible to have cross-platform complex
> data
> > > > > > structures,
> > > > > > > > > for example see protobuf.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Binary marshaller, before packing object to byte[], will
> > try
> > > to
> > > > > use
> > > > > > > > > > discovery processor and send message containing class
> > > > descriptor.
> > > > > > But
> > > > > > > > > thin
> > > > > > > > > > clients don't have discovery. Furthermore, if we write
> > binary
> > > > > > > > marshaller
> > > > > > > > > > without class descriptor synchronization, we can get
> > objects
> > > > with
[jira] [Created] (IGNITE-12588) Failed to run MLeap or Spark examples
Stepan Pilschikov created IGNITE-12588:
--
Summary: Failed to run MLeap or Spark examples
Key: IGNITE-12588
URL: https://issues.apache.org/jira/browse/IGNITE-12588
Project: Ignite
Issue Type: Bug
Components: ml, spark
Affects Versions: 2.8
Environment: Java 8
Ubuntu/Win 10
Reporter: Stepan Pilschikov
New release build comes with missed class
org.apache.ignite.ml.mleap.MLeapModelParser
Now i can't build any spark examples
Steps:
- Try to mvn clean package -P scala
Actual:
-
{code:java}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/apache-ignite-2.8.0-bin/examples/src/main/spark/org/apache/ignite/examples/ml/mleap/MLeapModelParserExample.java:[34,34]
package org.apache.ignite.ml.mleap does not exist
[ERROR]
/apache-ignite-2.8.0-bin/examples/src/main/spark/org/apache/ignite/examples/ml/mleap/MLeapModelParserExample.java:[45,26]
cannot find symbol
symbol: class MLeapModelParser
location: class org.apache.ignite.examples.ml.mleap.MLeapModelParserExample
[ERROR]
/apache-ignite-2.8.0-bin/examples/src/main/spark/org/apache/ignite/examples/ml/mleap/MLeapModelParserExample.java:[45,56]
cannot find symbol
symbol: class MLeapModelParser
location: class org.apache.ignite.examples.ml.mleap.MLeapModelParserExample
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12587) ML examples failed on start
Stepan Pilschikov created IGNITE-12587:
--
Summary: ML examples failed on start
Key: IGNITE-12587
URL: https://issues.apache.org/jira/browse/IGNITE-12587
Project: Ignite
Issue Type: Bug
Components: ml
Affects Versions: 2.8
Environment: Java 8
Linux/Win
Reporter: Stepan Pilschikov
New release build comes with lost data sets for ML 2.8
Steps:
- Try to run any ML examples used MLSandboxDatasets
(org.apache.ignite.examples.ml.environment.TrainingWithCustomPreprocessorsExample
for examples)
Actual:
- FileNotFoundException
{code}
Exception in thread "main" java.io.FileNotFoundException:
modules/ml/src/main/resources/datasets/boston_housing_dataset.txt
at
org.apache.ignite.ml.util.SandboxMLCache.fillCacheWith(SandboxMLCache.java:119)
at
org.apache.ignite.examples.ml.environment.TrainingWithCustomPreprocessorsExample.main(TrainingWithCustomPreprocessorsExample.java:62)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: AWS EBS Discovery: Contributor Wanted
Denis, Emmanouil, Sure, I'll take a look at the code shortly. -- Thank you, Sergey. On Mon, Jan 27, 2020 at 8:59 PM Denis Magda wrote: > I support the idea of triggering such tests on demand. We can create a wiki > page with instructions on how to run the tests. Unless there is a more > elegant solution. > > Sergey, would you be able to review Emmanouil's changes in the IP Finder > source code? > https://issues.apache.org/jira/browse/IGNITE-8617 > > - > Denis > > > On Sun, Jan 26, 2020 at 2:22 AM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > wrote: > > > Hi all! > > > > I do believe being able to execute some AWS integration tests on demand > > would be of value, especially for reviewers who cannot have an AWS stack > > created on demand. > > More than happy to help on that. > > > > Kind regards > > *Emmanouil Gkatziouras* > > https://egkatzioura.com/ | > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > https://github.com/gkatzioura > > > > > > On Fri, 24 Jan 2020 at 15:15, Sergey Chugunov > > > wrote: > > > > > Hello Emmanouil, > > > > > > It would be great if we have at least basic integration tests in real > AWS > > > environment. Even though they may require more work to keep them green > (I > > > mean here AWS quotas and additional configuration/reconfiguration > > efforts) > > > it worth it because these tests can also be useful as an examples. > > > > > > As the same time as IpFinder is such a basic component I don't think we > > > need to include them in constantly triggered suites like Run All but to > > > trigger them manually before/right after merging them to master branch > or > > > when developing something in related code. > > > > > > What do you think? > > > > > > -- > > > Thank you, > > > Sergey Chugunov. > > > > > > On Thu, Jan 23, 2020 at 5:32 PM Emmanouil Gkatziouras < > > > gkatzio...@gmail.com> > > > wrote: > > > > > > > Hi all! > > > > > > > > Yes It seems possible to get some free quota for integration tests on > > AWS > > > > [1] however most probably they are not gonna last forever. > > > > > > > > [1] > > > > > > > > > > > > > > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ > > > > > > > > King Regards > > > > *Emmanouil Gkatziouras* > > > > https://egkatzioura.com/ | > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > https://github.com/gkatzioura > > > > > > > > > > > > On Wed, 22 Jan 2020 at 16:48, Denis Magda wrote: > > > > > > > > > Hi Emmanouil, > > > > > > > > > > Thanks for preparing a pull-request for Application Load Balancer: > > > > > https://issues.apache.org/jira/browse/IGNITE-8617 > > > > > > > > > > Igniters, who is willing to step in as a primary reviewer? > > > > > > > > > > As for automated testing on AWS, are you aware of any sponsorship > > > program > > > > > of AWS for open source projects of our kind? It will be ideal to > have > > > > real > > > > > testing on AWS but someone needs to pay. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Sun, Jan 19, 2020 at 6:45 AM Emmanouil Gkatziouras < > > > > > gkatzio...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi all! > > > > > > > > > > > > I have spinned up an Application Load Balancer and an autoscaling > > > group > > > > > on > > > > > > AWS and the Ignite discovery using TcpDiscoveryAlbIpFinder works > as > > > > > > expected. > > > > > > > > > > > >- On startup nodes discover each other. > > > > > >- On ec2 node down, connection is lost and the cluster > > decreases. > > > > > >- On an extra node addition the cluster size increases > > > > > > > > > > > > This contribution is essential since the Previous ELB based > > discovery > > > > > uses > > > > > > the Classic Load Balancer which is still available however > > > > > > AWS advices users to use the Application one. [1] > > > > > > While my pull request gets reviewed I will also have a look at > > > > > > the IGNITE-12398 [2] issue which has to do with the S3 discovery. > > > > > > Another idea would also be to implement a `TCP Load Balancer > based` > > > > > > discovery. > > > > > > > > > > > > In order to test this issue and future ones I implemented some > > > > terraform > > > > > > scripts (which I shall use for other issues too) [3]. > > > > > > If some automated e2e testing on AWS is being considered they > might > > > be > > > > of > > > > > > value. > > > > > > I can help on implementing those tests by provisioning the > > > > infrastructure > > > > > > in an automated way and validate the discovery. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/migrate-to-application-load-balancer.html > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12398 > > > > > > [3] https://github.com/gkatzioura/ignite-aws-deploy > > > > > > > > > > > > Kind regards, > > > > > > *Emmanouil Gkatziouras* > > > > > > https://egkatzioura.com/ | > > >
