Re: BEAM-9045 Implement an Ignite runner using Apache Ignite compute grid
Hi Val,
Thank you for the feedback, yes instead of using checkpointing we can store
the intermediate snapshot of results directly in Ignite caches.
Also, the underlying support for exactly-once guarantee in the Ignite core
module will be great and we can use it for Ignite Compute.
Regards,
Saikat
On Wed, Aug 19, 2020 at 4:13 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> Hi Saikat,
>
> Makes sense. Note that the checkpointing API is a candidate for removal in
> Ignite 3.0 - it's better to store intermediate results directly in Ignite
> caches.
>
> Also, my feeling is that simple checkpointing might not be enough for the
> integration, especially if we want to pursue the exactly-once guarantee. I
> will create a separate thread on the Ignite dev list to discuss how we can
> add such support.
>
> -Val
>
> On Tue, Aug 18, 2020 at 6:46 PM Saikat Maitra
> wrote:
>
>> Hi Val,
>>
>> Thank you for your response. I like the idea of reactive event based
>> processing engine for fault tolerance. As you mentioned it will be upto
>> underlying system to manage job execution and offer fault tolerance and we
>> will need to build it in Ignite compute execution model.
>>
>> I looked into Flink and Samza runners and they both offer fault
>> tolerance using checkpointing mechanism.
>>
>> Flink - Fault-tolerance with *exactly-once* processing guarantees [1]
>> Samza - Fault-tolerance with support for incremental checkpointing of
>> state
>> instead of full snapshots. This enables Samza to scale to applications
>> with
>> very large state. [2]
>>
>> I will look into it further how we can implement checkpointing[3] for
>> Ignite compute job when running beam pipeline.
>>
>> [1] https://beam.apache.org/documentation/runners/flink/
>> [2] https://beam.apache.org/documentation/runners/samza/
>> [3] https://apacheignite.readme.io/docs/checkpointing
>>
>> Regards,
>> Saikat
>>
>>
>>
>>
>>
>> On Tue, Aug 18, 2020 at 7:29 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>> > Hi Saikat,
>> >
>> > Thanks for clarifying. Is there a Beam component that monitors the
>> state,
>> > or this is up to the application? If something fails, will the
>> application
>> > have to retry the whole pipeline?
>> >
>> > My concern is that Ignite compute actually provides very limited
>> > guarantees, especially for the async execution. There are some failover
>> > mechanisms, but overall it's up to the application to track the state
>> and
>> > retry. Moreover, if the application fails, all jobs it has submitted are
>> > canceled.
>> >
>> > I'm thinking that Ignite should have a reactive event-based processing
>> > engine. The basic idea is this:
>> > - an application submits an event into the cluster
>> > - the event is persisted in Ignite to be eventually processed
>> > - a processed event may result in some new events that are submitted in
>> > the similar fashion
>> >
>> > Ignite will provide the at-least-once guarantee (or even exactly-once
>> > under certain assumptions) for all the event handlers, so a user can
>> create
>> > a whole chain by submitting a single event, and they don't have to worry
>> > about failures - it's up to Ignite to handle them.
>> >
>> > It seems to me that it might be beneficial for the Beam runner to have
>> > such an engine under the hood. What do you think?
>> >
>> > -Val
>> >
>> > On Mon, Aug 17, 2020 at 6:00 PM Saikat Maitra
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> Luke - Thank you for sharing the details for the portability layer for
>> >> Flink, Samza and Spark. I will look into them and will reach out if I
>> have
>> >> any questions.
>> >>
>> >> Val - Thank you for your response, yes I am planning to run the beam
>> >> pipeline using Ignite compute engine in async run. Here is a sample
>> code
>> >> for the run method.
>> >>
>> >> IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
>> >> metricsAccumulator);
>> >> ComputeTaskFuture computeTaskFuture =
>> >> ignite.compute().withAsync().run(
>> >> (r, f) -> {
>> >> pipelineResult.freeze(f);
>> >> metricsAccumulator.destroy();
>> >> ignite.shutdown();
>> >> });
>> >> pipelineResult.setComputeFuture(asyncCompute.future());
>> >>
>> >> return pipelineResult;
>> >>
>> >>
>> >> My understanding is for failover scenarios we will need to map the job
>> >> state from Ignite known state to Beam Job state, an example like in
>> >> JetPipelineResult
>> >>
>> >>
>> https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90
>> >>
>> >> Regards,
>> >> Saikat
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
>> >> valentin.kuliche...@gmail.com> wrote:
>> >>
>> >> > Hi Saikat,
>> >> >
>> >> > This sounds very interesting - I've been thinking about how Ignite
>> >> compute
>> >> > engine could be
Re: Micronaut/Ignite
Hi Michael, Denis I was looking into tiering options for ehcache[1] and network options for Hazelcast[2] and I am thinking we can implement something similar to configure CommunicationSpi [1] https://micronaut-projects.github.io/micronaut-cache/snapshot/guide/#ehcache [2] https://micronaut-projects.github.io/micronaut-cache/snapshot/guide/#hazelcast Let me know what you think. Regards, Saikat On Wed, Aug 19, 2020 at 7:09 PM Michael Pollind wrote: > A lot of this was just figured out through experimentation. You can ask > questions in the micronaut gitter: > https://gitter.im/micronautfw/questions > . Micronaut documentation is pretty comprehensive: > https://docs.micronaut.io/latest/guide/index.html. look for EachProperty > and ConfigurationProperty. you can also hunt through the current existing > micronaut modules and find how those configuration items are setup. There > is also the unit test cases in micronaut-core which have been pretty > helpful in the past in working out how some of these annotations work in > practice. > > On Wed, Aug 19, 2020 at 4:50 PM Denis Magda wrote: > > > Michael, > > > > Alright, then the question on the possible quantity of Ignite instances > is > > settled - the integration will allow to auto-configure a single instance > > only. > > > > Give me a couple of days to look into the configuration matters of > > DefaultIgniteConfiguration and see what I can suggest. Could you > recommend > > any materials (or sources) that on Micronaut configuration specifies > > (through YAML and programmatically via source code)? > > > > Denis > > > > On Wednesday, August 19, 2020, Michael Pollind > wrote: > > > > > I don't think micronaut will be able to infer the communicationSpi, so > > you > > > need to define it separately as follows: > > > https://github.com/pollend/micronaut-ignite/blob/feature/ > > > rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/ > > > DefaultIgniteConfiguration.java#L40-L43. > > > With this setup the configuration should look pretty much like the > > > spring-boot sample you showed me: > > > https://apacheignite-mix.readme.io/docs/spring-boot# > > > set-ignite-up-via-spring-boot-configuration. > > > I agree it should make the configuration easier with just allowing a > > single > > > instance and it matches up well with spring-boot configuration: > > > https://docs.micronaut.io/latest/api/io/micronaut/ > > > context/annotation/Requires.html. > > > Since its mostly a niche usecase then having that as the default use > case > > > seems pretty ideal to me. the definition will work as follows: > > > > > > ignite: > > > enable true > > > ignite-instance-name: name > > > communication-spi: > > > local-port: > > > data-storage-configuration: > > > ... > > > cache-configurations: > > >- name: accounts > > > queryEntities: > > > - tableName: NAME > > >... > > >- ... > > > ignite-thin: > > > enable: false > > > instance-name: name > > > > > > > > > Micronaut has some mechanism to enforce the presence of something that > > > should suffice for this usecase: > > > https://docs.micronaut.io/latest/api/io/micronaut/ > > > context/annotation/Requires.html > > > > > > > > > On Wed, Aug 19, 2020 at 2:45 PM Denis Magda wrote: > > > > > > > Michael, > > > > > > > > > > > > > The current way I have it setup is the primary bean is used by > > default > > > so > > > > > you won't be able to use micronaut-cache with anything but the > > default > > > > > bean. I guess one can override the other if the configuration is > > > present. > > > > > > > > > > > > The more I'm thinking the more I'm convinced that we shouldn't bother > > > about > > > > the auto-configuration of several Ignite instances. As I said before, > > > > that's an occasional use case. Furthermore, Micronout is designed for > > > > micro-services and serverless functions and I can hardly think of a > use > > > > case when a micro-service or function would need to boot up several > > > Ignite > > > > clients. What if we let to auto-configure a single Ignite instance > per > > > > application process? What's your view on this? It will significantly > > > > simplify the design and implementation of integration. If anybody > needs > > > > several Ignite instances, then he can instantiate them manually. > > > > > > > > By default the > > > > > thick client instance will replace the thin-client DynamicCache if > > that > > > > > would be ok? > > > > > > > > > > > > If you agree on my proposal above, then I would simply disallow > > > > auto-starting more than one Ignite instance (let it be a thick or > thin > > > > client). For example, if a thick client is already started, then > throw > > an > > > > exception on an attempt to initialize a thin client (and vice versa). > > As > > > > for thick vs. thin client usage in relation to Micronaut, I would > > > recommend > > > > using the thin client if Micronaut is deployed in a serverless >
Re: Micronaut/Ignite
A lot of this was just figured out through experimentation. You can ask questions in the micronaut gitter: https://gitter.im/micronautfw/questions . Micronaut documentation is pretty comprehensive: https://docs.micronaut.io/latest/guide/index.html. look for EachProperty and ConfigurationProperty. you can also hunt through the current existing micronaut modules and find how those configuration items are setup. There is also the unit test cases in micronaut-core which have been pretty helpful in the past in working out how some of these annotations work in practice. On Wed, Aug 19, 2020 at 4:50 PM Denis Magda wrote: > Michael, > > Alright, then the question on the possible quantity of Ignite instances is > settled - the integration will allow to auto-configure a single instance > only. > > Give me a couple of days to look into the configuration matters of > DefaultIgniteConfiguration and see what I can suggest. Could you recommend > any materials (or sources) that on Micronaut configuration specifies > (through YAML and programmatically via source code)? > > Denis > > On Wednesday, August 19, 2020, Michael Pollind wrote: > > > I don't think micronaut will be able to infer the communicationSpi, so > you > > need to define it separately as follows: > > https://github.com/pollend/micronaut-ignite/blob/feature/ > > rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/ > > DefaultIgniteConfiguration.java#L40-L43. > > With this setup the configuration should look pretty much like the > > spring-boot sample you showed me: > > https://apacheignite-mix.readme.io/docs/spring-boot# > > set-ignite-up-via-spring-boot-configuration. > > I agree it should make the configuration easier with just allowing a > single > > instance and it matches up well with spring-boot configuration: > > https://docs.micronaut.io/latest/api/io/micronaut/ > > context/annotation/Requires.html. > > Since its mostly a niche usecase then having that as the default use case > > seems pretty ideal to me. the definition will work as follows: > > > > ignite: > > enable true > > ignite-instance-name: name > > communication-spi: > > local-port: > > data-storage-configuration: > > ... > > cache-configurations: > >- name: accounts > > queryEntities: > > - tableName: NAME > >... > >- ... > > ignite-thin: > > enable: false > > instance-name: name > > > > > > Micronaut has some mechanism to enforce the presence of something that > > should suffice for this usecase: > > https://docs.micronaut.io/latest/api/io/micronaut/ > > context/annotation/Requires.html > > > > > > On Wed, Aug 19, 2020 at 2:45 PM Denis Magda wrote: > > > > > Michael, > > > > > > > > > > The current way I have it setup is the primary bean is used by > default > > so > > > > you won't be able to use micronaut-cache with anything but the > default > > > > bean. I guess one can override the other if the configuration is > > present. > > > > > > > > > The more I'm thinking the more I'm convinced that we shouldn't bother > > about > > > the auto-configuration of several Ignite instances. As I said before, > > > that's an occasional use case. Furthermore, Micronout is designed for > > > micro-services and serverless functions and I can hardly think of a use > > > case when a micro-service or function would need to boot up several > > Ignite > > > clients. What if we let to auto-configure a single Ignite instance per > > > application process? What's your view on this? It will significantly > > > simplify the design and implementation of integration. If anybody needs > > > several Ignite instances, then he can instantiate them manually. > > > > > > By default the > > > > thick client instance will replace the thin-client DynamicCache if > that > > > > would be ok? > > > > > > > > > If you agree on my proposal above, then I would simply disallow > > > auto-starting more than one Ignite instance (let it be a thick or thin > > > client). For example, if a thick client is already started, then throw > an > > > exception on an attempt to initialize a thin client (and vice versa). > As > > > for thick vs. thin client usage in relation to Micronaut, I would > > recommend > > > using the thin client if Micronaut is deployed in a serverless function > > > (the thin client connects to the cluster faster), while for > > micro-services > > > you can use both types of clients. > > > > > > The main reason why I was using the spring bean definition was mainly > for > > > > convenience and I'm not sure what fields are the most relevant. > > > > > > > > > Ok, seems that I'm missing some important point about Micronaut. Let me > > > double-check the following with you. > > > > > > Assume these are the only fields of the DefaultIgniteConfiguration: > > > > > > private final String name; > > > > > > @ConfigurationBuilder() > > > private IgniteConfiguration igniteConfiguration = new > > > IgniteConfiguration(); > > > > > > > > > Will I be able
Re: Micronaut/Ignite
Michael,
Alright, then the question on the possible quantity of Ignite instances is
settled - the integration will allow to auto-configure a single instance
only.
Give me a couple of days to look into the configuration matters of
DefaultIgniteConfiguration and see what I can suggest. Could you recommend
any materials (or sources) that on Micronaut configuration specifies
(through YAML and programmatically via source code)?
Denis
On Wednesday, August 19, 2020, Michael Pollind wrote:
> I don't think micronaut will be able to infer the communicationSpi, so you
> need to define it separately as follows:
> https://github.com/pollend/micronaut-ignite/blob/feature/
> rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/
> DefaultIgniteConfiguration.java#L40-L43.
> With this setup the configuration should look pretty much like the
> spring-boot sample you showed me:
> https://apacheignite-mix.readme.io/docs/spring-boot#
> set-ignite-up-via-spring-boot-configuration.
> I agree it should make the configuration easier with just allowing a single
> instance and it matches up well with spring-boot configuration:
> https://docs.micronaut.io/latest/api/io/micronaut/
> context/annotation/Requires.html.
> Since its mostly a niche usecase then having that as the default use case
> seems pretty ideal to me. the definition will work as follows:
>
> ignite:
> enable true
> ignite-instance-name: name
> communication-spi:
> local-port:
> data-storage-configuration:
> ...
> cache-configurations:
>- name: accounts
> queryEntities:
> - tableName: NAME
>...
>- ...
> ignite-thin:
> enable: false
> instance-name: name
>
>
> Micronaut has some mechanism to enforce the presence of something that
> should suffice for this usecase:
> https://docs.micronaut.io/latest/api/io/micronaut/
> context/annotation/Requires.html
>
>
> On Wed, Aug 19, 2020 at 2:45 PM Denis Magda wrote:
>
> > Michael,
> >
> >
> > > The current way I have it setup is the primary bean is used by default
> so
> > > you won't be able to use micronaut-cache with anything but the default
> > > bean. I guess one can override the other if the configuration is
> present.
> >
> >
> > The more I'm thinking the more I'm convinced that we shouldn't bother
> about
> > the auto-configuration of several Ignite instances. As I said before,
> > that's an occasional use case. Furthermore, Micronout is designed for
> > micro-services and serverless functions and I can hardly think of a use
> > case when a micro-service or function would need to boot up several
> Ignite
> > clients. What if we let to auto-configure a single Ignite instance per
> > application process? What's your view on this? It will significantly
> > simplify the design and implementation of integration. If anybody needs
> > several Ignite instances, then he can instantiate them manually.
> >
> > By default the
> > > thick client instance will replace the thin-client DynamicCache if that
> > > would be ok?
> >
> >
> > If you agree on my proposal above, then I would simply disallow
> > auto-starting more than one Ignite instance (let it be a thick or thin
> > client). For example, if a thick client is already started, then throw an
> > exception on an attempt to initialize a thin client (and vice versa). As
> > for thick vs. thin client usage in relation to Micronaut, I would
> recommend
> > using the thin client if Micronaut is deployed in a serverless function
> > (the thin client connects to the cluster faster), while for
> micro-services
> > you can use both types of clients.
> >
> > The main reason why I was using the spring bean definition was mainly for
> > > convenience and I'm not sure what fields are the most relevant.
> >
> >
> > Ok, seems that I'm missing some important point about Micronaut. Let me
> > double-check the following with you.
> >
> > Assume these are the only fields of the DefaultIgniteConfiguration:
> >
> > private final String name;
> >
> > @ConfigurationBuilder()
> > private IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> >
> > Will I be able to set up the communicationSpi bean below without having
> it
> > as a field of the DefaultIgniteConfiguration? Are you getting a
> > NullPointerException?
> >
> > ignite:
> > name: some_name
> > igniteConfiguration:
> > communicationSpi:
> > {redefining some fields of the SPI}
> > -
> > Denis
> >
> >
> > On Wed, Aug 19, 2020 at 12:17 AM Michael Pollind
> > wrote:
> >
> > > Here is the initial setup that I quickly threw together along with some
> > > example test cases. I feel like this might get a little complicated
> but I
> > > think it's doable.
> > >
> > >
> > >
> > https://github.com/pollend/micronaut-ignite/blob/feature/
> rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/
> DefaultIgniteConfiguration.java
> > > along with some relevant test:
> > >
> >
[MTCGA]: new failures in builds [5551609] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New Critical Failure in master MVCC Cache 7 https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccCache7?branch=%3Cdefault%3E Changes may lead to failure were done by - nikolay https://ci.ignite.apache.org/viewModification.html?modId=905932 - kirill tkalenko https://ci.ignite.apache.org/viewModification.html?modId=905945 - ivan rakov https://ci.ignite.apache.org/viewModification.html?modId=905944 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 01:44:20 20-08-2020
Re: Micronaut/Ignite
I don't think micronaut will be able to infer the communicationSpi, so you
need to define it separately as follows:
https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/DefaultIgniteConfiguration.java#L40-L43.
With this setup the configuration should look pretty much like the
spring-boot sample you showed me:
https://apacheignite-mix.readme.io/docs/spring-boot#set-ignite-up-via-spring-boot-configuration.
I agree it should make the configuration easier with just allowing a single
instance and it matches up well with spring-boot configuration:
https://docs.micronaut.io/latest/api/io/micronaut/context/annotation/Requires.html.
Since its mostly a niche usecase then having that as the default use case
seems pretty ideal to me. the definition will work as follows:
ignite:
enable true
ignite-instance-name: name
communication-spi:
local-port:
data-storage-configuration:
...
cache-configurations:
- name: accounts
queryEntities:
- tableName: NAME
...
- ...
ignite-thin:
enable: false
instance-name: name
Micronaut has some mechanism to enforce the presence of something that
should suffice for this usecase:
https://docs.micronaut.io/latest/api/io/micronaut/context/annotation/Requires.html
On Wed, Aug 19, 2020 at 2:45 PM Denis Magda wrote:
> Michael,
>
>
> > The current way I have it setup is the primary bean is used by default so
> > you won't be able to use micronaut-cache with anything but the default
> > bean. I guess one can override the other if the configuration is present.
>
>
> The more I'm thinking the more I'm convinced that we shouldn't bother about
> the auto-configuration of several Ignite instances. As I said before,
> that's an occasional use case. Furthermore, Micronout is designed for
> micro-services and serverless functions and I can hardly think of a use
> case when a micro-service or function would need to boot up several Ignite
> clients. What if we let to auto-configure a single Ignite instance per
> application process? What's your view on this? It will significantly
> simplify the design and implementation of integration. If anybody needs
> several Ignite instances, then he can instantiate them manually.
>
> By default the
> > thick client instance will replace the thin-client DynamicCache if that
> > would be ok?
>
>
> If you agree on my proposal above, then I would simply disallow
> auto-starting more than one Ignite instance (let it be a thick or thin
> client). For example, if a thick client is already started, then throw an
> exception on an attempt to initialize a thin client (and vice versa). As
> for thick vs. thin client usage in relation to Micronaut, I would recommend
> using the thin client if Micronaut is deployed in a serverless function
> (the thin client connects to the cluster faster), while for micro-services
> you can use both types of clients.
>
> The main reason why I was using the spring bean definition was mainly for
> > convenience and I'm not sure what fields are the most relevant.
>
>
> Ok, seems that I'm missing some important point about Micronaut. Let me
> double-check the following with you.
>
> Assume these are the only fields of the DefaultIgniteConfiguration:
>
> private final String name;
>
> @ConfigurationBuilder()
> private IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
>
> Will I be able to set up the communicationSpi bean below without having it
> as a field of the DefaultIgniteConfiguration? Are you getting a
> NullPointerException?
>
> ignite:
> name: some_name
> igniteConfiguration:
> communicationSpi:
> {redefining some fields of the SPI}
> -
> Denis
>
>
> On Wed, Aug 19, 2020 at 12:17 AM Michael Pollind
> wrote:
>
> > Here is the initial setup that I quickly threw together along with some
> > example test cases. I feel like this might get a little complicated but I
> > think it's doable.
> >
> >
> >
> https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/DefaultIgniteConfiguration.java
> > along with some relevant test:
> >
> https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/test/groovy/io/micronaut/ignite/IgniteConfigurationSpec.groovy#L55-L73
> >
> > On Tue, Aug 18, 2020 at 11:49 PM Michael Pollind
> > wrote:
> >
> >>
> >>
> >> The main reason why I was using the spring bean definition was mainly
> for
> >> convenience and I'm not sure what fields are the most relevant. Will
> have
> >> to be kind of specific since the configuration might get a little
> >> complicated. The other thing you can do is use
> >>
> https://docs.micronaut.io/latest/api/io/micronaut/core/convert/format/MapFormat.html
> >> which will just map fields and values and you can pass that to somewhere
> >> else to be manage it.
> >>
> >> so you will need to do something like this as follows:
> >>
>
Re: Micronaut/Ignite
Michael,
> The current way I have it setup is the primary bean is used by default so
> you won't be able to use micronaut-cache with anything but the default
> bean. I guess one can override the other if the configuration is present.
The more I'm thinking the more I'm convinced that we shouldn't bother about
the auto-configuration of several Ignite instances. As I said before,
that's an occasional use case. Furthermore, Micronout is designed for
micro-services and serverless functions and I can hardly think of a use
case when a micro-service or function would need to boot up several Ignite
clients. What if we let to auto-configure a single Ignite instance per
application process? What's your view on this? It will significantly
simplify the design and implementation of integration. If anybody needs
several Ignite instances, then he can instantiate them manually.
By default the
> thick client instance will replace the thin-client DynamicCache if that
> would be ok?
If you agree on my proposal above, then I would simply disallow
auto-starting more than one Ignite instance (let it be a thick or thin
client). For example, if a thick client is already started, then throw an
exception on an attempt to initialize a thin client (and vice versa). As
for thick vs. thin client usage in relation to Micronaut, I would recommend
using the thin client if Micronaut is deployed in a serverless function
(the thin client connects to the cluster faster), while for micro-services
you can use both types of clients.
The main reason why I was using the spring bean definition was mainly for
> convenience and I'm not sure what fields are the most relevant.
Ok, seems that I'm missing some important point about Micronaut. Let me
double-check the following with you.
Assume these are the only fields of the DefaultIgniteConfiguration:
private final String name;
@ConfigurationBuilder()
private IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
Will I be able to set up the communicationSpi bean below without having it
as a field of the DefaultIgniteConfiguration? Are you getting a
NullPointerException?
ignite:
name: some_name
igniteConfiguration:
communicationSpi:
{redefining some fields of the SPI}
-
Denis
On Wed, Aug 19, 2020 at 12:17 AM Michael Pollind wrote:
> Here is the initial setup that I quickly threw together along with some
> example test cases. I feel like this might get a little complicated but I
> think it's doable.
>
>
> https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/DefaultIgniteConfiguration.java
> along with some relevant test:
> https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/test/groovy/io/micronaut/ignite/IgniteConfigurationSpec.groovy#L55-L73
>
> On Tue, Aug 18, 2020 at 11:49 PM Michael Pollind
> wrote:
>
>>
>>
>> The main reason why I was using the spring bean definition was mainly for
>> convenience and I'm not sure what fields are the most relevant. Will have
>> to be kind of specific since the configuration might get a little
>> complicated. The other thing you can do is use
>> https://docs.micronaut.io/latest/api/io/micronaut/core/convert/format/MapFormat.html
>> which will just map fields and values and you can pass that to somewhere
>> else to be manage it.
>>
>> so you will need to do something like this as follows:
>>
>> private final String name;
>> @ConfigurationBuilder()
>> private IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>> @ConfigurationBuilder(value = "communicationSpi")
>> private TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
>>
>> [image: image.png]
>>
>>
>> On Tue, Aug 18, 2020 at 11:05 PM Michael Pollind
>> wrote:
>>
>>> Its whatever is setup by default when the object is declared. I think we
>>> will have to define multiple ConfigurationBuilders If i'm not mistaken for
>>> the IgniteConfiguration. you don't need to provide the name since that is
>>> provided by the key for each configuration under EachProperty. The name is
>>> the qualified name that refers to that bean and also the same qualifier for
>>> the Ignite instance. For the most part will just use the primary bean for
>>> most part. I think you can only have one cache instance active at a time.
>>> The current way I have it setup is the primary bean is used by default so
>>> you won't be able to use micronaut-cache with anything but the default
>>> bean. I guess one can override the other if the configuration is present.
>>> One problem I see is micronaut-cache. We can only use one instance of
>>> DynamicCache but I need to verify how that works again. By default the
>>> thick client instance will replace the thin-client DynamicCache if that
>>> would be ok?
>>>
>>> ignite:
>>> thick-clients:
>>>default: <--- primary bean
>>> ...
>>>second-bean:
>>> ...
>>> thin-clients:
>>>default: <---
Re: BEAM-9045 Implement an Ignite runner using Apache Ignite compute grid
Hi Saikat,
Makes sense. Note that the checkpointing API is a candidate for removal in
Ignite 3.0 - it's better to store intermediate results directly in Ignite
caches.
Also, my feeling is that simple checkpointing might not be enough for the
integration, especially if we want to pursue the exactly-once guarantee. I
will create a separate thread on the Ignite dev list to discuss how we can
add such support.
-Val
On Tue, Aug 18, 2020 at 6:46 PM Saikat Maitra
wrote:
> Hi Val,
>
> Thank you for your response. I like the idea of reactive event based
> processing engine for fault tolerance. As you mentioned it will be upto
> underlying system to manage job execution and offer fault tolerance and we
> will need to build it in Ignite compute execution model.
>
> I looked into Flink and Samza runners and they both offer fault
> tolerance using checkpointing mechanism.
>
> Flink - Fault-tolerance with *exactly-once* processing guarantees [1]
> Samza - Fault-tolerance with support for incremental checkpointing of state
> instead of full snapshots. This enables Samza to scale to applications with
> very large state. [2]
>
> I will look into it further how we can implement checkpointing[3] for
> Ignite compute job when running beam pipeline.
>
> [1] https://beam.apache.org/documentation/runners/flink/
> [2] https://beam.apache.org/documentation/runners/samza/
> [3] https://apacheignite.readme.io/docs/checkpointing
>
> Regards,
> Saikat
>
>
>
>
>
> On Tue, Aug 18, 2020 at 7:29 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Saikat,
> >
> > Thanks for clarifying. Is there a Beam component that monitors the state,
> > or this is up to the application? If something fails, will the
> application
> > have to retry the whole pipeline?
> >
> > My concern is that Ignite compute actually provides very limited
> > guarantees, especially for the async execution. There are some failover
> > mechanisms, but overall it's up to the application to track the state and
> > retry. Moreover, if the application fails, all jobs it has submitted are
> > canceled.
> >
> > I'm thinking that Ignite should have a reactive event-based processing
> > engine. The basic idea is this:
> > - an application submits an event into the cluster
> > - the event is persisted in Ignite to be eventually processed
> > - a processed event may result in some new events that are submitted in
> > the similar fashion
> >
> > Ignite will provide the at-least-once guarantee (or even exactly-once
> > under certain assumptions) for all the event handlers, so a user can
> create
> > a whole chain by submitting a single event, and they don't have to worry
> > about failures - it's up to Ignite to handle them.
> >
> > It seems to me that it might be beneficial for the Beam runner to have
> > such an engine under the hood. What do you think?
> >
> > -Val
> >
> > On Mon, Aug 17, 2020 at 6:00 PM Saikat Maitra
> > wrote:
> >
> >> Hi,
> >>
> >> Luke - Thank you for sharing the details for the portability layer for
> >> Flink, Samza and Spark. I will look into them and will reach out if I
> have
> >> any questions.
> >>
> >> Val - Thank you for your response, yes I am planning to run the beam
> >> pipeline using Ignite compute engine in async run. Here is a sample code
> >> for the run method.
> >>
> >> IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
> >> metricsAccumulator);
> >> ComputeTaskFuture computeTaskFuture =
> >> ignite.compute().withAsync().run(
> >> (r, f) -> {
> >> pipelineResult.freeze(f);
> >> metricsAccumulator.destroy();
> >> ignite.shutdown();
> >> });
> >> pipelineResult.setComputeFuture(asyncCompute.future());
> >>
> >> return pipelineResult;
> >>
> >>
> >> My understanding is for failover scenarios we will need to map the job
> >> state from Ignite known state to Beam Job state, an example like in
> >> JetPipelineResult
> >>
> >>
> https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90
> >>
> >> Regards,
> >> Saikat
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >>
> >> > Hi Saikat,
> >> >
> >> > This sounds very interesting - I've been thinking about how Ignite
> >> compute
> >> > engine could be enhanced, and integration with Apache Beam is one of
> the
> >> > options I have in mind. Can you please describe how you plan to
> >> implement
> >> > this? Will it run on top of the Ignite Compute Grid? How are you going
> >> to
> >> > handle the failover, especially in the case of async pipeline
> execution?
> >> >
> >> > -Val
> >> >
> >> > On Sat, Aug 15, 2020 at 12:50 PM Saikat Maitra <
> saikat.mai...@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I have been working on implementing the Apache Ignite Runner to run
> >> >
Re: Editing rights
Yes, I confused JIRA with wiki in the last response. Ivan, thanks for catching that. - Denis On Wed, Aug 19, 2020 at 11:08 AM Ivan Pavlukhin wrote: > > We'd like to edit an article on cwiki. How can I get the editing rights > as a technical writer? > > I suppose we are talking about cwiki, not JIRA. Have you registered there? > https://cwiki.apache.org/confluence/signup.action > > 19.08.2020, Denis Magda написал(а): > > Evgeniy, > > > > Please tell your JIRA id/account_name. For some reason, I can't locate > your > > account with the email. > > > > - > > Denis > > > > > > On Wed, Aug 19, 2020 at 7:29 AM lonsdalel lonsdalel < > > eugenezabot...@gmail.com> wrote: > > > >> Hi Denis, > >> > >> my JIRA account is evzabot...@sberbank.ru. > >> > >> BR, > >> Evgeniy > >> > >> On 2020/08/17 18:31:33, Denis Magda wrote: > >> > Hi Evgeniy, > >> > > >> > Please tell the email address of your JIRA account and we'll grant you > >> all > >> > necessary permissions. > >> > > >> > - > >> > Denis > >> > > >> > > >> > On Mon, Aug 17, 2020 at 9:28 AM Заботкин Евгений Владимирович > >> > wrote: > >> > > >> > > Hello, > >> > > > >> > > We'd like to edit an article on cwiki. How can I get the editing > >> rights as > >> > > a technical writer? > >> > > > >> > > Best regards, > >> > > Evgeniy Zabotkin > >> > > > >> > > С уважением, > >> > > Евгений Заботкин< > >> > > http://addressbook.ca.sbrf.ru/home/Index?type=0=1818683> > >> > > ИТ-инженер (Технический писатель) > >> > > > >> > > > >> > > > >> > > > >> > > Даем людям уверенность и надежность! > >> > > > >> > > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые > >> > > документы, приложенные к нему, содержат конфиденциальную информацию. > >> > > Настоящим уведомляем Вас о том, что если это сообщение не > >> > > предназначено > >> > > Вам, использование, копирование, распространение информации, > >> содержащейся в > >> > > настоящем сообщении, а также осуществление любых действий на основе > >> этой > >> > > информации, строго запрещено. Если Вы получили это сообщение по > >> > > ошибке, > >> > > пожалуйста, сообщите об этом отправителю по электронной почте и > >> удалите это > >> > > сообщение. CONFIDENTIALITY NOTICE: This email and any files attached > >> to it > >> > > are confidential. If you are not the intended recipient you are > >> notified > >> > > that using, copying, distributing or taking any action in reliance > on > >> the > >> > > contents of this information is strictly prohibited. If you have > >> received > >> > > this email in error please notify the sender and delete this email. > >> > > > >> > > >> > > > > > -- > > Best regards, > Ivan Pavlukhin >
Re: Editing rights
> We'd like to edit an article on cwiki. How can I get the editing rights as a > technical writer? I suppose we are talking about cwiki, not JIRA. Have you registered there? https://cwiki.apache.org/confluence/signup.action 19.08.2020, Denis Magda написал(а): > Evgeniy, > > Please tell your JIRA id/account_name. For some reason, I can't locate your > account with the email. > > - > Denis > > > On Wed, Aug 19, 2020 at 7:29 AM lonsdalel lonsdalel < > eugenezabot...@gmail.com> wrote: > >> Hi Denis, >> >> my JIRA account is evzabot...@sberbank.ru. >> >> BR, >> Evgeniy >> >> On 2020/08/17 18:31:33, Denis Magda wrote: >> > Hi Evgeniy, >> > >> > Please tell the email address of your JIRA account and we'll grant you >> all >> > necessary permissions. >> > >> > - >> > Denis >> > >> > >> > On Mon, Aug 17, 2020 at 9:28 AM Заботкин Евгений Владимирович >> > wrote: >> > >> > > Hello, >> > > >> > > We'd like to edit an article on cwiki. How can I get the editing >> rights as >> > > a technical writer? >> > > >> > > Best regards, >> > > Evgeniy Zabotkin >> > > >> > > С уважением, >> > > Евгений Заботкин< >> > > http://addressbook.ca.sbrf.ru/home/Index?type=0=1818683> >> > > ИТ-инженер (Технический писатель) >> > > >> > > >> > > >> > > >> > > Даем людям уверенность и надежность! >> > > >> > > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые >> > > документы, приложенные к нему, содержат конфиденциальную информацию. >> > > Настоящим уведомляем Вас о том, что если это сообщение не >> > > предназначено >> > > Вам, использование, копирование, распространение информации, >> содержащейся в >> > > настоящем сообщении, а также осуществление любых действий на основе >> этой >> > > информации, строго запрещено. Если Вы получили это сообщение по >> > > ошибке, >> > > пожалуйста, сообщите об этом отправителю по электронной почте и >> удалите это >> > > сообщение. CONFIDENTIALITY NOTICE: This email and any files attached >> to it >> > > are confidential. If you are not the intended recipient you are >> notified >> > > that using, copying, distributing or taking any action in reliance on >> the >> > > contents of this information is strictly prohibited. If you have >> received >> > > this email in error please notify the sender and delete this email. >> > > >> > >> > -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [5552028] needs to be handled
Hi Taras, Thank you for the quick fix! I suppose it is fine to drop an email to such threads related to build failures in case one is working on a fix. So, nobody should worry. 2020-08-19 15:01 GMT+03:00, Taras Ledkov : > Hi, > > Thanks a lot. > I've already fixed it. > > On 19.08.2020 14:42, Ivan Pavlukhin wrote: >> A missing link for a previous message >> https://github.com/apache/ignite/pull/8167 >> >> 2020-08-19 14:42 GMT+03:00, Ivan Pavlukhin : >>> Travis gives a green light for a quick fix. >>> >>> 2020-08-19 14:08 GMT+03:00, Ivan Pavlukhin : Taras, Could you please take a look into this? It seems that checkstyle fails build due to a malicious extra space here https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558 2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more > than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you > were > a > volunteer to make the contribution to this project, but things change > and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue > and > fix test failures or step down and some committer may revert you > commit. > > *New Critical Failure in master ~Build Apache Ignite~ > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E > Changes may lead to failure were done by >- tledkov > https://ci.ignite.apache.org/viewModification.html?modId=905946 > >- Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >- Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 13:59:20 19-08-2020 > -- Best regards, Ivan Pavlukhin >>> >>> -- >>> >>> Best regards, >>> Ivan Pavlukhin >>> >> > -- > Taras Ledkov > Mail-To: tled...@gridgain.com > > -- Best regards, Ivan Pavlukhin
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Search does not seem to work. It uses mockups right now, but it should be ready when the docs are released. I can see that note blocks are just annotated with "Note." Can we have some image there? Do you have a preference as to which image you would like to see there? -Artem On 19.08.2020 17:37, Ilya Kasnacheev wrote: Hello! Search does not seem to work. Are we going to have a proper search results page? It is often the case that there's none. I can see that note blocks are just annotated with "Note." Can we have some image there? Example is http://64.227.57.229/docs/2.9.0/persistence/persistence-tuning Regards,
Re: Editing rights
Evgeniy, Please tell your JIRA id/account_name. For some reason, I can't locate your account with the email. - Denis On Wed, Aug 19, 2020 at 7:29 AM lonsdalel lonsdalel < eugenezabot...@gmail.com> wrote: > Hi Denis, > > my JIRA account is evzabot...@sberbank.ru. > > BR, > Evgeniy > > On 2020/08/17 18:31:33, Denis Magda wrote: > > Hi Evgeniy, > > > > Please tell the email address of your JIRA account and we'll grant you > all > > necessary permissions. > > > > - > > Denis > > > > > > On Mon, Aug 17, 2020 at 9:28 AM Заботкин Евгений Владимирович > > wrote: > > > > > Hello, > > > > > > We'd like to edit an article on cwiki. How can I get the editing > rights as > > > a technical writer? > > > > > > Best regards, > > > Evgeniy Zabotkin > > > > > > С уважением, > > > Евгений Заботкин< > > > http://addressbook.ca.sbrf.ru/home/Index?type=0=1818683> > > > ИТ-инженер (Технический писатель) > > > > > > > > > > > > > > > Даем людям уверенность и надежность! > > > > > > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые > > > документы, приложенные к нему, содержат конфиденциальную информацию. > > > Настоящим уведомляем Вас о том, что если это сообщение не предназначено > > > Вам, использование, копирование, распространение информации, > содержащейся в > > > настоящем сообщении, а также осуществление любых действий на основе > этой > > > информации, строго запрещено. Если Вы получили это сообщение по ошибке, > > > пожалуйста, сообщите об этом отправителю по электронной почте и > удалите это > > > сообщение. CONFIDENTIALITY NOTICE: This email and any files attached > to it > > > are confidential. If you are not the intended recipient you are > notified > > > that using, copying, distributing or taking any action in reliance on > the > > > contents of this information is strictly prohibited. If you have > received > > > this email in error please notify the sender and delete this email. > > > > > >
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Hello! Search does not seem to work. Are we going to have a proper search results page? It is often the case that there's none. I can see that note blocks are just annotated with "Note." Can we have some image there? Example is http://64.227.57.229/docs/2.9.0/persistence/persistence-tuning Regards, -- Ilya Kasnacheev ср, 19 авг. 2020 г. в 13:48, Artem Budnikov : > Hi everyone, > > We've set up a staging site for the Ignite 2.9 docs: > http://64.227.57.229/docs/2.9.0/installation/installing-using-docker > > user/pass: ignite/apache2020 > > More content is being added, but that's basically what it will look like. > > Your comments are welcome. > > -Artem > > On 18.08.2020 22:01, Pavel Tupitsyn wrote: > > Ok, I'm just asking, there is no rush for this ticket. > > If the release is postponed anyway, I thought that we can include more > > fixes. > > > > On Tue, Aug 18, 2020 at 8:10 PM Alex Plehanov > > wrote: > > > >> Pavel, > >> > >> We still can't find the root cause of performance drop. > >> > >> Ticket IGNITE-13369 still in progress. When it will be resolved? Is it > >> really critical bug? According to the user-list thread attached to the > >> ticket, there is a workaround exists for this problem and looks like > it's > >> not so critical. > >> > >> вт, 18 авг. 2020 г. в 12:30, Pavel Tupitsyn : > >> > >>> Alex, > >>> > >>> What's the status of the release? > >>> Can we include a bug fix there [1]? > >>> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-13369 > >>> > >>> On Fri, Aug 14, 2020 at 11:14 AM Alex Plehanov < > plehanov.a...@gmail.com> > >>> wrote: > >>> > Hello, > > > What is the release date for 2.9, from the cwiki it still says > >> August 7 > We have a performance drop on the new release and still trying to find > >>> the > problematic commit. I hope we will find it in a week + there are one > or > >>> two > weeks needed for voting and further release steps. > > пт, 14 авг. 2020 г. в 11:01, gaurav.ag...@gmail.com < > gaurav.ag...@gmail.com > > : > > What is the release date for 2.9, from the cwiki it still says August > >>> 7 : > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9 > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > >
Re: Editing rights
Hi Denis, my JIRA account is evzabot...@sberbank.ru. BR, Evgeniy On 2020/08/17 18:31:33, Denis Magda wrote: > Hi Evgeniy, > > Please tell the email address of your JIRA account and we'll grant you all > necessary permissions. > > - > Denis > > > On Mon, Aug 17, 2020 at 9:28 AM Заботкин Евгений Владимирович > wrote: > > > Hello, > > > > We'd like to edit an article on cwiki. How can I get the editing rights as > > a technical writer? > > > > Best regards, > > Evgeniy Zabotkin > > > > С уважением, > > Евгений Заботкин< > > http://addressbook.ca.sbrf.ru/home/Index?type=0=1818683> > > ИТ-инженер (Технический писатель) > > > > > > > > > > Даем людям уверенность и надежность! > > > > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые > > документы, приложенные к нему, содержат конфиденциальную информацию. > > Настоящим уведомляем Вас о том, что если это сообщение не предназначено > > Вам, использование, копирование, распространение информации, содержащейся в > > настоящем сообщении, а также осуществление любых действий на основе этой > > информации, строго запрещено. Если Вы получили это сообщение по ошибке, > > пожалуйста, сообщите об этом отправителю по электронной почте и удалите это > > сообщение. CONFIDENTIALITY NOTICE: This email and any files attached to it > > are confidential. If you are not the intended recipient you are notified > > that using, copying, distributing or taking any action in reliance on the > > contents of this information is strictly prohibited. If you have received > > this email in error please notify the sender and delete this email. > > >
[jira] [Created] (IGNITE-13376) Primary index can be created with fields sequence differ from declared.
Stanilovsky Evgeny created IGNITE-13376:
---
Summary: Primary index can be created with fields sequence differ
from declared.
Key: IGNITE-13376
URL: https://issues.apache.org/jira/browse/IGNITE-13376
Project: Ignite
Issue Type: Improvement
Components: sql
Affects Versions: 2.8.1, 2.7.6
Reporter: Stanilovsky Evgeny
Assignee: Stanilovsky Evgeny
Steps to reproduce:
Execute the following DDL (create table + create index):
{noformat}
CREATE TABLE IF NOT EXISTS Workspace (
id UUID NOT NULL,
accountId UUID NOT NULL,
jsonModel VARCHAR,
PRIMARY KEY (accountId, id)
) WITH
"template=partitioned,atomicity=transactional,key_type=org.gridgain.gmc.dto.workspace.WorkspaceKey,value_type=workspace.Workspace,cache_name=WorkspaceCache";
CREATE INDEX IF NOT EXISTS workspace_id_account_id_idx ON Workspace (id,
accountId);
{noformat}
On node start got the following warning:
{noformat}
Index with the given set or subset of columns already exists (consider dropping
either new or existing index) [cacheName=WorkspaceCache, schemaName=PUBLIC,
tableName=WORKSPACE, newIndexName=WORKSPACE_ID_ACCOUNT_ID_IDX,
existingIndexName=_key_PK, existingIndexColumns=[ID, ACCOUNTID]]
{noformat}
But PK and index have different order!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13375) Operations started on client nodes are not traced.
Alexander Lapin created IGNITE-13375: Summary: Operations started on client nodes are not traced. Key: IGNITE-13375 URL: https://issues.apache.org/jira/browse/IGNITE-13375 Project: Ignite Issue Type: Bug Reporter: Alexander Lapin Assignee: Alexander Lapin Fix For: 2.10 Root spans should be created on client node like it’s done on server node. I suppose that to already existent node join span creation we should add: * addMessage * node stop * custom event -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [MTCGA]: new failures in builds [5552028] needs to be handled
Hi, Thanks a lot. I've already fixed it. On 19.08.2020 14:42, Ivan Pavlukhin wrote: A missing link for a previous message https://github.com/apache/ignite/pull/8167 2020-08-19 14:42 GMT+03:00, Ivan Pavlukhin : Travis gives a green light for a quick fix. 2020-08-19 14:08 GMT+03:00, Ivan Pavlukhin : Taras, Could you please take a look into this? It seems that checkstyle fails build due to a malicious extra space here https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558 2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com : Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New Critical Failure in master ~Build Apache Ignite~ https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E Changes may lead to failure were done by - tledkov https://ci.ignite.apache.org/viewModification.html?modId=905946 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 13:59:20 19-08-2020 -- Best regards, Ivan Pavlukhin -- Best regards, Ivan Pavlukhin -- Taras Ledkov Mail-To: tled...@gridgain.com
Re: [MTCGA]: new failures in builds [5552028] needs to be handled
A missing link for a previous message https://github.com/apache/ignite/pull/8167 2020-08-19 14:42 GMT+03:00, Ivan Pavlukhin : > Travis gives a green light for a quick fix. > > 2020-08-19 14:08 GMT+03:00, Ivan Pavlukhin : >> Taras, >> >> Could you please take a look into this? It seems that checkstyle fails >> build due to a malicious extra space here >> https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558 >> >> 2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com >> : >>> Hi Igniters, >>> >>> I've detected some new issue on TeamCity to be handled. You are more >>> than >>> welcomed to help. >>> >>> If your changes can lead to this failure(s): We're grateful that you >>> were >>> a >>> volunteer to make the contribution to this project, but things change >>> and >>> you may no longer be able to finalize your contribution. >>> Could you respond to this email and indicate if you wish to continue >>> and >>> fix test failures or step down and some committer may revert you commit. >>> >>> *New Critical Failure in master ~Build Apache Ignite~ >>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E >>> Changes may lead to failure were done by >>> - tledkov >>> https://ci.ignite.apache.org/viewModification.html?modId=905946 >>> >>> - Here's a reminder of what contributors were agreed to do >>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >>> - Should you have any questions please contact dev@ignite.apache.org >>> >>> Best Regards, >>> Apache Ignite TeamCity Bot >>> https://github.com/apache/ignite-teamcity-bot >>> Notification generated at 13:59:20 19-08-2020 >>> >> >> >> -- >> >> Best regards, >> Ivan Pavlukhin >> > > > -- > > Best regards, > Ivan Pavlukhin > -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [5552028] needs to be handled
Travis gives a green light for a quick fix. 2020-08-19 14:08 GMT+03:00, Ivan Pavlukhin : > Taras, > > Could you please take a look into this? It seems that checkstyle fails > build due to a malicious extra space here > https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558 > > 2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com > : >> Hi Igniters, >> >> I've detected some new issue on TeamCity to be handled. You are more >> than >> welcomed to help. >> >> If your changes can lead to this failure(s): We're grateful that you were >> a >> volunteer to make the contribution to this project, but things change and >> you may no longer be able to finalize your contribution. >> Could you respond to this email and indicate if you wish to continue and >> fix test failures or step down and some committer may revert you commit. >> >> *New Critical Failure in master ~Build Apache Ignite~ >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E >> Changes may lead to failure were done by >> - tledkov >> https://ci.ignite.apache.org/viewModification.html?modId=905946 >> >> - Here's a reminder of what contributors were agreed to do >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >> - Should you have any questions please contact dev@ignite.apache.org >> >> Best Regards, >> Apache Ignite TeamCity Bot >> https://github.com/apache/ignite-teamcity-bot >> Notification generated at 13:59:20 19-08-2020 >> > > > -- > > Best regards, > Ivan Pavlukhin > -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [5552028] needs to be handled
Taras, Could you please take a look into this? It seems that checkstyle fails build due to a malicious extra space here https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558 2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and > fix test failures or step down and some committer may revert you commit. > > *New Critical Failure in master ~Build Apache Ignite~ > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E > Changes may lead to failure were done by >- tledkov > https://ci.ignite.apache.org/viewModification.html?modId=905946 > >- Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >- Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 13:59:20 19-08-2020 > -- Best regards, Ivan Pavlukhin
[MTCGA]: new failures in builds [5552028] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New Critical Failure in master ~Build Apache Ignite~ https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E Changes may lead to failure were done by - tledkov https://ci.ignite.apache.org/viewModification.html?modId=905946 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 13:59:20 19-08-2020
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Hi everyone, We've set up a staging site for the Ignite 2.9 docs: http://64.227.57.229/docs/2.9.0/installation/installing-using-docker user/pass: ignite/apache2020 More content is being added, but that's basically what it will look like. Your comments are welcome. -Artem On 18.08.2020 22:01, Pavel Tupitsyn wrote: Ok, I'm just asking, there is no rush for this ticket. If the release is postponed anyway, I thought that we can include more fixes. On Tue, Aug 18, 2020 at 8:10 PM Alex Plehanov wrote: Pavel, We still can't find the root cause of performance drop. Ticket IGNITE-13369 still in progress. When it will be resolved? Is it really critical bug? According to the user-list thread attached to the ticket, there is a workaround exists for this problem and looks like it's not so critical. вт, 18 авг. 2020 г. в 12:30, Pavel Tupitsyn : Alex, What's the status of the release? Can we include a bug fix there [1]? [1] https://issues.apache.org/jira/browse/IGNITE-13369 On Fri, Aug 14, 2020 at 11:14 AM Alex Plehanov wrote: Hello, What is the release date for 2.9, from the cwiki it still says August 7 We have a performance drop on the new release and still trying to find the problematic commit. I hope we will find it in a week + there are one or two weeks needed for voting and further release steps. пт, 14 авг. 2020 г. в 11:01, gaurav.ag...@gmail.com < gaurav.ag...@gmail.com : What is the release date for 2.9, from the cwiki it still says August 7 : https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9 -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[jira] [Created] (IGNITE-13374) Initial PME hangs because of multiple blinking nodes
Alexander Lapin created IGNITE-13374: Summary: Initial PME hangs because of multiple blinking nodes Key: IGNITE-13374 URL: https://issues.apache.org/jira/browse/IGNITE-13374 Project: Ignite Issue Type: Bug Reporter: Alexander Lapin Assignee: Alexander Lapin Fix For: 2.10 *Root cause* of the issue is a race inside GridDhtPartitionsExchangeFuture on client side between two processes: # When old coordinator fails and the new one takes over it sends GridDhtPartitionsSingleRequest messages to all nodes including clients to restore exchange results. Processing this message on client includes updating current coordinator reference (crd field). # When future receives discovery notification about old coordinator failure it should detect change of coordinator and send GridDhtPartitionsSingleMessage to new coordinator to obtain affinity. But updated crd field prevents client from detecting coordinator failure and sending SingleMessage to new coordinator which in turn leads to hanging client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Update of the default inline size for variable types
Hi, In my mind, the inline size 64 will be able to significant grow of storage size. It can be difficult to understand by users. Earlier I remember we panned to replace inline value to hash code in the case where size of value more than inline size. It will help to comparison of "==", "!=", but will not grow size of storage. I think optimization with hash code looks more preferable and in last way anyone can to grow size of baseline though API. On Wed, Aug 19, 2020 at 9:22 AM Zhenya Stanilovsky wrote: > > > >Hi guys, > > Evgeniy, hola! > > > >Currently if a varlength type (such as String or byte[]) is encountered in > >the composite index inline size just defaults to 10, which is almost > always > >not enough. I am going to change this and implement following changes: > > > >1) For a column of the variable length keep using 10 as the default size > in > >case of the one-column index. But if the index is composite the default > >index size will be calculated as the sum of sizes of all indexed columns. > >For example, for the index like (INT, VARCHAR, VARCHAR, INT) default > inline > >size will be 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). > > Why exactly this approach ? Why not 5 + 10 and its all here ? Do you have > some logical base, statistical distribution or something near it, for now > this look as your own decision and nothing more, i`m wrong ? > > > >2) For sql varchar and binary columns with defined length (for example > >VARCHAR(XX)) use XX + 3 as default inline size for the column (need 3 > extra > >bytes for the inner representation of the type). > > The same question here, why you want o cover all varchar len ? do you > compare with other vendors approach ? > > > >3) Maximum default index size still will be limited by > >IGNITE_MAX_INDEX_PAYLOAD_SIZE, but its default value will be increased to > >64. For example for the index (VARCHAR, VARCHAR, VARCHAR, VARCHAR, > VARCHAR, > >VARCHAR, VARCHAR) default index size will be only 64. Same for the columns > >with defined length: by default VARCHAR(100) column will create index only > >with size equal to 64. > > > >Please tell if you have any concerns. Update can be found at > >https://github.com/apache/ignite/pull/8161 > > > >Best regards, > >Evgeniy > > > > > > -- Vladislav Pyatkov
Re: Micronaut/Ignite
Here is the initial setup that I quickly threw together along with some example test cases. I feel like this might get a little complicated but I think it's doable. https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/main/java/io/micronaut/ignite/configuration/DefaultIgniteConfiguration.java along with some relevant test: https://github.com/pollend/micronaut-ignite/blob/feature/rework-1/ignite-core/src/test/groovy/io/micronaut/ignite/IgniteConfigurationSpec.groovy#L55-L73 On Tue, Aug 18, 2020 at 11:49 PM Michael Pollind wrote: > > > The main reason why I was using the spring bean definition was mainly for > convenience and I'm not sure what fields are the most relevant. Will have > to be kind of specific since the configuration might get a little > complicated. The other thing you can do is use > https://docs.micronaut.io/latest/api/io/micronaut/core/convert/format/MapFormat.html > which will just map fields and values and you can pass that to somewhere > else to be manage it. > > so you will need to do something like this as follows: > > private final String name; > @ConfigurationBuilder() > private IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); > @ConfigurationBuilder(value = "communicationSpi") > private TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi(); > > [image: image.png] > > > On Tue, Aug 18, 2020 at 11:05 PM Michael Pollind > wrote: > >> Its whatever is setup by default when the object is declared. I think we >> will have to define multiple ConfigurationBuilders If i'm not mistaken for >> the IgniteConfiguration. you don't need to provide the name since that is >> provided by the key for each configuration under EachProperty. The name is >> the qualified name that refers to that bean and also the same qualifier for >> the Ignite instance. For the most part will just use the primary bean for >> most part. I think you can only have one cache instance active at a time. >> The current way I have it setup is the primary bean is used by default so >> you won't be able to use micronaut-cache with anything but the default >> bean. I guess one can override the other if the configuration is present. >> One problem I see is micronaut-cache. We can only use one instance of >> DynamicCache but I need to verify how that works again. By default the >> thick client instance will replace the thin-client DynamicCache if that >> would be ok? >> >> ignite: >> thick-clients: >>default: <--- primary bean >> ... >>second-bean: >> ... >> thin-clients: >>default: <--- primary bean >> ... >>second-bean: >> >> >> >> >> https://docs.micronaut.io/latest/api/io/micronaut/context/annotation/Requires.html >> >> On Tue, Aug 18, 2020 at 10:13 PM Denis Magda wrote: >> >>> > >>> > oh, so we probably don't need to work with multiple instances. This is >>> what >>> > I have in the current master branch. >>> >>> >>> In most cases, people start a single instance of a thick or thin client >>> per >>> application. The clients are multi-threaded and can utilize all the CPUs >>> effectively. However, it's not harmful to have the ability to configure >>> several clients per application. As far as I understand, Micronaut >>> distinguishes clients per the "IgniteClientConfiguration.name" property, >>> right? >>> >>> So what defaults are set for IgniteConfiguration? >>> >>> >>> Does it matter to Micronaut what those defaults are? By looking at the >>> IgniteThinClientConfiguration >>> < >>> https://micronaut-projects.github.io/micronaut-ignite/snapshot/api/io/micronaut/ignite/configuration/IgniteThinClientConfiguration.html >>> >, >>> that defines org.apache.ignite.configuration.ClientConfiguration property >>> (under the name of "configuration"), I see that Micronaut could >>> introspect >>> all the fields of the ClientConfiguration and prepared these properties >>> table >>> < >>> https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/#io.micronaut.ignite.configuration.IgniteThinClientConfiguration >>> >. >>> For me, it means that whenever I am configuring the thin client in a YAML >>> file, Micronaut will create an instance of the ClientConfiguration >>> (Ignite >>> sets the defaults), and then I can override some settings such as >>> "addresses" or "enablePartitionAwareness". Does this sound accurate >>> concerning Micronaut? >>> >>> Jumping back to the IgniteConfiguration, I would just swap the "path" >>> that >>> is the String with the "config" that is IgniteConfiguration. Then let >>> Ignite take care of the IgniteConfiguration defaults and allow a >>> developer >>> to override some defaults (such as discoverySPI.ipFinder or memory >>> settings). Just in case, you can find IgniteConfiguration defaults here >>> < >>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/IgniteConfiguration.java >>> > >>> . >>> >>> - >>> Denis >>> >>> >>> On Tue, Aug 18, 2020 at
Re: Micronaut/Ignite
The main reason why I was using the spring bean definition was mainly for convenience and I'm not sure what fields are the most relevant. Will have to be kind of specific since the configuration might get a little complicated. The other thing you can do is use https://docs.micronaut.io/latest/api/io/micronaut/core/convert/format/MapFormat.html which will just map fields and values and you can pass that to somewhere else to be manage it. so you will need to do something like this as follows: private final String name; @ConfigurationBuilder() private IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); @ConfigurationBuilder(value = "communicationSpi") private TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi(); [image: image.png] On Tue, Aug 18, 2020 at 11:05 PM Michael Pollind wrote: > Its whatever is setup by default when the object is declared. I think we > will have to define multiple ConfigurationBuilders If i'm not mistaken for > the IgniteConfiguration. you don't need to provide the name since that is > provided by the key for each configuration under EachProperty. The name is > the qualified name that refers to that bean and also the same qualifier for > the Ignite instance. For the most part will just use the primary bean for > most part. I think you can only have one cache instance active at a time. > The current way I have it setup is the primary bean is used by default so > you won't be able to use micronaut-cache with anything but the default > bean. I guess one can override the other if the configuration is present. > One problem I see is micronaut-cache. We can only use one instance of > DynamicCache but I need to verify how that works again. By default the > thick client instance will replace the thin-client DynamicCache if that > would be ok? > > ignite: > thick-clients: >default: <--- primary bean > ... >second-bean: > ... > thin-clients: >default: <--- primary bean > ... >second-bean: > > > > > https://docs.micronaut.io/latest/api/io/micronaut/context/annotation/Requires.html > > On Tue, Aug 18, 2020 at 10:13 PM Denis Magda wrote: > >> > >> > oh, so we probably don't need to work with multiple instances. This is >> what >> > I have in the current master branch. >> >> >> In most cases, people start a single instance of a thick or thin client >> per >> application. The clients are multi-threaded and can utilize all the CPUs >> effectively. However, it's not harmful to have the ability to configure >> several clients per application. As far as I understand, Micronaut >> distinguishes clients per the "IgniteClientConfiguration.name" property, >> right? >> >> So what defaults are set for IgniteConfiguration? >> >> >> Does it matter to Micronaut what those defaults are? By looking at the >> IgniteThinClientConfiguration >> < >> https://micronaut-projects.github.io/micronaut-ignite/snapshot/api/io/micronaut/ignite/configuration/IgniteThinClientConfiguration.html >> >, >> that defines org.apache.ignite.configuration.ClientConfiguration property >> (under the name of "configuration"), I see that Micronaut could introspect >> all the fields of the ClientConfiguration and prepared these properties >> table >> < >> https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/#io.micronaut.ignite.configuration.IgniteThinClientConfiguration >> >. >> For me, it means that whenever I am configuring the thin client in a YAML >> file, Micronaut will create an instance of the ClientConfiguration (Ignite >> sets the defaults), and then I can override some settings such as >> "addresses" or "enablePartitionAwareness". Does this sound accurate >> concerning Micronaut? >> >> Jumping back to the IgniteConfiguration, I would just swap the "path" that >> is the String with the "config" that is IgniteConfiguration. Then let >> Ignite take care of the IgniteConfiguration defaults and allow a developer >> to override some defaults (such as discoverySPI.ipFinder or memory >> settings). Just in case, you can find IgniteConfiguration defaults here >> < >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/IgniteConfiguration.java >> > >> . >> >> - >> Denis >> >> >> On Tue, Aug 18, 2020 at 8:59 PM Michael Pollind >> wrote: >> >> > oh, so we probably don't need to work with multiple instances. This is >> what >> > I have in the current master branch. I believe I was originally trying >> to >> > set-up the configuration with the default ignite instance but found I >> > couldn't cover enough of the configuration. So what defaults are set for >> > IgniteConfiguration? some of those factory items can't be covered with >> how >> > micronaut sets up configurations. @ConfigurationProperty can only be >> > defined on a known factory, there are ways to have multiple factories >> and >> > label them optional but that easily gets overwhelming. In this >> situation >> > providing your own bean
[jira] [Created] (IGNITE-13373) WAL segmentns do not released on releaseHistoryForPreloading()
Alexander Lapin created IGNITE-13373: Summary: WAL segmentns do not released on releaseHistoryForPreloading() Key: IGNITE-13373 URL: https://issues.apache.org/jira/browse/IGNITE-13373 Project: Ignite Issue Type: Bug Reporter: Alexander Lapin Assignee: Alexander Lapin * Reserve/releaseHistoryForPreloading() was reworked, now we store oldest WALPointer that was reserved during reserveHistoryForPreloading in reservedForPreloading field. As a result it's possible to release wall reservation on releaseHIstoryForPreloading(). * searchAndReserveCheckpoints() was slightly refactored: now it returns not only an earliestValidCheckpoints but also an oldest reservedCheckpoint, so that there’s no need to recalculate it within reserveHistoryForExchange(). * -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Update of the default inline size for variable types
>Hi guys, Evgeniy, hola! > >Currently if a varlength type (such as String or byte[]) is encountered in >the composite index inline size just defaults to 10, which is almost always >not enough. I am going to change this and implement following changes: > >1) For a column of the variable length keep using 10 as the default size in >case of the one-column index. But if the index is composite the default >index size will be calculated as the sum of sizes of all indexed columns. >For example, for the index like (INT, VARCHAR, VARCHAR, INT) default inline >size will be 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). Why exactly this approach ? Why not 5 + 10 and its all here ? Do you have some logical base, statistical distribution or something near it, for now this look as your own decision and nothing more, i`m wrong ? > >2) For sql varchar and binary columns with defined length (for example >VARCHAR(XX)) use XX + 3 as default inline size for the column (need 3 extra >bytes for the inner representation of the type). The same question here, why you want o cover all varchar len ? do you compare with other vendors approach ? > >3) Maximum default index size still will be limited by >IGNITE_MAX_INDEX_PAYLOAD_SIZE, but its default value will be increased to >64. For example for the index (VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, >VARCHAR, VARCHAR) default index size will be only 64. Same for the columns >with defined length: by default VARCHAR(100) column will create index only >with size equal to 64. > >Please tell if you have any concerns. Update can be found at >https://github.com/apache/ignite/pull/8161 > >Best regards, >Evgeniy >
Update of the default inline size for variable types
Hi guys, Currently if a varlength type (such as String or byte[]) is encountered in the composite index inline size just defaults to 10, which is almost always not enough. I am going to change this and implement following changes: 1) For a column of the variable length keep using 10 as the default size in case of the one-column index. But if the index is composite the default index size will be calculated as the sum of sizes of all indexed columns. For example, for the index like (INT, VARCHAR, VARCHAR, INT) default inline size will be 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). 2) For sql varchar and binary columns with defined length (for example VARCHAR(XX)) use XX + 3 as default inline size for the column (need 3 extra bytes for the inner representation of the type). 3) Maximum default index size still will be limited by IGNITE_MAX_INDEX_PAYLOAD_SIZE, but its default value will be increased to 64. For example for the index (VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR) default index size will be only 64. Same for the columns with defined length: by default VARCHAR(100) column will create index only with size equal to 64. Please tell if you have any concerns. Update can be found at https://github.com/apache/ignite/pull/8161 Best regards, Evgeniy
Re: Micronaut/Ignite
Its whatever is setup by default when the object is declared. I think we will have to define multiple ConfigurationBuilders If i'm not mistaken for the IgniteConfiguration. you don't need to provide the name since that is provided by the key for each configuration under EachProperty. The name is the qualified name that refers to that bean and also the same qualifier for the Ignite instance. For the most part will just use the primary bean for most part. I think you can only have one cache instance active at a time. The current way I have it setup is the primary bean is used by default so you won't be able to use micronaut-cache with anything but the default bean. I guess one can override the other if the configuration is present. One problem I see is micronaut-cache. We can only use one instance of DynamicCache but I need to verify how that works again. By default the thick client instance will replace the thin-client DynamicCache if that would be ok? ignite: thick-clients: default: <--- primary bean ... second-bean: ... thin-clients: default: <--- primary bean ... second-bean: https://docs.micronaut.io/latest/api/io/micronaut/context/annotation/Requires.html On Tue, Aug 18, 2020 at 10:13 PM Denis Magda wrote: > > > > oh, so we probably don't need to work with multiple instances. This is > what > > I have in the current master branch. > > > In most cases, people start a single instance of a thick or thin client per > application. The clients are multi-threaded and can utilize all the CPUs > effectively. However, it's not harmful to have the ability to configure > several clients per application. As far as I understand, Micronaut > distinguishes clients per the "IgniteClientConfiguration.name" property, > right? > > So what defaults are set for IgniteConfiguration? > > > Does it matter to Micronaut what those defaults are? By looking at the > IgniteThinClientConfiguration > < > https://micronaut-projects.github.io/micronaut-ignite/snapshot/api/io/micronaut/ignite/configuration/IgniteThinClientConfiguration.html > >, > that defines org.apache.ignite.configuration.ClientConfiguration property > (under the name of "configuration"), I see that Micronaut could introspect > all the fields of the ClientConfiguration and prepared these properties > table > < > https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/#io.micronaut.ignite.configuration.IgniteThinClientConfiguration > >. > For me, it means that whenever I am configuring the thin client in a YAML > file, Micronaut will create an instance of the ClientConfiguration (Ignite > sets the defaults), and then I can override some settings such as > "addresses" or "enablePartitionAwareness". Does this sound accurate > concerning Micronaut? > > Jumping back to the IgniteConfiguration, I would just swap the "path" that > is the String with the "config" that is IgniteConfiguration. Then let > Ignite take care of the IgniteConfiguration defaults and allow a developer > to override some defaults (such as discoverySPI.ipFinder or memory > settings). Just in case, you can find IgniteConfiguration defaults here > < > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/IgniteConfiguration.java > > > . > > - > Denis > > > On Tue, Aug 18, 2020 at 8:59 PM Michael Pollind > wrote: > > > oh, so we probably don't need to work with multiple instances. This is > what > > I have in the current master branch. I believe I was originally trying to > > set-up the configuration with the default ignite instance but found I > > couldn't cover enough of the configuration. So what defaults are set for > > IgniteConfiguration? some of those factory items can't be covered with > how > > micronaut sets up configurations. @ConfigurationProperty can only be > > defined on a known factory, there are ways to have multiple factories and > > label them optional but that easily gets overwhelming. In this situation > > providing your own bean would probably be more ideal in this situation > when > > I think about it. I was worrying that I wouldn't be able to cover enough > > of the configuration with > > > > ignite: enabled: true thin-clients:default: address: - > > "127.0.0.1:10800" - "127.0.0.1:10801" > > > > thin-client-2: > > address: - "127.0.0.1:10800" - "127.0.0.1:10801" > > > > > > you can see it in the current snapshot documentation: > > https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/ > > > > On Tue, Aug 18, 2020 at 4:16 PM Denis Magda wrote: > > > > > Michael, thanks for filling out the wiki page. > > > > > > I'm looking at the Auto-Configuration wiki section and the current > > version > > > of the io.micronaut.ignite.configuration.IgniteClientConfiguration > > > < > > > > > >
