Re: Azure Cloud IP Finder
Thank you for the review. I have updated the PR. Please see. On Mon, Mar 29, 2021 at 8:27 PM Ilya Kasnacheev wrote: > > Hello! > > I have left some more comments after trying this change on a real Azure > cluster. > > Regards, > -- > Ilya Kasnacheev > > > вт, 23 мар. 2021 г. в 18:32, Atri Sharma : > > > Thank you! > > > > I have updated the PR. Please see and let me know. > > > > On Tue, Mar 23, 2021 at 4:27 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > I am going to check this change out when I have time, using my Azure > > > account. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > вт, 23 мар. 2021 г. в 07:20, Atri Sharma : > > > > > > > Gentle reminder on this -- please help in reviewing this. > > > > > > > > On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma wrote: > > > > > > > > > > Thanks Denis. > > > > > > > > > > I have raised a PR for the same: > > > > > > > > > > https://github.com/apache/ignite/pull/8897 > > > > > > > > > > Regards, > > > > > > > > > > Atri > > > > > > > > > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda > > wrote: > > > > > > > > > > > > Atri, > > > > > > > > > > > > Let's discuss the subj together with the community. Ignite already > > > > supports > > > > > > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is > > > > still > > > > > > missing. I can confirm that the demand exists, and rather > > frequently, > > > > I see > > > > > > developers asking for an Azure-native IP finder for Ignite. > > > > > > > > > > > > Atri, could you please research how to implement the IP finder and > > > > suggest > > > > > > a solution in this discussion thread? See how it was done for AWS > > and > > > > GCE, > > > > > > we might go the same route or use a more contemporary and > > > > easy-to-configure > > > > > > approach for Azure. > > > > > > > > > > > > [1] > > > > > > > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > > > > > > [2] > > > > > > > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Atri > > > > > Apache Concerted > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
These are actually some interesting points. As I'm thinking more about this, I'm leaning towards changing my opinion and voting for the CompletableFuture. Here is my reasoning. First, it's important to keep in mind that CompletableFuture is not an interface that we will implement, it's an implemented class. Therefore, some of the concerns around complete() and cancel() method are not really relevant -- it's not up to us how these methods behave, they're already implemented. Second, CompletableFuture does provide some useful functionality (anyOf is one of the examples). I can even envision users wanting to complete the future under certain circumstances, e.g. after a timeout, using the completeOnTimeout method. Stripping them from such functionality, which they are used to, is most likely a bad idea. And finally, we can have an IgniteFuture that extends CompletableFuture. This might be useful if want the cancel() operation to cancel the underlying operation. This way we keep all the functionality of CompletableFuture while keeping a certain amount of flexibility for specific cases. Thoughts? -Val On Tue, Mar 30, 2021 at 5:36 AM Denis Garus wrote: > > Completing future from outside will never respect other subscribers that > > may expect other guatantees. > > For example, if we talk about public API like IgniteCache, what subscribers > may expect other guatantees? > IMHO, the best solution is to get the well-known standard interface to a > user, and he will be happy. > > But when we talk about internal classes like "exchange future" they could > be custom futures if convenient. > > вт, 30 мар. 2021 г. в 15:25, Atri Sharma : > > > IMO the only way Ignite should cancel computations is iff cancel method > is > > invoked, not implicitly if complete is invoked. > > > > On Tue, 30 Mar 2021 at 4:58 PM, Denis Garus wrote: > > > > > Hello! > > > > > > > Let's say a user started a compute with fut = compute.runAsync(task); > > > > and now calls fut.complete(someVal); Does this mean that Ignite no > > longer > > > needs to execute the task? > > > > If the task is currently running, does it need to be canceled? > > > > > > Yes, this case looks like Ignite should cancel computations because a > > user > > > wants to complete the future. Why not? > > > If there will be an opportunity to cancel a future, why is it a bad > > option > > > to finish a future through a complete() method? > > > > > > > If you look at Ignite-2 code, you may found a number of places where > we > > > return e.g. exchange futures or partition release futures. > > > > Assume the impact if we will return CompletableFuture instead, which > > can > > > be completed in 3-rd party plugin by mistake? > > > > > > If exchange futures or partition release futures can be returned to > 3-rd > > > party plugin by mistake, it is poor encapsulation. > > > And if it will be IgniteFuter rather than CompletedFuture, anyway, this > > can > > > harm. > > > > > > вт, 30 мар. 2021 г. в 13:14, Andrey Mashenkov < > > andrey.mashen...@gmail.com > > > >: > > > > > > > Guys, > > > > > > > > I want to remember there is one more point to pay attention to. > > > > Extending Future and CompletableStage is more than just prevents > > > unexpected > > > > behavior if a user completed the future. > > > > > > > > First of all, it helps us to write safer code as we won't a method > > > contract > > > > exposed such methods as to a user as to a developer. > > > > If you look at Ignite-2 code, you may found a number of places where > we > > > > return e.g. exchange futures or partition release futures. > > > > Assume the impact if we will return CompletableFuture instead, which > > can > > > be > > > > completed in 3-rd party plugin by mistake? > > > > > > > > The suggested approach allows us to don't bother if a > CompletableFuture > > > has > > > > to be wrapped or not. > > > > > > > > > > > > On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk < > > > > alexey.goncha...@gmail.com> wrote: > > > > > > > > > Ivan, > > > > > > > > > > My concern with the concept of a user completing the future > returned > > > from > > > > > Ignite public API is that it is unclear how to interpret this > action > > > > (this > > > > > backs Val's message). > > > > > Let's say a user started a compute with fut = > compute.runAsync(task); > > > and > > > > > now calls fut.complete(someVal); Does this mean that Ignite no > longer > > > > needs > > > > > to execute the task? If the task is currently running, does it need > > to > > > be > > > > > canceled? > > > > > > > > > > Using CompletableFuture.anyOf() is a good instrument in this case > > > because > > > > > it makes the 'first future wins' contract explicit in the code. > > Besides > > > > > that, the point regarding the cancel() method is valid, and we will > > > need > > > > > some custom mechanics to cancel a computation, so a custom > interface > > > that > > > > > simply extends both Future and CompletableStage seems reasonable to > >
Re: Terms clarification and modules splitting logic
+1 to package and module naming. +1 to service definition as "component providing a high-level API to user/other components/services" I would avoid defining strict rules for Manager and Processor. For me it just adds confusion without real value. A component can be a Manager if it manages something, a Processor if it processes something, and so on. I think having Component and Service (which is also a Component) is enough. Any component can be singleton or not - it's defined by its lifecycle. +1 to renaming core to something more meaningful, but the name lang doesn't fit for a collection of utility classes for me, I would prefer ignite-util. Apache Tomcat has the same jar, for reference. I'm also fine to leave it as is. -1 to have an "internal" package. All modules are known to be internal except api and (partially) util, so why bother at all? вт, 30 мар. 2021 г. в 12:05, Andrey Mashenkov : > Agree with package and module naming. > > I just thought that > Service is a self-suffucient component and provides high-level API to > user/other components/services (e.g. RaftService to TableService). > Manager is internal component - a logical brick of the Service (e.g. > RaftGroupManager or TableSchemaManager, TableAffinityManager), it is not > self-sufficient as affinity or schema make no sense without the table. > Processor is just helper-component of the Service that routes messages, > executes async tasks, manages subscriptions and implements some secondary > functions. > > On Tue, Mar 30, 2021 at 11:24 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > > > Hello Alexander, Igniters, > > > > I support the suggestion, we need to work out some ground rules to have a > > consistent naming convention. Agree with having at most one component per > > project module - this requirement may turn out to be too strict in the > > future, but now it seems reasonable and may help us to better structure > the > > code. Additionally, I would encourage us to make package names consistent > > with the module's structure to make modules Jigsaw-compliant. We do not > > have module definitions now, but I think it would be great to have them, > it > > should help us to enforce component boundaries and proper responsibility > > encapsulation. > > > > As for the naming, it's not entirely clear for me when to use the term > > Service vs Manager. Serice is an entry point to a component/server, but > so > > is Manager - a Manager defines an API that is exposed by a module to > other > > modules. Subjectively, I see the following difference between a Manager > and > > a Service in the examples of entities you provided: > > * A Manager is a node singleton. Its whole purpose is to provide an API > > gateway for other components into a particular subsystem of a node > > * A Service is an object that is bound to a particular runtime entity > > (raft group service is bound to a raft group, and we can have multiple > Raft > > groups; partition service is bound to a particular partition). We can > > re-create services based on changing runtime state and/or configuration. > > Does this make sense? > > > > Finally, I would use lang module name instead of core (the core is > > confusing because right now core contains all necessary classes required > to > > start a minimal Ignite instance; this sets up wrong expectations for > Ignite > > 3). Additionally, I think it would be good to exploit the old > > org.apache.ignite and org.apache.ignite.internal naming scheme: all > public > > classes must go to the non-internal package. The ignite-lang module will > > have both public and internal packages. This automatically implies that > all > > modules except ignite-api and ignite-lang must reside solely in > > org.apache.ignite.internal.* packages. This will be easy to check and > > maintain. > > > > Throughts? > > > > --AG > > > > пт, 26 мар. 2021 г. в 20:28, Alexander Lapin : > > > > > Igniters, > > > > > > Seems that within Ignite-3 we have some mess in terms like manager, > cpu, > > > service, module, etc. Let's clarify this point. Also It'll be great to > > > discuss the rules of dividing code into modules. > > > I'll use the context of Ignite cluster & node lifecycle > > > < > > > > > > https://github.com/apache/ignite-3/blob/ignite-14393/modules/runner/README.md > > > > > > > for terms definition and as an example source. > > > > > > *Terms clarification.* > > > > > >- Component - semantically consistent part of Ignite that in most > > cases > > >will have component-public but ignite-internal API and a lifecycle, > > > somehow > > >related to the lifecycle of a node or cluster. So, *structurally* > > >TableManager, SchemaManager, AffinityManager, etc are all > components. > > > For > > >example, TableManager will have methods like createTable(), > > > alterTable(), > > >dropTable(), etc and a lifecycle that will create listeners (aka > > >DistributedMetastorage watches) on schema and affinity updates
[jira] [Created] (IGNITE-14446) Implement simple version of watching mechanism
Kirill Gusakov created IGNITE-14446: --- Summary: Implement simple version of watching mechanism Key: IGNITE-14446 URL: https://issues.apache.org/jira/browse/IGNITE-14446 Project: Ignite Issue Type: Sub-task Reporter: Kirill Gusakov Assignee: Kirill Gusakov -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
> Completing future from outside will never respect other subscribers that > may expect other guatantees. For example, if we talk about public API like IgniteCache, what subscribers may expect other guatantees? IMHO, the best solution is to get the well-known standard interface to a user, and he will be happy. But when we talk about internal classes like "exchange future" they could be custom futures if convenient. вт, 30 мар. 2021 г. в 15:25, Atri Sharma : > IMO the only way Ignite should cancel computations is iff cancel method is > invoked, not implicitly if complete is invoked. > > On Tue, 30 Mar 2021 at 4:58 PM, Denis Garus wrote: > > > Hello! > > > > > Let's say a user started a compute with fut = compute.runAsync(task); > > > and now calls fut.complete(someVal); Does this mean that Ignite no > longer > > needs to execute the task? > > > If the task is currently running, does it need to be canceled? > > > > Yes, this case looks like Ignite should cancel computations because a > user > > wants to complete the future. Why not? > > If there will be an opportunity to cancel a future, why is it a bad > option > > to finish a future through a complete() method? > > > > > If you look at Ignite-2 code, you may found a number of places where we > > return e.g. exchange futures or partition release futures. > > > Assume the impact if we will return CompletableFuture instead, which > can > > be completed in 3-rd party plugin by mistake? > > > > If exchange futures or partition release futures can be returned to 3-rd > > party plugin by mistake, it is poor encapsulation. > > And if it will be IgniteFuter rather than CompletedFuture, anyway, this > can > > harm. > > > > вт, 30 мар. 2021 г. в 13:14, Andrey Mashenkov < > andrey.mashen...@gmail.com > > >: > > > > > Guys, > > > > > > I want to remember there is one more point to pay attention to. > > > Extending Future and CompletableStage is more than just prevents > > unexpected > > > behavior if a user completed the future. > > > > > > First of all, it helps us to write safer code as we won't a method > > contract > > > exposed such methods as to a user as to a developer. > > > If you look at Ignite-2 code, you may found a number of places where we > > > return e.g. exchange futures or partition release futures. > > > Assume the impact if we will return CompletableFuture instead, which > can > > be > > > completed in 3-rd party plugin by mistake? > > > > > > The suggested approach allows us to don't bother if a CompletableFuture > > has > > > to be wrapped or not. > > > > > > > > > On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk < > > > alexey.goncha...@gmail.com> wrote: > > > > > > > Ivan, > > > > > > > > My concern with the concept of a user completing the future returned > > from > > > > Ignite public API is that it is unclear how to interpret this action > > > (this > > > > backs Val's message). > > > > Let's say a user started a compute with fut = compute.runAsync(task); > > and > > > > now calls fut.complete(someVal); Does this mean that Ignite no longer > > > needs > > > > to execute the task? If the task is currently running, does it need > to > > be > > > > canceled? > > > > > > > > Using CompletableFuture.anyOf() is a good instrument in this case > > because > > > > it makes the 'first future wins' contract explicit in the code. > Besides > > > > that, the point regarding the cancel() method is valid, and we will > > need > > > > some custom mechanics to cancel a computation, so a custom interface > > that > > > > simply extends both Future and CompletableStage seems reasonable to > me. > > > > > > > > --AG > > > > > > > > пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin : > > > > > > > > > Val, > > > > > > > > > > There were enough hype around Reactive programming past years. I > > > > > remind a lot of talks about RxJava. And I suppose it worth to > > consider > > > > > it. But it requires some time to study modern trends to make a > > choice. > > > > > So far I am not ready to facilitate Reactive API for Ignite 3. > > > > > > > > > > Regarding CompletableFuture. > > > > > > > > > > > The point is that currently a future returned from any of > Ignite's > > > > async > > > > > > operations is supposed to be completed with a value only by > Ignite > > > > > itself, > > > > > > not by the user. If we follow the same approach in Ignite 3, > > > returning > > > > > > CompletableFuture is surely wrong in my view. > > > > > > > > > > My first thoughts was similar. But later I thought what a user > would > > > > > like do with returned future. And one of cases I imagined was a > case > > > > > of alternative result. E.g. a user uses Ignite and another data > > source > > > > > in his application. He wants to use a value arrived faster. He > > > > > combines 2 futures like CompletableFuture.anyOf(...). Consequently > > > > > even if we prohibit CompletableFuture.complete(...) explicitly then > > it > > > > > will be possible to create a combination that will
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
IMO the only way Ignite should cancel computations is iff cancel method is invoked, not implicitly if complete is invoked. On Tue, 30 Mar 2021 at 4:58 PM, Denis Garus wrote: > Hello! > > > Let's say a user started a compute with fut = compute.runAsync(task); > > and now calls fut.complete(someVal); Does this mean that Ignite no longer > needs to execute the task? > > If the task is currently running, does it need to be canceled? > > Yes, this case looks like Ignite should cancel computations because a user > wants to complete the future. Why not? > If there will be an opportunity to cancel a future, why is it a bad option > to finish a future through a complete() method? > > > If you look at Ignite-2 code, you may found a number of places where we > return e.g. exchange futures or partition release futures. > > Assume the impact if we will return CompletableFuture instead, which can > be completed in 3-rd party plugin by mistake? > > If exchange futures or partition release futures can be returned to 3-rd > party plugin by mistake, it is poor encapsulation. > And if it will be IgniteFuter rather than CompletedFuture, anyway, this can > harm. > > вт, 30 мар. 2021 г. в 13:14, Andrey Mashenkov >: > > > Guys, > > > > I want to remember there is one more point to pay attention to. > > Extending Future and CompletableStage is more than just prevents > unexpected > > behavior if a user completed the future. > > > > First of all, it helps us to write safer code as we won't a method > contract > > exposed such methods as to a user as to a developer. > > If you look at Ignite-2 code, you may found a number of places where we > > return e.g. exchange futures or partition release futures. > > Assume the impact if we will return CompletableFuture instead, which can > be > > completed in 3-rd party plugin by mistake? > > > > The suggested approach allows us to don't bother if a CompletableFuture > has > > to be wrapped or not. > > > > > > On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk < > > alexey.goncha...@gmail.com> wrote: > > > > > Ivan, > > > > > > My concern with the concept of a user completing the future returned > from > > > Ignite public API is that it is unclear how to interpret this action > > (this > > > backs Val's message). > > > Let's say a user started a compute with fut = compute.runAsync(task); > and > > > now calls fut.complete(someVal); Does this mean that Ignite no longer > > needs > > > to execute the task? If the task is currently running, does it need to > be > > > canceled? > > > > > > Using CompletableFuture.anyOf() is a good instrument in this case > because > > > it makes the 'first future wins' contract explicit in the code. Besides > > > that, the point regarding the cancel() method is valid, and we will > need > > > some custom mechanics to cancel a computation, so a custom interface > that > > > simply extends both Future and CompletableStage seems reasonable to me. > > > > > > --AG > > > > > > пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin : > > > > > > > Val, > > > > > > > > There were enough hype around Reactive programming past years. I > > > > remind a lot of talks about RxJava. And I suppose it worth to > consider > > > > it. But it requires some time to study modern trends to make a > choice. > > > > So far I am not ready to facilitate Reactive API for Ignite 3. > > > > > > > > Regarding CompletableFuture. > > > > > > > > > The point is that currently a future returned from any of Ignite's > > > async > > > > > operations is supposed to be completed with a value only by Ignite > > > > itself, > > > > > not by the user. If we follow the same approach in Ignite 3, > > returning > > > > > CompletableFuture is surely wrong in my view. > > > > > > > > My first thoughts was similar. But later I thought what a user would > > > > like do with returned future. And one of cases I imagined was a case > > > > of alternative result. E.g. a user uses Ignite and another data > source > > > > in his application. He wants to use a value arrived faster. He > > > > combines 2 futures like CompletableFuture.anyOf(...). Consequently > > > > even if we prohibit CompletableFuture.complete(...) explicitly then > it > > > > will be possible to create a combination that will allow premature > > > > future completion. After all generally CompletableFuture is a > > > > placeholder for async computaion result and if a user wants to > > > > substitute result returned from Ignite why should we disallow him to > > > > do it? > > > > > > > > Also I found one more suspicious thing with CompletableFuture. As it > > > > is a concrete class it implements a cancel() method. And as I see the > > > > implementation does not try to cancel underlying computations. Is not > > > > it a problem? > > > > > > > > 2021-03-29 7:30 GMT+03:00, Valentin Kulichenko < > > > > valentin.kuliche...@gmail.com>: > > > > > Ivan, > > > > > > > > > > It's not really about the "harm", but more about "what should we do > > if > > > > this
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
> > Yes, this case looks like Ignite should cancel computations because a user > wants to complete the future. Why not? > If there will be an opportunity to cancel a future, why is it a bad option > to finish a future through a complete() method? Future has cancel() method for this. Completing future from outside will never respect other subscribers that may expect other guatantees. > > If exchange futures or partition release futures can be returned to 3-rd > party plugin by mistake, it is poor encapsulation. > And if it will be IgniteFuter rather than CompletedFuture, anyway, this can > harm. > IgniteFuture contract will not have complete() or similar methods, but we a free to pass any implementation. Obviously, if a user casts the future to CompletableFuture or uses a reflection to access IgniteFutureImpl hidden methods - that means the user breaks the contract and CompletableFuture defensive copy will not help either. On Tue, Mar 30, 2021 at 2:28 PM Denis Garus wrote: > Hello! > > > Let's say a user started a compute with fut = compute.runAsync(task); > > and now calls fut.complete(someVal); Does this mean that Ignite no longer > needs to execute the task? > > If the task is currently running, does it need to be canceled? > > Yes, this case looks like Ignite should cancel computations because a user > wants to complete the future. Why not? > If there will be an opportunity to cancel a future, why is it a bad option > to finish a future through a complete() method? > > > If you look at Ignite-2 code, you may found a number of places where we > return e.g. exchange futures or partition release futures. > > Assume the impact if we will return CompletableFuture instead, which can > be completed in 3-rd party plugin by mistake? > > If exchange futures or partition release futures can be returned to 3-rd > party plugin by mistake, it is poor encapsulation. > And if it will be IgniteFuter rather than CompletedFuture, anyway, this can > harm. > > вт, 30 мар. 2021 г. в 13:14, Andrey Mashenkov >: > > > Guys, > > > > I want to remember there is one more point to pay attention to. > > Extending Future and CompletableStage is more than just prevents > unexpected > > behavior if a user completed the future. > > > > First of all, it helps us to write safer code as we won't a method > contract > > exposed such methods as to a user as to a developer. > > If you look at Ignite-2 code, you may found a number of places where we > > return e.g. exchange futures or partition release futures. > > Assume the impact if we will return CompletableFuture instead, which can > be > > completed in 3-rd party plugin by mistake? > > > > The suggested approach allows us to don't bother if a CompletableFuture > has > > to be wrapped or not. > > > > > > On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk < > > alexey.goncha...@gmail.com> wrote: > > > > > Ivan, > > > > > > My concern with the concept of a user completing the future returned > from > > > Ignite public API is that it is unclear how to interpret this action > > (this > > > backs Val's message). > > > Let's say a user started a compute with fut = compute.runAsync(task); > and > > > now calls fut.complete(someVal); Does this mean that Ignite no longer > > needs > > > to execute the task? If the task is currently running, does it need to > be > > > canceled? > > > > > > Using CompletableFuture.anyOf() is a good instrument in this case > because > > > it makes the 'first future wins' contract explicit in the code. Besides > > > that, the point regarding the cancel() method is valid, and we will > need > > > some custom mechanics to cancel a computation, so a custom interface > that > > > simply extends both Future and CompletableStage seems reasonable to me. > > > > > > --AG > > > > > > пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin : > > > > > > > Val, > > > > > > > > There were enough hype around Reactive programming past years. I > > > > remind a lot of talks about RxJava. And I suppose it worth to > consider > > > > it. But it requires some time to study modern trends to make a > choice. > > > > So far I am not ready to facilitate Reactive API for Ignite 3. > > > > > > > > Regarding CompletableFuture. > > > > > > > > > The point is that currently a future returned from any of Ignite's > > > async > > > > > operations is supposed to be completed with a value only by Ignite > > > > itself, > > > > > not by the user. If we follow the same approach in Ignite 3, > > returning > > > > > CompletableFuture is surely wrong in my view. > > > > > > > > My first thoughts was similar. But later I thought what a user would > > > > like do with returned future. And one of cases I imagined was a case > > > > of alternative result. E.g. a user uses Ignite and another data > source > > > > in his application. He wants to use a value arrived faster. He > > > > combines 2 futures like CompletableFuture.anyOf(...). Consequently > > > > even if we prohibit CompletableFuture.complete(...)
[jira] [Created] (IGNITE-14445) "Remote node does not observe current" after failure by not receiving metrics from client
Ilya Kasnacheev created IGNITE-14445:
Summary: "Remote node does not observe current" after failure by
not receiving metrics from client
Key: IGNITE-14445
URL: https://issues.apache.org/jira/browse/IGNITE-14445
Project: Ignite
Issue Type: Bug
Affects Versions: 2.9.1
Reporter: Ilya Kasnacheev
A server node might fail a client node due to pauses in the network connection:
[15:07:16,330][WARNING][tcp-disco-msg-worker-[11cf0c06 10.212.120.71:57500
crd]-#2%hh_DynamicGrid_v2%][TcpDiscoverySpi] Failing client node due to not
receiving metrics updates from client node within
'IgniteConfiguration.clientFailureDetectionTimeout' (consider increasing
configuration property) [timeout=12, node=TcpDiscoveryNode
[id=9dbcfb86-a60e-4382-904f-57bffbe18c5c,consistentId=73B5811B-9644-48FD-A533-B4609FDAD591,
addrs=ArrayList [10.212.120.190], sockAddrs=HashSet
[VWNV02AX07080.HH.com/10.212.120.190:0], discPort=0, order=488, intOrder=248,
lastExchangeTime=1612397142960, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true]]
Then, the client node will never understand that it is dropped by cluster and
will be endlessly trying to connect. I'm not sure what does discovery do on the
client node:
{code}
[15:07:42,689][SEVERE][Thread-219][TcpCommunicationSpi] Failed to send message
to remote node [node=TcpDiscoveryNode [id=83fd7c70-839d-46ca-969f-bbb9661d6ab2,
consistentId=127.1.1.1:57500, addrs=ArrayList [127.1.1.1], sockAddrs=HashSet
[test.com/127.1.1.1:57500], discPort=57500, order=1, intOrder=1,
lastExchangeTime=1612397256785, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false,
msg=GridNearAtomicFullUpdateRequest [keys=ArrayList [UserKeyCacheObjectImpl
[part=292, val=TestModel:TEST|bbf4da4d-c3d7-4b46-98b6-0de70c30f668,
hasValBytes=true]], conflictTtls=null, conflictExpireTimes=null,
expiryPlc=org.apache.ignite.internal.processors.platform.cache.expiry.PlatformExpiryPolicy@3fb1b76e,
initSize=1, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
flags=keepBinary
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Remote node does not observe current node in topology :
83fd7c70-839d-46ca-969f-bbb9661d6ab2
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3622)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2132)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1296)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:312)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
Hello!
> Let's say a user started a compute with fut = compute.runAsync(task);
> and now calls fut.complete(someVal); Does this mean that Ignite no longer
needs to execute the task?
> If the task is currently running, does it need to be canceled?
Yes, this case looks like Ignite should cancel computations because a user
wants to complete the future. Why not?
If there will be an opportunity to cancel a future, why is it a bad option
to finish a future through a complete() method?
> If you look at Ignite-2 code, you may found a number of places where we
return e.g. exchange futures or partition release futures.
> Assume the impact if we will return CompletableFuture instead, which can
be completed in 3-rd party plugin by mistake?
If exchange futures or partition release futures can be returned to 3-rd
party plugin by mistake, it is poor encapsulation.
And if it will be IgniteFuter rather than CompletedFuture, anyway, this can
harm.
вт, 30 мар. 2021 г. в 13:14, Andrey Mashenkov :
> Guys,
>
> I want to remember there is one more point to pay attention to.
> Extending Future and CompletableStage is more than just prevents unexpected
> behavior if a user completed the future.
>
> First of all, it helps us to write safer code as we won't a method contract
> exposed such methods as to a user as to a developer.
> If you look at Ignite-2 code, you may found a number of places where we
> return e.g. exchange futures or partition release futures.
> Assume the impact if we will return CompletableFuture instead, which can be
> completed in 3-rd party plugin by mistake?
>
> The suggested approach allows us to don't bother if a CompletableFuture has
> to be wrapped or not.
>
>
> On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Ivan,
> >
> > My concern with the concept of a user completing the future returned from
> > Ignite public API is that it is unclear how to interpret this action
> (this
> > backs Val's message).
> > Let's say a user started a compute with fut = compute.runAsync(task); and
> > now calls fut.complete(someVal); Does this mean that Ignite no longer
> needs
> > to execute the task? If the task is currently running, does it need to be
> > canceled?
> >
> > Using CompletableFuture.anyOf() is a good instrument in this case because
> > it makes the 'first future wins' contract explicit in the code. Besides
> > that, the point regarding the cancel() method is valid, and we will need
> > some custom mechanics to cancel a computation, so a custom interface that
> > simply extends both Future and CompletableStage seems reasonable to me.
> >
> > --AG
> >
> > пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin :
> >
> > > Val,
> > >
> > > There were enough hype around Reactive programming past years. I
> > > remind a lot of talks about RxJava. And I suppose it worth to consider
> > > it. But it requires some time to study modern trends to make a choice.
> > > So far I am not ready to facilitate Reactive API for Ignite 3.
> > >
> > > Regarding CompletableFuture.
> > >
> > > > The point is that currently a future returned from any of Ignite's
> > async
> > > > operations is supposed to be completed with a value only by Ignite
> > > itself,
> > > > not by the user. If we follow the same approach in Ignite 3,
> returning
> > > > CompletableFuture is surely wrong in my view.
> > >
> > > My first thoughts was similar. But later I thought what a user would
> > > like do with returned future. And one of cases I imagined was a case
> > > of alternative result. E.g. a user uses Ignite and another data source
> > > in his application. He wants to use a value arrived faster. He
> > > combines 2 futures like CompletableFuture.anyOf(...). Consequently
> > > even if we prohibit CompletableFuture.complete(...) explicitly then it
> > > will be possible to create a combination that will allow premature
> > > future completion. After all generally CompletableFuture is a
> > > placeholder for async computaion result and if a user wants to
> > > substitute result returned from Ignite why should we disallow him to
> > > do it?
> > >
> > > Also I found one more suspicious thing with CompletableFuture. As it
> > > is a concrete class it implements a cancel() method. And as I see the
> > > implementation does not try to cancel underlying computations. Is not
> > > it a problem?
> > >
> > > 2021-03-29 7:30 GMT+03:00, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > > > Ivan,
> > > >
> > > > It's not really about the "harm", but more about "what should we do
> if
> > > this
> > > > method is called?". Imagine the following code:
> > > >
> > > > CompletableFuture fut = cache.getAsync(key);
> > > > fut.complete("something");
> > > >
> > > > What should happen in this case?
> > > >
> > > > The point is that currently a future returned from any of Ignite's
> > async
> > > > operations is supposed to be completed with a value only by Ignite
> > > itself,
> > > > not by the user. If we
Re: Significant Items to Tackle
Hello, One more area where newcomers' contributions will be very helpful is MTCGA activity (see [1], [2]). There are a lot of flaky tests in Ignite, usually, the reason of flakiness is some concurrency issue in test code but sometimes tests flaky due to some bugs in production code. Why MTCGA is a good starting point? For the contributor, investigation of such problems gives an understanding of Ignite internals. For the reviewer, usually, such fixes have less code and require less time to review compared to new features. For the community, less flaky tests mean more relevant results of TC runs and less time to rerun some test suites. If you are interested in this activity, you can take some flaky test from TC (for example, on this page [3] you can find a list of failed tests with failure rate from the last 10 TC runs on the master branch), try to reproduce failure locally (some problems very hard to reproduce locally, they are fired only on TC) and if it's reproducible, create a ticket and try to fix it. [1]: http://apache-ignite-developers.2346864.n4.nabble.com/Make-Teamcity-Green-Again-td19873.html [2]: https://cwiki.apache.org/confluence/display/IGNITE/Make+Teamcity+Green+Again [3]: https://mtcga.ignite.apache.org/current.html?branch=master=10 пн, 29 мар. 2021 г. в 23:39, Atri Sharma : > On further look, this looks interesting and I am happy to dive into the > spring framework. > > Please help me in getting started > > > On Mon, 29 Mar 2021, 11:49 Atri Sharma, wrote: > > > Thanks Sam and Ivan. > > > > I am happy to look into that but it looks like most of the work is done > > there? > > > > Also, is there something in core that I can pickup as well? > > > > > > On Mon, 29 Mar 2021, 03:20 Ivan Pavlukhin, wrote: > > > >> Gmail treated last message as spam. Bumping. > >> > >> 2021-03-28 16:05 GMT+03:00, Данилов Семён : > >> > Hello Atri! > >> > > >> > Would you like to contribute into Ignite Spring integrations? You can > >> take a > >> > look at this particular jira filter: > >> > > >> > https://issues.apache.org/jira/browse/IGNITE-9524?jql=project%20%3D%20Ignite%20and%20summary%20~%20%22spring%22%20and%20status%20not%20in%20(Resolved%2C%20Closed) > >> > . > >> > Personally, I think one of the most valuable issues there is > >> > https://issues.apache.org/jira/browse/IGNITE-9524. I have a lot of > >> > experience with Spring, so you can reach out to me if you have any > >> > questions. > >> > > >> > Cheers, Sam. > >> > > >> > 25.03.2021, 23:33, "Atri Sharma" : > >> >> Hi Community, > >> >> > >> >> First off, I want to thank the community for being so welcoming and > >> >> helpful. This is an awesome place to be in. > >> >> > >> >> Now that I have worked on some issues, I would like to get my hands > >> >> deeper > >> >> and with larger issues in the core. Also, some more intermediate > >> tickets > >> >> to > >> >> tackle, my queue has gotten empty :) > >> >> > >> >> Please help and advice. > >> >> > >> >> Regards, > >> >> > >> >> Atri > >> > > >> > >> > >> -- > >> > >> Best regards, > >> Ivan Pavlukhin > >> > > >
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
Guys,
I want to remember there is one more point to pay attention to.
Extending Future and CompletableStage is more than just prevents unexpected
behavior if a user completed the future.
First of all, it helps us to write safer code as we won't a method contract
exposed such methods as to a user as to a developer.
If you look at Ignite-2 code, you may found a number of places where we
return e.g. exchange futures or partition release futures.
Assume the impact if we will return CompletableFuture instead, which can be
completed in 3-rd party plugin by mistake?
The suggested approach allows us to don't bother if a CompletableFuture has
to be wrapped or not.
On Tue, Mar 30, 2021 at 12:22 PM Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:
> Ivan,
>
> My concern with the concept of a user completing the future returned from
> Ignite public API is that it is unclear how to interpret this action (this
> backs Val's message).
> Let's say a user started a compute with fut = compute.runAsync(task); and
> now calls fut.complete(someVal); Does this mean that Ignite no longer needs
> to execute the task? If the task is currently running, does it need to be
> canceled?
>
> Using CompletableFuture.anyOf() is a good instrument in this case because
> it makes the 'first future wins' contract explicit in the code. Besides
> that, the point regarding the cancel() method is valid, and we will need
> some custom mechanics to cancel a computation, so a custom interface that
> simply extends both Future and CompletableStage seems reasonable to me.
>
> --AG
>
> пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin :
>
> > Val,
> >
> > There were enough hype around Reactive programming past years. I
> > remind a lot of talks about RxJava. And I suppose it worth to consider
> > it. But it requires some time to study modern trends to make a choice.
> > So far I am not ready to facilitate Reactive API for Ignite 3.
> >
> > Regarding CompletableFuture.
> >
> > > The point is that currently a future returned from any of Ignite's
> async
> > > operations is supposed to be completed with a value only by Ignite
> > itself,
> > > not by the user. If we follow the same approach in Ignite 3, returning
> > > CompletableFuture is surely wrong in my view.
> >
> > My first thoughts was similar. But later I thought what a user would
> > like do with returned future. And one of cases I imagined was a case
> > of alternative result. E.g. a user uses Ignite and another data source
> > in his application. He wants to use a value arrived faster. He
> > combines 2 futures like CompletableFuture.anyOf(...). Consequently
> > even if we prohibit CompletableFuture.complete(...) explicitly then it
> > will be possible to create a combination that will allow premature
> > future completion. After all generally CompletableFuture is a
> > placeholder for async computaion result and if a user wants to
> > substitute result returned from Ignite why should we disallow him to
> > do it?
> >
> > Also I found one more suspicious thing with CompletableFuture. As it
> > is a concrete class it implements a cancel() method. And as I see the
> > implementation does not try to cancel underlying computations. Is not
> > it a problem?
> >
> > 2021-03-29 7:30 GMT+03:00, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> > > Ivan,
> > >
> > > It's not really about the "harm", but more about "what should we do if
> > this
> > > method is called?". Imagine the following code:
> > >
> > > CompletableFuture fut = cache.getAsync(key);
> > > fut.complete("something");
> > >
> > > What should happen in this case?
> > >
> > > The point is that currently a future returned from any of Ignite's
> async
> > > operations is supposed to be completed with a value only by Ignite
> > itself,
> > > not by the user. If we follow the same approach in Ignite 3, returning
> > > CompletableFuture is surely wrong in my view.
> > >
> > > At the same time, if we take a fundamentally different route with the
> > async
> > > APIs, this whole discussion might become irrelevant. For example, can
> you
> > > elaborate on your thinking around the reactive API? Do you have any
> > > specifics in mind?
> > >
> > > -Val
> > >
> > > On Sat, Mar 27, 2021 at 9:18 PM Ivan Pavlukhin
> > wrote:
> > >
> > >> > The methods below shouldn't be accessible for user:
> > >> > complete()
> > >> > completeExceptionaly()
> > >>
> > >> Folks, in case of user-facing API, do you think there is a real harm
> > >> in allowing a user to manually "complete" a future? I suppose a user
> > >> employs some post-processing for future results and potentially wants
> > >> to have control of these results as well. E.g. premature completion in
> > >> case when a result is no longer needed is possible usage.
> > >>
> > >> Also I thinkg it might be a good time to ponder about Future/Promise
> > >> APIs in general. Why such API is our choice? Can we choose e.g.
> > >> Reactive API style instead?
> > >>
> > >> 2021-03-27 0:33
[jira] [Created] (IGNITE-14444) Move affinity calculation and storage to client
Ivan Daschinskiy created IGNITE-1: - Summary: Move affinity calculation and storage to client Key: IGNITE-1 URL: https://issues.apache.org/jira/browse/IGNITE-1 Project: Ignite Issue Type: Improvement Components: python Reporter: Ivan Daschinskiy In current implementation, affinity storage and affinity calculation are located in cache. It is not optimal: 1. affinity is not shared between Cache instance with same name 2. affinity mapping requests per cache and add additional loads. 3. if we start implementing transactions or expiry policy, this can be an issue. I propose to move affinity storage to Client and AioClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14443) Calcite integration. SqlFirstLastValueAggFunction support
Pavel Vinokurov created IGNITE-14443: Summary: Calcite integration. SqlFirstLastValueAggFunction support Key: IGNITE-14443 URL: https://issues.apache.org/jira/browse/IGNITE-14443 Project: Ignite Issue Type: New Feature Components: sql Affects Versions: 3.0.0-alpha1 Reporter: Pavel Vinokurov We need to support aggregation functions, especially SqlFirstLastValueAggFunction that allows simplify and optimize the wide range of sql queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
Ivan,
My concern with the concept of a user completing the future returned from
Ignite public API is that it is unclear how to interpret this action (this
backs Val's message).
Let's say a user started a compute with fut = compute.runAsync(task); and
now calls fut.complete(someVal); Does this mean that Ignite no longer needs
to execute the task? If the task is currently running, does it need to be
canceled?
Using CompletableFuture.anyOf() is a good instrument in this case because
it makes the 'first future wins' contract explicit in the code. Besides
that, the point regarding the cancel() method is valid, and we will need
some custom mechanics to cancel a computation, so a custom interface that
simply extends both Future and CompletableStage seems reasonable to me.
--AG
пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin :
> Val,
>
> There were enough hype around Reactive programming past years. I
> remind a lot of talks about RxJava. And I suppose it worth to consider
> it. But it requires some time to study modern trends to make a choice.
> So far I am not ready to facilitate Reactive API for Ignite 3.
>
> Regarding CompletableFuture.
>
> > The point is that currently a future returned from any of Ignite's async
> > operations is supposed to be completed with a value only by Ignite
> itself,
> > not by the user. If we follow the same approach in Ignite 3, returning
> > CompletableFuture is surely wrong in my view.
>
> My first thoughts was similar. But later I thought what a user would
> like do with returned future. And one of cases I imagined was a case
> of alternative result. E.g. a user uses Ignite and another data source
> in his application. He wants to use a value arrived faster. He
> combines 2 futures like CompletableFuture.anyOf(...). Consequently
> even if we prohibit CompletableFuture.complete(...) explicitly then it
> will be possible to create a combination that will allow premature
> future completion. After all generally CompletableFuture is a
> placeholder for async computaion result and if a user wants to
> substitute result returned from Ignite why should we disallow him to
> do it?
>
> Also I found one more suspicious thing with CompletableFuture. As it
> is a concrete class it implements a cancel() method. And as I see the
> implementation does not try to cancel underlying computations. Is not
> it a problem?
>
> 2021-03-29 7:30 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Ivan,
> >
> > It's not really about the "harm", but more about "what should we do if
> this
> > method is called?". Imagine the following code:
> >
> > CompletableFuture fut = cache.getAsync(key);
> > fut.complete("something");
> >
> > What should happen in this case?
> >
> > The point is that currently a future returned from any of Ignite's async
> > operations is supposed to be completed with a value only by Ignite
> itself,
> > not by the user. If we follow the same approach in Ignite 3, returning
> > CompletableFuture is surely wrong in my view.
> >
> > At the same time, if we take a fundamentally different route with the
> async
> > APIs, this whole discussion might become irrelevant. For example, can you
> > elaborate on your thinking around the reactive API? Do you have any
> > specifics in mind?
> >
> > -Val
> >
> > On Sat, Mar 27, 2021 at 9:18 PM Ivan Pavlukhin
> wrote:
> >
> >> > The methods below shouldn't be accessible for user:
> >> > complete()
> >> > completeExceptionaly()
> >>
> >> Folks, in case of user-facing API, do you think there is a real harm
> >> in allowing a user to manually "complete" a future? I suppose a user
> >> employs some post-processing for future results and potentially wants
> >> to have control of these results as well. E.g. premature completion in
> >> case when a result is no longer needed is possible usage.
> >>
> >> Also I thinkg it might be a good time to ponder about Future/Promise
> >> APIs in general. Why such API is our choice? Can we choose e.g.
> >> Reactive API style instead?
> >>
> >> 2021-03-27 0:33 GMT+03:00, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com>:
> >> > Andrey,
> >> >
> >> > I see. So in a nutshell, you're saying that we want to return a future
> >> that
> >> > the user's code is not allowed to complete. In this case, I think it's
> >> > clear that CompletableFuture is not what we need. We actually need a
> >> > NonCompletableFuture :)
> >> >
> >> > My vote is for the custom interface.
> >> >
> >> > -Val
> >> >
> >> > On Fri, Mar 26, 2021 at 2:25 AM Andrey Mashenkov
> >> >
> >> > wrote:
> >> >
> >> >> Val,
> >> >>
> >> >> The methods below shouldn't be accessible for user:
> >> >> complete()
> >> >> completeExceptionaly()
> >> >>
> >> >> Returning CompletableFuture we must always make a copy to prevent the
> >> >> original future from being completed by mistake.
> >> >> I think it will NOT be enough to do that returing the future to the
> >> >> end-user, but from every critical module to the outside of the
> module,
Re: Terms clarification and modules splitting logic
Agree with package and module naming. I just thought that Service is a self-suffucient component and provides high-level API to user/other components/services (e.g. RaftService to TableService). Manager is internal component - a logical brick of the Service (e.g. RaftGroupManager or TableSchemaManager, TableAffinityManager), it is not self-sufficient as affinity or schema make no sense without the table. Processor is just helper-component of the Service that routes messages, executes async tasks, manages subscriptions and implements some secondary functions. On Tue, Mar 30, 2021 at 11:24 AM Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Hello Alexander, Igniters, > > I support the suggestion, we need to work out some ground rules to have a > consistent naming convention. Agree with having at most one component per > project module - this requirement may turn out to be too strict in the > future, but now it seems reasonable and may help us to better structure the > code. Additionally, I would encourage us to make package names consistent > with the module's structure to make modules Jigsaw-compliant. We do not > have module definitions now, but I think it would be great to have them, it > should help us to enforce component boundaries and proper responsibility > encapsulation. > > As for the naming, it's not entirely clear for me when to use the term > Service vs Manager. Serice is an entry point to a component/server, but so > is Manager - a Manager defines an API that is exposed by a module to other > modules. Subjectively, I see the following difference between a Manager and > a Service in the examples of entities you provided: > * A Manager is a node singleton. Its whole purpose is to provide an API > gateway for other components into a particular subsystem of a node > * A Service is an object that is bound to a particular runtime entity > (raft group service is bound to a raft group, and we can have multiple Raft > groups; partition service is bound to a particular partition). We can > re-create services based on changing runtime state and/or configuration. > Does this make sense? > > Finally, I would use lang module name instead of core (the core is > confusing because right now core contains all necessary classes required to > start a minimal Ignite instance; this sets up wrong expectations for Ignite > 3). Additionally, I think it would be good to exploit the old > org.apache.ignite and org.apache.ignite.internal naming scheme: all public > classes must go to the non-internal package. The ignite-lang module will > have both public and internal packages. This automatically implies that all > modules except ignite-api and ignite-lang must reside solely in > org.apache.ignite.internal.* packages. This will be easy to check and > maintain. > > Throughts? > > --AG > > пт, 26 мар. 2021 г. в 20:28, Alexander Lapin : > > > Igniters, > > > > Seems that within Ignite-3 we have some mess in terms like manager, cpu, > > service, module, etc. Let's clarify this point. Also It'll be great to > > discuss the rules of dividing code into modules. > > I'll use the context of Ignite cluster & node lifecycle > > < > > > https://github.com/apache/ignite-3/blob/ignite-14393/modules/runner/README.md > > > > > for terms definition and as an example source. > > > > *Terms clarification.* > > > >- Component - semantically consistent part of Ignite that in most > cases > >will have component-public but ignite-internal API and a lifecycle, > > somehow > >related to the lifecycle of a node or cluster. So, *structurally* > >TableManager, SchemaManager, AffinityManager, etc are all components. > > For > >example, TableManager will have methods like createTable(), > > alterTable(), > >dropTable(), etc and a lifecycle that will create listeners (aka > >DistributedMetastorage watches) on schema and affinity updates in > order > > to > >create/drop raft servers for particular partitions that should be > > hosted on > >local node). Components are lined up in a graph without cycles, for > more > >details please see mentioned above Ignite cluster & node lifecycle. > >< > > > https://github.com/apache/ignite-3/blob/ignite-14393/modules/runner/README.md > > > > >- Manager is a driving point of a component with high level lifecycle > >logic and API methods. My intention here is to agree about naming: > > should > >we use the term Manager, Processor or anything else? > >- Service is an entry point to some component/server or a group of > >components/servers. See RaftGroupService.java > >< > > > https://github.com/apache/ignite-3/blob/main/modules/raft-client/src/main/java/org/apache/ignite/raft/client/service/RaftGroupService.java > > > > >as an example. > >- Server, for example RaftServer, seems to be self-explanatory itself. > > > > > > *Dividing code into modules.* > > It seems useful to introduce a restriction that a module should contain > at > >
Re: Terms clarification and modules splitting logic
Hello Alexander, Igniters, I support the suggestion, we need to work out some ground rules to have a consistent naming convention. Agree with having at most one component per project module - this requirement may turn out to be too strict in the future, but now it seems reasonable and may help us to better structure the code. Additionally, I would encourage us to make package names consistent with the module's structure to make modules Jigsaw-compliant. We do not have module definitions now, but I think it would be great to have them, it should help us to enforce component boundaries and proper responsibility encapsulation. As for the naming, it's not entirely clear for me when to use the term Service vs Manager. Serice is an entry point to a component/server, but so is Manager - a Manager defines an API that is exposed by a module to other modules. Subjectively, I see the following difference between a Manager and a Service in the examples of entities you provided: * A Manager is a node singleton. Its whole purpose is to provide an API gateway for other components into a particular subsystem of a node * A Service is an object that is bound to a particular runtime entity (raft group service is bound to a raft group, and we can have multiple Raft groups; partition service is bound to a particular partition). We can re-create services based on changing runtime state and/or configuration. Does this make sense? Finally, I would use lang module name instead of core (the core is confusing because right now core contains all necessary classes required to start a minimal Ignite instance; this sets up wrong expectations for Ignite 3). Additionally, I think it would be good to exploit the old org.apache.ignite and org.apache.ignite.internal naming scheme: all public classes must go to the non-internal package. The ignite-lang module will have both public and internal packages. This automatically implies that all modules except ignite-api and ignite-lang must reside solely in org.apache.ignite.internal.* packages. This will be easy to check and maintain. Throughts? --AG пт, 26 мар. 2021 г. в 20:28, Alexander Lapin : > Igniters, > > Seems that within Ignite-3 we have some mess in terms like manager, cpu, > service, module, etc. Let's clarify this point. Also It'll be great to > discuss the rules of dividing code into modules. > I'll use the context of Ignite cluster & node lifecycle > < > https://github.com/apache/ignite-3/blob/ignite-14393/modules/runner/README.md > > > for terms definition and as an example source. > > *Terms clarification.* > >- Component - semantically consistent part of Ignite that in most cases >will have component-public but ignite-internal API and a lifecycle, > somehow >related to the lifecycle of a node or cluster. So, *structurally* >TableManager, SchemaManager, AffinityManager, etc are all components. > For >example, TableManager will have methods like createTable(), > alterTable(), >dropTable(), etc and a lifecycle that will create listeners (aka >DistributedMetastorage watches) on schema and affinity updates in order > to >create/drop raft servers for particular partitions that should be > hosted on >local node). Components are lined up in a graph without cycles, for more >details please see mentioned above Ignite cluster & node lifecycle. >< > https://github.com/apache/ignite-3/blob/ignite-14393/modules/runner/README.md > > >- Manager is a driving point of a component with high level lifecycle >logic and API methods. My intention here is to agree about naming: > should >we use the term Manager, Processor or anything else? >- Service is an entry point to some component/server or a group of >components/servers. See RaftGroupService.java >< > https://github.com/apache/ignite-3/blob/main/modules/raft-client/src/main/java/org/apache/ignite/raft/client/service/RaftGroupService.java > > >as an example. >- Server, for example RaftServer, seems to be self-explanatory itself. > > > *Dividing code into modules.* > It seems useful to introduce a restriction that a module should contain at > most one component. So that, combining component-specific modules and ones > of api, lang, etc we will end up with something like following: > >- affinity // TO be created. >- api [public] >- baseline // TO be created. >- bytecode >- cli >- cli-common >- configuration >- configuration-annotation-processor >- core // Module with classes like IgniteUuid. Should we raname it to >lang/utils/commons? >- metastorage-client // To be created. >- metastorage-common // To be created. >- metastorage-server // TO be created. >- network >- raft // raft-server? >- raft-client >- rest >- runner >- schema >- table // Seems that there might be a conflict between the meaning of >table module that we already have and table module with > create/dropTable() >-
[jira] [Created] (IGNITE-14442) IgniteRunner fails with NPE after REST module was broken by incompatible changes.
Ivan Bessonov created IGNITE-14442: -- Summary: IgniteRunner fails with NPE after REST module was broken by incompatible changes. Key: IGNITE-14442 URL: https://issues.apache.org/jira/browse/IGNITE-14442 Project: Ignite Issue Type: Sub-task Reporter: Ivan Bessonov Assignee: Ivan Bessonov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14441) Query plan printed for long running query must use IGNITE_TO_STRING_INCLUDE_SENSITIVE policy
Taras Ledkov created IGNITE-14441: - Summary: Query plan printed for long running query must use IGNITE_TO_STRING_INCLUDE_SENSITIVE policy Key: IGNITE-14441 URL: https://issues.apache.org/jira/browse/IGNITE-14441 Project: Ignite Issue Type: Bug Components: sql Reporter: Taras Ledkov Assignee: Taras Ledkov Fix For: 2.11 Query plan printed for long running query must use IGNITE_TO_STRING_INCLUDE_SENSITIVE policy. Now query parameters may be printed in the query plan. -- This message was sent by Atlassian Jira (v8.3.4#803005)
