Re: Node and cluster life-cycle in ignite-3
Alexei, In general I agree with the given proposal. However I'd also like to clarify a few things: 1. start() is called in "depends on" relation order. 2. It might have sense to add a runLevel parameter to start(): start(int runLevel). 3. Seems that we don't need an afterStart() method. Cause we already have component construction either with constructor or DI and component initialization with start(). Before component.start() all dependencies of a given component are already initialized, after component.start() the component is also fully initialized. 4. Each component has its own busyLock. 5. beforeStop() is called without busyLock.block(). Currently it has sense at most for Network and MetaStorage components where we prevent any new distributed communication by throwing a NodeStoppingException that will guarantee graceful termination on the sender side, including closing sender-scoped resources such as cursors and futures. 6. After components.foreach(beforeStop()) shutdown process gets a busyLock.block() on every component and stops it in "depends on" relation order. Best regards, Alexander чт, 3 июн. 2021 г. в 17:37, Alexei Scherbakov : > Sergey, I'm ok with the runlevel approach. > > I've thought about the node/components lifecycle, here my ideas: > > 1. The Component interface. > Each manageable component must implement it. > > 2. Define components hierarchy. > Each component can depend on others - this produces component hierarchy > defined by "depends on" relation. > For example, RaftGroup depends on a ClusterService to send messages to > other nodes. > > 3. Cyclic dependencies in the component hierarchy are forbidden. > > 4. Some form of dependency injection for easier component construction. > > 5. Transparent component lifecycle, defined by following methods: > > void start(); // Start a component > void afterStart(); // Called then all component dependencies are > initialized. Invoked in reverse to "depends on" relation order. > void beforeStop(); // Called before the component is going to stop (for > example, to cancel a pending operation) Invoked in reverse to "depends on" > relation order. > void stop(); // Stop a component. Invoked in "depends on" relation order. > In the example above, Ignite is stopped before the Network. > boolean isStopping(); // Flag to check if a node is stopping right now. > boolean runnable(int runLevel) // Defines if a component has to be started > on a specific run level. > > 6. Dynamic components (can be started/stopped at any time) > > 7. enterBusy/leaveBusy/block logic (similar to Ignite2) to avoid races on > node stopping. > > чт, 3 июн. 2021 г. в 13:08, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Hi Sergey, > > > > Sounds interesting, I do agree that it might be beneficial to improve the > > lifecycle management in 3.0 - 2.x version is far from perfect. > > > > Regarding your questions: > > > > 1. Can this be done via the metastore? > > 2. I think we should list the run levels that we think should be there, > and > > then it will be easier to identify dependencies between them. Can you > give > > an example of independent run levels? > > > > -Val > > > > On Tue, Jun 1, 2021 at 7:57 AM Sergey Chugunov < > sergey.chugu...@gmail.com> > > wrote: > > > > > Hello Igniters, > > > > > > I would like to start a discussion on evolving IEP-73 [1]. Now it > covers > > a > > > narrow topic about components dependencies but it makes sense to cover > in > > > the IEP a broader question: how different components should be > > initialized > > > to support different modes of an individual node or a whole cluster. > > > > > > There is an idea to borrow the notion of run-levels from Unix-like > > systems, > > > and I suggest the following design to implement it. > > > > > >1. To start and function at a specific run-level node needs to start > > and > > >initialize components in a proper order. During initialization > > > components > > >may need to notify each other about reaching a particular run-level > so > > >other components are able to execute their actions. Orchestrating of > > > this > > >process should be a responsibility of a new component. > > > > > >2. Orchestration component doesn't manage the initialization process > > >directly but uses another abstraction called scenario. Examples of > > >run-levels in the context of Ignite 2.x may be Maintenance Mode, > > >INACTIVE-READONLY-ACTIVE states of a cluster, and each level is > > reached > > >when a corresponding scenario has executed. > > > > > >So the responsibility of the orchestrator will be managing scenarios > > and > > >providing them with infrastructure of spreading notification events > > > between > > >components. All low-level details and knowledge of existing > components > > > and > > >their dependencies are encapsulated inside scenarios. > > > > > >3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster > state >
Re: Node and cluster life-cycle in ignite-3
I've made a small mistake above, the correct sentence is void stop(); // Stop a component. Invoked in "depends on" relation order. In the example above, RaftGroup is stopped before the Network. чт, 3 июн. 2021 г. в 17:36, Alexei Scherbakov : > Sergey, I'm ok with the runlevel approach. > > I've thought about the node/components lifecycle, here my ideas: > > 1. The Component interface. > Each manageable component must implement it. > > 2. Define components hierarchy. > Each component can depend on others - this produces component hierarchy > defined by "depends on" relation. > For example, RaftGroup depends on a ClusterService to send messages to > other nodes. > > 3. Cyclic dependencies in the component hierarchy are forbidden. > > 4. Some form of dependency injection for easier component construction. > > 5. Transparent component lifecycle, defined by following methods: > > void start(); // Start a component > void afterStart(); // Called then all component dependencies are > initialized. Invoked in reverse to "depends on" relation order. > void beforeStop(); // Called before the component is going to stop (for > example, to cancel a pending operation) Invoked in reverse to "depends on" > relation order. > void stop(); // Stop a component. Invoked in "depends on" relation order. > In the example above, Ignite is stopped before the Network. > boolean isStopping(); // Flag to check if a node is stopping right now. > boolean runnable(int runLevel) // Defines if a component has to be started > on a specific run level. > > 6. Dynamic components (can be started/stopped at any time) > > 7. enterBusy/leaveBusy/block logic (similar to Ignite2) to avoid races on > node stopping. > > чт, 3 июн. 2021 г. в 13:08, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > >> Hi Sergey, >> >> Sounds interesting, I do agree that it might be beneficial to improve the >> lifecycle management in 3.0 - 2.x version is far from perfect. >> >> Regarding your questions: >> >> 1. Can this be done via the metastore? >> 2. I think we should list the run levels that we think should be there, >> and >> then it will be easier to identify dependencies between them. Can you give >> an example of independent run levels? >> >> -Val >> >> On Tue, Jun 1, 2021 at 7:57 AM Sergey Chugunov > > >> wrote: >> >> > Hello Igniters, >> > >> > I would like to start a discussion on evolving IEP-73 [1]. Now it >> covers a >> > narrow topic about components dependencies but it makes sense to cover >> in >> > the IEP a broader question: how different components should be >> initialized >> > to support different modes of an individual node or a whole cluster. >> > >> > There is an idea to borrow the notion of run-levels from Unix-like >> systems, >> > and I suggest the following design to implement it. >> > >> >1. To start and function at a specific run-level node needs to start >> and >> >initialize components in a proper order. During initialization >> > components >> >may need to notify each other about reaching a particular run-level >> so >> >other components are able to execute their actions. Orchestrating of >> > this >> >process should be a responsibility of a new component. >> > >> >2. Orchestration component doesn't manage the initialization process >> >directly but uses another abstraction called scenario. Examples of >> >run-levels in the context of Ignite 2.x may be Maintenance Mode, >> >INACTIVE-READONLY-ACTIVE states of a cluster, and each level is >> reached >> >when a corresponding scenario has executed. >> > >> >So the responsibility of the orchestrator will be managing scenarios >> and >> >providing them with infrastructure of spreading notification events >> > between >> >components. All low-level details and knowledge of existing >> components >> > and >> >their dependencies are encapsulated inside scenarios. >> > >> >3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster >> state >> >can be "upgraded" to READONLY state by executing diff between >> INACTIVE >> > and >> >READONLY scenarios. >> > >> > >> > I see several advantages of this design compared to existing model in >> > Ignite 2.x (mostly implemented in IgniteKernal and based on two main >> > methods: start and onKernalStart): >> > >> >1. More flexible model allows implementing more diverse run-levels >> for >> >different needs (already mentioned Maintenance Mode, cluster state >> modes >> >like ACTIVE-INACTIVE and smart strategies for cache warmup on node >> > start). >> > >> >2. Knowledge of components and their dependencies is encapsulated >> inside >> >scenarios which makes it easier to create new scenarios. >> > >> > >> > Open questions: >> > >> >1. As I see right now it is hard to standardize initialization events >> >components notify each other with. >> > >> >2. It is not clear if run-levels should be organized into one rigid >> >hierarchy (when the
Re: Node and cluster life-cycle in ignite-3
Sergey, I'm ok with the runlevel approach. I've thought about the node/components lifecycle, here my ideas: 1. The Component interface. Each manageable component must implement it. 2. Define components hierarchy. Each component can depend on others - this produces component hierarchy defined by "depends on" relation. For example, RaftGroup depends on a ClusterService to send messages to other nodes. 3. Cyclic dependencies in the component hierarchy are forbidden. 4. Some form of dependency injection for easier component construction. 5. Transparent component lifecycle, defined by following methods: void start(); // Start a component void afterStart(); // Called then all component dependencies are initialized. Invoked in reverse to "depends on" relation order. void beforeStop(); // Called before the component is going to stop (for example, to cancel a pending operation) Invoked in reverse to "depends on" relation order. void stop(); // Stop a component. Invoked in "depends on" relation order. In the example above, Ignite is stopped before the Network. boolean isStopping(); // Flag to check if a node is stopping right now. boolean runnable(int runLevel) // Defines if a component has to be started on a specific run level. 6. Dynamic components (can be started/stopped at any time) 7. enterBusy/leaveBusy/block logic (similar to Ignite2) to avoid races on node stopping. чт, 3 июн. 2021 г. в 13:08, Valentin Kulichenko < valentin.kuliche...@gmail.com>: > Hi Sergey, > > Sounds interesting, I do agree that it might be beneficial to improve the > lifecycle management in 3.0 - 2.x version is far from perfect. > > Regarding your questions: > > 1. Can this be done via the metastore? > 2. I think we should list the run levels that we think should be there, and > then it will be easier to identify dependencies between them. Can you give > an example of independent run levels? > > -Val > > On Tue, Jun 1, 2021 at 7:57 AM Sergey Chugunov > wrote: > > > Hello Igniters, > > > > I would like to start a discussion on evolving IEP-73 [1]. Now it covers > a > > narrow topic about components dependencies but it makes sense to cover in > > the IEP a broader question: how different components should be > initialized > > to support different modes of an individual node or a whole cluster. > > > > There is an idea to borrow the notion of run-levels from Unix-like > systems, > > and I suggest the following design to implement it. > > > >1. To start and function at a specific run-level node needs to start > and > >initialize components in a proper order. During initialization > > components > >may need to notify each other about reaching a particular run-level so > >other components are able to execute their actions. Orchestrating of > > this > >process should be a responsibility of a new component. > > > >2. Orchestration component doesn't manage the initialization process > >directly but uses another abstraction called scenario. Examples of > >run-levels in the context of Ignite 2.x may be Maintenance Mode, > >INACTIVE-READONLY-ACTIVE states of a cluster, and each level is > reached > >when a corresponding scenario has executed. > > > >So the responsibility of the orchestrator will be managing scenarios > and > >providing them with infrastructure of spreading notification events > > between > >components. All low-level details and knowledge of existing components > > and > >their dependencies are encapsulated inside scenarios. > > > >3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster state > >can be "upgraded" to READONLY state by executing diff between INACTIVE > > and > >READONLY scenarios. > > > > > > I see several advantages of this design compared to existing model in > > Ignite 2.x (mostly implemented in IgniteKernal and based on two main > > methods: start and onKernalStart): > > > >1. More flexible model allows implementing more diverse run-levels for > >different needs (already mentioned Maintenance Mode, cluster state > modes > >like ACTIVE-INACTIVE and smart strategies for cache warmup on node > > start). > > > >2. Knowledge of components and their dependencies is encapsulated > inside > >scenarios which makes it easier to create new scenarios. > > > > > > Open questions: > > > >1. As I see right now it is hard to standardize initialization events > >components notify each other with. > > > >2. It is not clear if run-levels should be organized into one rigid > >hierarchy (when the first run-level should always precede the second > > and so > >on) or they should be more independent. > > > > > > What do you think? > > > > [1] > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-73%3A+Node+startup > > > -- Best regards, Alexei Scherbakov
Re: Node and cluster life-cycle in ignite-3
Hi Sergey, Sounds interesting, I do agree that it might be beneficial to improve the lifecycle management in 3.0 - 2.x version is far from perfect. Regarding your questions: 1. Can this be done via the metastore? 2. I think we should list the run levels that we think should be there, and then it will be easier to identify dependencies between them. Can you give an example of independent run levels? -Val On Tue, Jun 1, 2021 at 7:57 AM Sergey Chugunov wrote: > Hello Igniters, > > I would like to start a discussion on evolving IEP-73 [1]. Now it covers a > narrow topic about components dependencies but it makes sense to cover in > the IEP a broader question: how different components should be initialized > to support different modes of an individual node or a whole cluster. > > There is an idea to borrow the notion of run-levels from Unix-like systems, > and I suggest the following design to implement it. > >1. To start and function at a specific run-level node needs to start and >initialize components in a proper order. During initialization > components >may need to notify each other about reaching a particular run-level so >other components are able to execute their actions. Orchestrating of > this >process should be a responsibility of a new component. > >2. Orchestration component doesn't manage the initialization process >directly but uses another abstraction called scenario. Examples of >run-levels in the context of Ignite 2.x may be Maintenance Mode, >INACTIVE-READONLY-ACTIVE states of a cluster, and each level is reached >when a corresponding scenario has executed. > >So the responsibility of the orchestrator will be managing scenarios and >providing them with infrastructure of spreading notification events > between >components. All low-level details and knowledge of existing components > and >their dependencies are encapsulated inside scenarios. > >3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster state >can be "upgraded" to READONLY state by executing diff between INACTIVE > and >READONLY scenarios. > > > I see several advantages of this design compared to existing model in > Ignite 2.x (mostly implemented in IgniteKernal and based on two main > methods: start and onKernalStart): > >1. More flexible model allows implementing more diverse run-levels for >different needs (already mentioned Maintenance Mode, cluster state modes >like ACTIVE-INACTIVE and smart strategies for cache warmup on node > start). > >2. Knowledge of components and their dependencies is encapsulated inside >scenarios which makes it easier to create new scenarios. > > > Open questions: > >1. As I see right now it is hard to standardize initialization events >components notify each other with. > >2. It is not clear if run-levels should be organized into one rigid >hierarchy (when the first run-level should always precede the second > and so >on) or they should be more independent. > > > What do you think? > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-73%3A+Node+startup >
Node and cluster life-cycle in ignite-3
Hello Igniters, I would like to start a discussion on evolving IEP-73 [1]. Now it covers a narrow topic about components dependencies but it makes sense to cover in the IEP a broader question: how different components should be initialized to support different modes of an individual node or a whole cluster. There is an idea to borrow the notion of run-levels from Unix-like systems, and I suggest the following design to implement it. 1. To start and function at a specific run-level node needs to start and initialize components in a proper order. During initialization components may need to notify each other about reaching a particular run-level so other components are able to execute their actions. Orchestrating of this process should be a responsibility of a new component. 2. Orchestration component doesn't manage the initialization process directly but uses another abstraction called scenario. Examples of run-levels in the context of Ignite 2.x may be Maintenance Mode, INACTIVE-READONLY-ACTIVE states of a cluster, and each level is reached when a corresponding scenario has executed. So the responsibility of the orchestrator will be managing scenarios and providing them with infrastructure of spreading notification events between components. All low-level details and knowledge of existing components and their dependencies are encapsulated inside scenarios. 3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster state can be "upgraded" to READONLY state by executing diff between INACTIVE and READONLY scenarios. I see several advantages of this design compared to existing model in Ignite 2.x (mostly implemented in IgniteKernal and based on two main methods: start and onKernalStart): 1. More flexible model allows implementing more diverse run-levels for different needs (already mentioned Maintenance Mode, cluster state modes like ACTIVE-INACTIVE and smart strategies for cache warmup on node start). 2. Knowledge of components and their dependencies is encapsulated inside scenarios which makes it easier to create new scenarios. Open questions: 1. As I see right now it is hard to standardize initialization events components notify each other with. 2. It is not clear if run-levels should be organized into one rigid hierarchy (when the first run-level should always precede the second and so on) or they should be more independent. What do you think? [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-73%3A+Node+startup
