Sergey, > From the code complexity perspective I'm trying to design the feature in > such a way that all maintenance code is as encapsulated as possible and > avoids massive interventions into main workflows of components.
Could please briefly tell what means do you use to achieve encapsulation? Are Discovery, Communication, Checkpointer and other components started in a maintenance mode in current design? 2020-09-21 15:19 GMT+03:00, Nikolay Izhikov <nizhi...@apache.org>: > Hello, Sergey. > >> At the moment I'm aware about two use cases for this feature: corrupted >> PDS cleanup and defragmentation. > > AFAIKU There is third use-case for this mode. > > Change encryption master key in case node was down during cluster master key > change. > In this case, node can’t join to the cluster, because it’s master key > differs from the cluster. > To recover node Ignite should locally change master key before join. > > Please, take a look into source code [1] > > [1] > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java#L710 > >> 21 сент. 2020 г., в 14:37, Sergey Chugunov <sergey.chugu...@gmail.com> >> написал(а): >> >> Ivan, >> >> Sorry for some confusion, MM indeed is not a normal mode. What I was >> trying >> to say is that when in MM node still starts and allows the user to >> perform >> actions with it like sending commands via control utility/JMX APIs or >> reading metrics. >> >> This is the key point: although the node is not in the cluster but it is >> still alive can be monitored and supports management to do maintenance. >> >> From the code complexity perspective I'm trying to design the feature in >> such a way that all maintenance code is as encapsulated as possible and >> avoids massive interventions into main workflows of components. >> At the moment I'm aware about two use cases for this feature: corrupted >> PDS >> cleanup and defragmentation. As far as I know it won't bring too much >> complexity in both cases. >> >> I cannot say for other components but I believe it will be possible to >> integrate MM feature into their workflow as well with reasonable amount >> of >> refactoring. >> >> Does it make sense to you? >> >> On Sun, Sep 6, 2020 at 8:08 AM Ivan Pavlukhin <vololo...@gmail.com> >> wrote: >> >>> Sergey, >>> >>> Thank you for your answer! >>> >>> Might be I am looking at the subject from a different angle. >>> >>>> I think of a node in MM as an almost normal one >>> I cannot think of such a mode as a normal one, because it apparently >>> does not perform usual cluster node functions. It is not a part of a >>> cluster, caches data is not available, Discovery and Communication are >>> not needed. >>> >>> I fear that with "node started in a special mode" approach we will get >>> an additional flag in the code making the code more complex and >>> fragile. Should not I worry about it? >>> >>> 2020-09-02 10:45 GMT+03:00, Sergey Chugunov <sergey.chugu...@gmail.com>: >>>> Vladislav, Ivan, >>>> >>>> Thank you for your questions and suggestions. Let me answer them. >>>> >>>> Vladislav, >>>> >>>> If I understood you correctly, you're talking about a node performing >>> some >>>> automatic actions to fix the problem and then join the cluster as >>>> usual. >>>> >>>> However the original ticket [1] where we faced the need for Maintenance >>>> Mode is about exactly the opposite: avoid doing automatic actions and >>> give >>>> a user the ability to decide what to do. >>>> >>>> Also the idea of Maintenance Mode is that the node is able to accept >>>> commands, expose metrics and so on, thus we need all components to be >>>> initialized (some of them may be partially initialized due to their own >>>> maintenance). >>>> To achieve that we need to go through a full cycle of node >>>> initialization >>>> including discovery initialization. When discovery is initialized (in >>>> special isolated mode) I don't think it is easy to switch back to >>>> normal >>>> operations without a restart. >>>> >>>> Ivan, >>>> >>>> I think of a node in MM as an almost normal one (maybe with some >>> components >>>> skipped some steps of their initialization). Commands are accepted, >>>> appropriate metrics are exposed e.g. through JMX API and so on. >>>> >>>> So as I see it we'll have special commands for control.{sh|bat} CLI >>>> allowing user to see reasons why node switched to maintenance mode >>>> and/or >>>> trigger actions to fix the problem (I'm still thinking about proper >>> design >>>> of these actions though). >>>> >>>> Of course the user should also be able to fix the problem manually e.g. >>> by >>>> manually deleting corrupted PDS files when node is down. Ideally >>>> Maintenance Mode should be smart enough to figure that out and switch >>>> to >>>> normal operations without a restart but I'm not sure if it is possible >>>> without invasive changes of our components' lifecycle. >>>> So I believe this model (node truly started in Maintenance Mode and new >>>> commands in control.{sh|bat}) is a good fit for our current APIs and >>>> ways >>>> to interact with the node. >>>> >>>> Does it sound reasonable to you? >>>> >>>> Thank you! >>>> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-13366 >>>> >>>> On Tue, Sep 1, 2020 at 2:07 PM Ivan Pavlukhin <vololo...@gmail.com> >>> wrote: >>>> >>>>> Sergey, >>>>> >>>>> Actually, I missed the point that the discussed mode affects a single >>>>> node but not a whole cluster. Perhaps I mixed terms "mode" and >>>>> "state". >>>>> >>>>> My next thoughts about maintenance routines are about special >>>>> utilities. As far as I remember MySQL provides a bunch of scripts for >>>>> various maintenance purposes. What user interface for maintenance >>>>> tasks execution is assumed? And what do we mean by "starting" a node >>>>> in a maintenance mode? Can we do some routines without "starting" >>>>> (e.g. try to recover PDS or cleanup)? >>>>> >>>>> 2020-08-31 23:41 GMT+03:00, Vladislav Pyatkov <vldpyat...@gmail.com>: >>>>>> Hi Sergey. >>>>>> >>>>>> As I understand any switching from/to MM possible only through manual >>>>>> restart a node. >>>>>> But in your example that look like a technical actions, that only >>>>> possible >>>>>> in the case. >>>>>> Do you plan to provide a possibility for client where he can make a >>>>>> decision without a manual intervention? >>>>>> >>>>>> For example: Start node and manually agree with an option and after >>>>>> automatically resolve conflict and back to topology as a stable node. >>>>>> >>>>>> On Mon, Aug 31, 2020 at 5:41 PM Sergey Chugunov < >>>>> sergey.chugu...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hello Ivan, >>>>>>> >>>>>>> Thank you for raising the good question, I didn't think of >>> Maintenance >>>>>>> Mode >>>>>>> from that perspective. >>>>>>> >>>>>>> In short, Maintenance Mode isn't related to Cluster States concept. >>>>>>> According to javadoc documentation of ClusterState enum [1] it is >>>>>>> solely >>>>>>> about cache operations and to some extent doesn't affect other >>>>> components >>>>>>> of Ignite node. >>>>>>> From APIs perspective putting the methods to manage Cluster State to >>>>>>> IgniteCluster interface doesn't look ideal to me but it is as it is. >>>>>>> >>>>>>> On the other hand Maintenance Mode as I see it will be managed >>> through >>>>>>> different APIs than a ClusterState and this difference definitely >>> will >>>>> be >>>>>>> reflected in the documentation of the feature. >>>>>>> >>>>>>> Ignite node is a complex piece of many components interacting with >>>>>>> each >>>>>>> other, they may have different lifecycles and states; states of >>>>> different >>>>>>> components cannot be reduced to the lowest common denominator. >>>>>>> >>>>>>> However if you have an idea of how to call the feature better to let >>>>>>> the >>>>>>> user easier distinguish it from other similar features please share >>> it >>>>>>> with >>>>>>> us. Personally I'm very welcome to any suggestions that make design >>>>>>> more >>>>>>> intuitive and easy-to-use. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> [1] >>>>>>> >>>>>>> >>>>> >>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cluster/ClusterState.java >>>>>>> >>>>>>> On Mon, Aug 31, 2020 at 12:32 PM Ivan Pavlukhin <vololo...@gmail.com >>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Sergey, >>>>>>>> >>>>>>>> Thank you for bringing attention to that important subject! >>>>>>>> >>>>>>>> My note here is about one more cluster mode. As far as I know >>>>>>>> currently we already have 3 modes (inactive, read-only, read-write) >>>>>>>> and the subject is about one more. From the first glance it could >>> be >>>>>>>> hard for a user to understand and use all modes properly. Do we >>>>>>>> really >>>>>>>> need all spectrum? Could we simplify things somehow? >>>>>>>> >>>>>>>> 2020-08-27 15:59 GMT+03:00, Sergey Chugunov >>>>>>>> <sergey.chugu...@gmail.com>: >>>>>>>>> Hello Nikolay, >>>>>>>>> >>>>>>>>> Created one, available by link [1] >>>>>>>>> >>>>>>>>> Initially there was an intention to develop it under IEP-47 [2] >>>>>>>>> and >>>>>>> there >>>>>>>>> is even a separate section for Maintenance Mode there. >>>>>>>>> But it looks like this feature is useful in more cases and >>>>>>>>> deserves >>>>>>>>> its >>>>>>>> own >>>>>>>>> IEP. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-53%3A+Maintenance+Mode >>>>>>>>> [2] >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47:+Native+persistence+defragmentation >>>>>>>>> >>>>>>>>> On Thu, Aug 27, 2020 at 11:01 AM Nikolay Izhikov >>>>>>>>> <nizhi...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello, Sergey! >>>>>>>>>> >>>>>>>>>> Thanks for the proposal. >>>>>>>>>> Let’s have IEP for this feature. >>>>>>>>>> >>>>>>>>>>> 27 авг. 2020 г., в 10:25, Sergey Chugunov < >>>>>>> sergey.chugu...@gmail.com> >>>>>>>>>> написал(а): >>>>>>>>>>> >>>>>>>>>>> Hello Igniters, >>>>>>>>>>> >>>>>>>>>>> I want to start a discussion about new supporting feature that >>>>>>>>>>> could >>>>>>>> be >>>>>>>>>>> very useful in many scenarios where persistent storage is >>>>>>>>>>> involved: >>>>>>>>>>> Maintenance Mode. >>>>>>>>>>> >>>>>>>>>>> *Summary* >>>>>>>>>>> Maintenance Mode (MM for short) is a special state of Ignite >>>>>>>>>>> node >>>>>>> when >>>>>>>>>> node >>>>>>>>>>> doesn't serve user requests nor joins the cluster but waits >>> for >>>>>>>>>>> user >>>>>>>>>>> commands or performs automatic actions for maintenance >>>>>>>>>>> purposes. >>>>>>>>>>> >>>>>>>>>>> *Motivation* >>>>>>>>>>> There are situations when node cannot participate in regular >>>>>>>> operations >>>>>>>>>> but >>>>>>>>>>> at the same time should not be shut down. >>>>>>>>>>> >>>>>>>>>>> One example is a ticket [1] where I developed the first draft >>>>>>>>>>> of >>>>>>>>>>> Maintenance Mode. >>>>>>>>>>> Here we get into a situation when node has potentially >>>>>>>>>>> corrupted >>>>>>>>>>> PDS >>>>>>>>>>> thus >>>>>>>>>>> cannot proceed with restore routine and join the cluster as >>>>> usual. >>>>>>>>>>> At the same time node should not fail nor be stopped for >>> manual >>>>>>>>>>> cleanup. >>>>>>>>>>> Manual cleanup is not always an option (e.g. restricted access >>>>>>>>>>> to >>>>>>> file >>>>>>>>>>> system); in managed environments failed node will be restarted >>>>>>>>>>> automatically so user won't have time for performing necessary >>>>>>>>>> operations. >>>>>>>>>>> Thus node needs to function in a special mode allowing user to >>>>>>> connect >>>>>>>>>>> to >>>>>>>>>>> it and perform necessary actions. >>>>>>>>>>> >>>>>>>>>>> Another example is described in IEP-47 [2] where >>>>>>>>>>> defragmentation >>>>>>>>>>> is >>>>>>>>>>> being >>>>>>>>>>> developed. Node defragmenting its PDS should not join the >>>>>>>>>>> cluster >>>>>>>> until >>>>>>>>>> the >>>>>>>>>>> process is finished so it needs to enter Maintenance Mode as >>>>> well. >>>>>>>>>>> >>>>>>>>>>> *Suggested design* >>>>>>>>>>> I suggest MM to work as follows: >>>>>>>>>>> 1. Node enters MM if special markers are found on disk. These >>>>>>> markers >>>>>>>>>>> called Maintenance Records could be created automatically >>> (e.g. >>>>>>>>>>> when >>>>>>>>>>> storage component detects corrupted storage) or by user >>> request >>>>>>> (when >>>>>>>>>> user >>>>>>>>>>> requests defragmentation of some caches). So entering MM >>>>>>>>>>> requires >>>>>>> node >>>>>>>>>>> restart. >>>>>>>>>>> 2. Started in MM node doesn't join the cluster but finishes >>>>>>>>>>> startup >>>>>>>>>> routine >>>>>>>>>>> so it is able to receive commands and provide metrics to the >>>>> user. >>>>>>>>>>> 3. When all necessary maintenance operations are finished, >>>>>>> Maintenance >>>>>>>>>>> Records for these operations are deleted from disk and node >>>>>>> restarted >>>>>>>>>> again >>>>>>>>>>> to enter normal service. >>>>>>>>>>> >>>>>>>>>>> *Example* >>>>>>>>>>> To put it into a context let's consider an example of how I >>> see >>>>>>>>>>> the >>>>>>> MM >>>>>>>>>>> workflow in case of PDS corruption. >>>>>>>>>>> >>>>>>>>>>> 1. Node has failed in the middle of checkpoint when WAL is >>>>>>> disabled >>>>>>>>>>> for >>>>>>>>>>> a particular cache -> data files of the cache are >>> potentially >>>>>>>>>> corrupted. >>>>>>>>>>> 2. On next startup node detects this situation, creates >>>>>>> Maintenance >>>>>>>>>>> Record on disk and shuts down. >>>>>>>>>>> 3. On next startup node sees Maintenance Record, enters >>>>>>> Maintenance >>>>>>>>>> Mode >>>>>>>>>>> and waits for user to do specific actions: clean potentially >>>>>>>>>>> corrupted >>>>>>>>>> PDS. >>>>>>>>>>> 4. When user has done necessary actions he/she removes >>>>>>>>>>> Maintenance >>>>>>>>>>> Record using Maintenance Mode API exposed via >>>>>>>>>>> control.{sh|bat} >>>>>>>> script >>>>>>>>>> or >>>>>>>>>>> JMX. >>>>>>>>>>> 5. On next startup node goes to normal operations as >>>>> maintenance >>>>>>>>>>> reason >>>>>>>>>>> is fixed. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I prepared a PR [3] for ticket [1] with draft implementation. >>>>>>>>>>> It >>>>>>>>>>> is >>>>>>>> not >>>>>>>>>>> ready to be merged to master branch but is already fully >>>>>>>>>>> functional >>>>>>>> and >>>>>>>>>> can >>>>>>>>>>> be reviewed. >>>>>>>>>>> >>>>>>>>>>> Hope you'll share your feedback on the feature and/or any >>>>> thoughts >>>>>>> on >>>>>>>>>>> implementation. >>>>>>>>>>> >>>>>>>>>>> Thank you! >>>>>>>>>>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13366 >>>>>>>>>>> [2] >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47:+Native+persistence+defragmentation >>>>>>>>>>> [3] https://github.com/apache/ignite/pull/8189 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Ivan Pavlukhin >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Vladislav Pyatkov >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Best regards, >>>>> Ivan Pavlukhin >>>>> >>>> >>> >>> >>> -- >>> >>> Best regards, >>> Ivan Pavlukhin >>> > > -- Best regards, Ivan Pavlukhin
