Perfect ! Thanks ! I will try to be back on Cellar asap.
Regards JB On Mon, Oct 17, 2022 at 9:28 AM Ephemeris Lappis <[email protected]> wrote: > > Hello. > > Done : https://issues.apache.org/jira/browse/KARAF-7585 > > Ask me for any detail if needed. > > Thanks. > > Regards. > > Le ven. 14 oct. 2022 à 15:23, Jean-Baptiste Onofré <[email protected]> a > écrit : > > > > Hi Ephemeris, > > > > I suspect installing http causes a refresh on Hazelcast and/or Cellar > > bundles, probably because javax.servlet is an optional import. > > > > Do you mind creating a Jira about refresh ? I will improve that as well. > > > > Thanks ! > > Regards > > JB > > > > On Fri, Oct 14, 2022 at 9:27 AM Ephemeris Lappis > > <[email protected]> wrote: > > > > > > Hello JB ! > > > > > > From my last tests I've observed other Cellar's strange behaviors > > > during features deployment. > > > Using 2 Karaf nodes in docker containers, I've tested using two scripts > > > for : > > > - install Cellar on each instance > > > - install features using cluster commands on one of the instance > > > > > > The deployment using Jolokia instead of client CLI produces the same > > > behavior. > > > > > > Adding some sleeping time between commands, it works, but I've > > > observed that during the installation of some features (http for > > > example), Cellar and Hazelcast are restarted, unregistering cluster > > > commands for some seconds, and reconstructing the cluster after. With > > > the sleeping time, all seems to be ok on the two instances, and my own > > > features and bundles also work as expected. Nice ! > > > > > > Perhaps this could help you to analyze what could be improved during > > > your future works on Cellar. So I join my scripts (a linux shell and a > > > karaf shell), and the logs of the two instances where the changing > > > states of the cluster appear. Not sure you need my own features source > > > code, but I could also send them... > > > > > > I hope it helps you ;) > > > > > > Waiting for your feedback to go ahead on my tests. > > > > > > Thanks again. > > > > > > Regards. > > > > > > Le jeu. 13 oct. 2022 à 08:37, Jean-Baptiste Onofré <[email protected]> a > > > écrit : > > > > > > > > It should work if you just add delay (sleep) between jolokia MBean > > > > calls. > > > > > > > > Yes, I already have some refactoring in mind to improve Cellar (I > > > > shared on the dev mailing list while ago): > > > > > > > > https://issues.apache.org/jira/browse/KARAF-6187 > > > > https://issues.apache.org/jira/browse/KARAF-6168 > > > > https://issues.apache.org/jira/browse/KARAF-5969 > > > > https://issues.apache.org/jira/browse/KARAF-5140 > > > > https://issues.apache.org/jira/browse/KARAF-4195 > > > > https://issues.apache.org/jira/browse/KARAF-4194 > > > > > > > > As said, I'm busy with Karaf runtime right now, I won't be able to be > > > > back on Cellar before next week. > > > > > > > > Regards > > > > JB > > > > > > > > On Thu, Oct 13, 2022 at 7:10 AM Ephemeris Lappis > > > > <[email protected]> wrote: > > > > > > > > > > Hello > > > > > > > > > > So what I've observed is a known behavior. Is there any workaround to > > > > > allow chaining commands in a safe way ? Indeed we have to take some > > > > > orientations about our future tooling for setup, deployment and > > > > > maintenance of our features in Karaf clusters, and chaining command in > > > > > Jolokia requests was an attractive solution. > > > > > > > > > > I hope you can provide soon a safer cluster synchronization... I'll be > > > > > happy to help testing it. > > > > > > > > > > A last question about Camel (perhaps not the best list for it) : I've > > > > > tested the master component in routes with a file lock cluster > > > > > service, > > > > > since my tests rely on docker containers running on a same machine and > > > > > can use a shared volume that accepts locks. Is there a Camel Cluster > > > > > Service implementation that benefits of Cellar and Hazelcast (it > > > > > provides distributed named locks) that could be more suitable on > > > > > really > > > > > distributed cases with no shared storage ? > > > > > > > > > > Thanks again. > > > > > > > > > > Regards. > > > > > > > > > > Ephemeris Lappis > > > > > > > > > > Le 12/10/2022 à 18:39, Jean-Baptiste Onofré a écrit : > > > > > > Hi, > > > > > > > > > > > > I guess you are "hitting": > > > > > > > > > > > > https://issues.apache.org/jira/browse/KARAF-6168 > > > > > > > > > > > > Currently, the "cluster commands" (sent between nodes) are not > > > > > > "ordered". So, depending of the network, you don't have the > > > > > > guarantee > > > > > > that command "2" is executed after "1" on a node (some nodes can be > > > > > > ok, some not). > > > > > > > > > > > > When you execute the command step by step, and let the cluster sync, > > > > > > it's OK. But if you send a bunch of commands at same time, you might > > > > > > have "non ordered" events. > > > > > > > > > > > > I propose (in the Jira) to refactore the event mechanism at follow: > > > > > > 1. instead of sending the full state in the command (for example, > > > > > > currently Cellar send "install feature A on cluster group 1"), the > > > > > > command will just send "feature update" to all nodes. > > > > > > 2. then, each node will check the cluster group status on the > > > > > > distributed data gris (in Hazelcast) and compare its local state > > > > > > with > > > > > > the cluster state, and sync. > > > > > > > > > > > > I'm working on Karaf runtime releases right now, I will be back on > > > > > > Cellar during the week end. > > > > > > > > > > > > Regards > > > > > > JB > > > > > > > > > > > > On Wed, Oct 12, 2022 at 3:39 PM Ephemeris Lappis > > > > > > <[email protected]> wrote: > > > > > >> Hello again. > > > > > >> > > > > > >> I'm still testing Cellar. I've detected strange behaviours when > > > > > >> installing features. I've done the same tests with 2 or 3 nodes in > > > > > >> my > > > > > >> cluster, and using both ssh shell or jolokia to execute commands to > > > > > >> add repositories and install new features. > > > > > >> > > > > > >> When commands are played "slowly", waiting some time between them, > > > > > >> all > > > > > >> seems to be OK. > > > > > >> > > > > > >> But when commands are chained, for example in a single jolokia > > > > > >> post, > > > > > >> the cluster seems to execute them in an unexpected order. > > > > > >> > > > > > >> For example, the following jolokia post (the same happens with two > > > > > >> commands on the shell CLI), adding a repository, and installing a > > > > > >> new > > > > > >> feature is executed on the first node (the node I use as jolokia > > > > > >> target), but it seems that the "installFeature" is propagated to > > > > > >> the > > > > > >> second node before it receives or processes the first repository > > > > > >> command : this results with an error because the feature is > > > > > >> unknown. > > > > > >> > > > > > >> [ > > > > > >> { > > > > > >> "type" : "exec", > > > > > >> "mbean" : > > > > > >> "org.apache.karaf.cellar:name=root,type=feature", > > > > > >> "operation" : "addRepository(java.lang.String, > > > > > >> java.lang.String)", > > > > > >> "arguments":["default", > > > > > >> "mvn:my.tests/my-test-26-karaf-s-jms/0.0.1-SNAPSHOT/xml/features"] > > > > > >> }, > > > > > >> { > > > > > >> "type" : "exec", > > > > > >> "mbean" : > > > > > >> "org.apache.karaf.cellar:name=root,type=feature", > > > > > >> "operation" : "installFeature(java.lang.String, > > > > > >> java.lang.String)", > > > > > >> "arguments":["default", "my-test-26-karaf-s-jms"] > > > > > >> } > > > > > >> ] > > > > > >> > > > > > >> After this kind of error the failing node seems not to be able to > > > > > >> synchronize itself, adding again the feature does nothing, and > > > > > >> removing the repository with -u option also leads to errors... > > > > > >> > > > > > >> This is the same if I try to install "standard" features like > > > > > >> "http", > > > > > >> the "webconsole", and so on, in a same jolokia call... > > > > > >> > > > > > >> During some tests, after removing the features and repositories, > > > > > >> some > > > > > >> bundles are still present on some nodes. The only way to clean > > > > > >> them is > > > > > >> to create brand new Karaf containers and create the Cellar cluster > > > > > >> again. > > > > > >> > > > > > >> So, some questions... > > > > > >> > > > > > >> Is there some "time logic" for chaining commands on a Cellar > > > > > >> cluster ? > > > > > >> > > > > > >> Is it important to have the same IDs for bundles on all the nodes ? > > > > > >> When commands are executed one by one, and waiting some time, this > > > > > >> is > > > > > >> always the case, but when commands are in batches, problems seem to > > > > > >> appear with bundle IDs differences. > > > > > >> > > > > > >> What do people usually do to install big numbers of features on > > > > > >> Cellar > > > > > >> clusters, using dev-ops tooling for example ? > > > > > >> > > > > > >> Thanks for your help. > > > > > >> > > > > > >> Le ven. 7 oct. 2022 à 18:48, Jean-Baptiste Onofré > > > > > >> <[email protected]> a écrit : > > > > > >>> The node ID should be still the same. > > > > > >>> > > > > > >>> Let me try to reproduce the "alias lost". It sounds like a bug in > > > > > >>> alias cluster sync indeed. > > > > > >>> > > > > > >>> Regards > > > > > >>> JB > > > > > >>> > > > > > >>> On Fri, Oct 7, 2022 at 6:25 PM Ephemeris Lappis > > > > > >>> <[email protected]> wrote: > > > > > >>>> Hello. > > > > > >>>> > > > > > >>>> Nice ! I'm waiting for the new package ;)... > > > > > >>>> > > > > > >>>> I'm still testing on my docker environment. I've tried deploying > > > > > >>>> features using either the shell or jolokia, always with Cellar > > > > > >>>> commands or MBean. For now, no problem about the features, but I > > > > > >>>> had > > > > > >>>> twice a strange issue : I've set aliases on my 3 instances, and > > > > > >>>> after > > > > > >>>> a feature install, the 3 aliases have been lost. The same when > > > > > >>>> stopping the instances (compose stop in my case), and restarting > > > > > >>>> them. > > > > > >>>> Perhaps some kind of bug on aliases ? > > > > > >>>> > > > > > >>>> Thanks again. > > > > > >>>> > > > > > >>>> Regards. > > > > > >>>> > > > > > >>>> Le ven. 7 oct. 2022 à 08:24, Jean-Baptiste Onofré > > > > > >>>> <[email protected]> a écrit : > > > > > >>>>> Hi, > > > > > >>>>> > > > > > >>>>> Catcha, let me create the Jira and work on this ;) > > > > > >>>>> > > > > > >>>>> Thanks ! > > > > > >>>>> Regards > > > > > >>>>> JB > > > > > >>>>> > > > > > >>>>> On Thu, Oct 6, 2022 at 11:39 PM Ephemeris Lappis > > > > > >>>>> <[email protected]> wrote: > > > > > >>>>>> Hello ! > > > > > >>>>>> > > > > > >>>>>> Thanks a lot for your very detailed answer. > > > > > >>>>>> > > > > > >>>>>> After you explanations, I'm not sure that listeners are really > > > > > >>>>>> needed in > > > > > >>>>>> our case, but I'm going to enable them and test again with > > > > > >>>>>> basic > > > > > >>>>>> features/bundles commands. If we can use the Cellar's MBean > > > > > >>>>>> to script > > > > > >>>>>> our deployments playbooks with Jolokia calls, perhaps basic > > > > > >>>>>> synchronization is enough for us. > > > > > >>>>>> > > > > > >>>>>> For the last point, a Karaf+Cellar "off the shelf" tarball > > > > > >>>>>> would > > > > > >>>>>> obviously be a nice gift. I don't know if someone may use a > > > > > >>>>>> prebuilt > > > > > >>>>>> image : we usually make our own Docker images based on common > > > > > >>>>>> linux+java > > > > > >>>>>> stacks that are elaborated and managed by our DevOps team. > > > > > >>>>>> Anyway, > > > > > >>>>>> working examples of configuration to build custom Karaf > > > > > >>>>>> assemblies could > > > > > >>>>>> really help : the few examples I've found seem to build > > > > > >>>>>> limited features > > > > > >>>>>> distributions, enumerating known features, adding some custom > > > > > >>>>>> ones, but > > > > > >>>>>> probably missing others. An explained example with all Karaf > > > > > >>>>>> features > > > > > >>>>>> and the addition of Cellar should be interesting for > > > > > >>>>>> learning... > > > > > >>>>>> > > > > > >>>>>> So if you could provide both... very happy :) ! > > > > > >>>>>> > > > > > >>>>>> Thanks again. > > > > > >>>>>> > > > > > >>>>>> Ephemeris Lappis > > > > > >>>>>> > > > > > >>>>>> Le 06/10/2022 à 19:56, Jean-Baptiste Onofré a écrit : > > > > > >>>>>>> Hi, > > > > > >>>>>>> > > > > > >>>>>>> 1. By default, only cluster:* commands spread the state on > > > > > >>>>>>> the cluster > > > > > >>>>>>> 2. If you want the "regular" non cluster commands (like > > > > > >>>>>>> feature:install) spread also the state, you have to enable the > > > > > >>>>>>> listeners. The listeners are all disabled by default. You can > > > > > >>>>>>> enable > > > > > >>>>>>> them in etc/org.apache.karaf.cellar.node.cfg. You have one > > > > > >>>>>>> listener > > > > > >>>>>>> per resource: bundle.listener, config.listener, > > > > > >>>>>>> feature.listener. If > > > > > >>>>>>> you set true to all of them, you will have sync on regular > > > > > >>>>>>> command and > > > > > >>>>>>> even for local event (like changing cfg file for instance). > > > > > >>>>>>> It's > > > > > >>>>>>> documented here: > > > > > >>>>>>> https://karaf.apache.org/manual/cellar/latest-4/#_synchronizers_and_sync_policy > > > > > >>>>>>> 3. As jolokia is just a JMX client, and Cellar exposes > > > > > >>>>>>> MBeans, you can > > > > > >>>>>>> interact with cluster using jolokia > > > > > >>>>>>> 4. About the distribution, I should definitely provide a full > > > > > >>>>>>> example > > > > > >>>>>>> to create it and even push a karaf/cellar official distro and > > > > > >>>>>>> docker > > > > > >>>>>>> image. Thoughts ? > > > > > >>>>>>> > > > > > >>>>>>> Regards > > > > > >>>>>>> JB > > > > > >>>>>>> > > > > > >>>>>>> On Thu, Oct 6, 2022 at 5:58 PM Ephemeris Lappis > > > > > >>>>>>> <[email protected]> wrote: > > > > > >>>>>>>> Hello again ! > > > > > >>>>>>>> > > > > > >>>>>>>> I've been testing Cellar on a simple cluster of 3 karaf > > > > > >>>>>>>> instances > > > > > >>>>>>>> created with docker images and compose. > > > > > >>>>>>>> I've seen that cluster commands provide a synchronized > > > > > >>>>>>>> provisioning of > > > > > >>>>>>>> features, and for example, that stopped nodes > > > > > >>>>>>>> synchronization is done > > > > > >>>>>>>> when restarting. > > > > > >>>>>>>> This is clearly what we need :) ! > > > > > >>>>>>>> > > > > > >>>>>>>> I've also noticed that "non cluster" feature commands > > > > > >>>>>>>> (repo-add, > > > > > >>>>>>>> install, unistall) do not produce the synchronization. I > > > > > >>>>>>>> suppose > > > > > >>>>>>>> that's normal. So a new question : will it be possible to > > > > > >>>>>>>> use jolokia > > > > > >>>>>>>> to execute cluster commands the same way we do it with > > > > > >>>>>>>> default > > > > > >>>>>>>> features commands ? > > > > > >>>>>>>> > > > > > >>>>>>>> Now I'd like to go a step further before testing on real k8s > > > > > >>>>>>>> clusters. > > > > > >>>>>>>> > > > > > >>>>>>>> In your presentation you said that for now there's not a > > > > > >>>>>>>> downloadable > > > > > >>>>>>>> karaf distribution including Cellar, but that the best way > > > > > >>>>>>>> to deploy > > > > > >>>>>>>> clusters is generating such a custom distribution, and then > > > > > >>>>>>>> providing > > > > > >>>>>>>> a docker image with it. I've not found any example of the > > > > > >>>>>>>> plugin > > > > > >>>>>>>> configuration to generate a Karaf+Cellar distribution with > > > > > >>>>>>>> all the > > > > > >>>>>>>> default Karaf features and configurations, just adding > > > > > >>>>>>>> Cellar. > > > > > >>>>>>>> > > > > > >>>>>>>> Could you please provide any link to working examples ? This > > > > > >>>>>>>> could be > > > > > >>>>>>>> very nice and help a lot ;) !!! > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks again. > > > > > >>>>>>>> > > > > > >>>>>>>> Regards. > > > > > >>>>>>>> > > > > > >>>>>>>> Le mar. 4 oct. 2022 à 07:40, Jean-Baptiste Onofré > > > > > >>>>>>>> <[email protected]> a écrit : > > > > > >>>>>>>>> Yes, you can mix the approaches together. For instance, you > > > > > >>>>>>>>> can > > > > > >>>>>>>>> package in docker image: karaf runtime + cellar + your apps > > > > > >>>>>>>>> and then > > > > > >>>>>>>>> you mix Kubernetes with Cellar. It's the presentation I did > > > > > >>>>>>>>> while ago > > > > > >>>>>>>>> at ApacheCon. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Regards > > > > > >>>>>>>>> JB > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Tue, Oct 4, 2022 at 7:12 AM Ephemeris Lappis > > > > > >>>>>>>>> <[email protected]> wrote: > > > > > >>>>>>>>>> Hello. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Thanks for your explanations. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> I understand that your 3rd choice is the only one to get > > > > > >>>>>>>>>> multiple active > > > > > >>>>>>>>>> and synchronized instances. But can't I run them as PODs > > > > > >>>>>>>>>> inside a > > > > > >>>>>>>>>> Kubernetes namespace, using deployments of an image based > > > > > >>>>>>>>>> on > > > > > >>>>>>>>>> Karaf+Cellar, and then using the Jolokia API, for example, > > > > > >>>>>>>>>> to deploy and > > > > > >>>>>>>>>> update my applications as features, targeting any one of > > > > > >>>>>>>>>> the scaled > > > > > >>>>>>>>>> instances, and let Cellar synchronizing the other > > > > > >>>>>>>>>> instances ? > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> We already use Jolokia this way via Ansible playbooks to > > > > > >>>>>>>>>> deploy > > > > > >>>>>>>>>> applications, as profiles instead of features, on Fuse > > > > > >>>>>>>>>> clusters... > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Thanks again. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Regards. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Ephemeris Lappis > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Le 03/10/2022 à 18:36, Jean-Baptiste Onofré a écrit : > > > > > >>>>>>>>>>> Hi, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> In order: > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> 1. Karaf HA Lock: you have one active, other instances > > > > > >>>>>>>>>>> are passive > > > > > >>>>>>>>>>> 2. Kubernetes: you can orchestrate start/stop of the > > > > > >>>>>>>>>>> Karaf docker > > > > > >>>>>>>>>>> image, but Kubernetes doesn't sync Karaf instances state > > > > > >>>>>>>>>>> (like config, > > > > > >>>>>>>>>>> feature installed, etc) > > > > > >>>>>>>>>>> 3. Cellar: sync Karaf instances together (you install one > > > > > >>>>>>>>>>> feature on > > > > > >>>>>>>>>>> one Karaf instance, the feature will be installed on > > > > > >>>>>>>>>>> other Karaf > > > > > >>>>>>>>>>> instances in the cluster) > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Regards > > > > > >>>>>>>>>>> JB > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> On Mon, Oct 3, 2022 at 5:44 PM Ephemeris Lappis > > > > > >>>>>>>>>>> <[email protected]> wrote: > > > > > >>>>>>>>>>>> Hello. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> I've just looked at the presentation of Cellar. If I > > > > > >>>>>>>>>>>> understand it > > > > > >>>>>>>>>>>> well, this presentation says that Cellar's main goal is > > > > > >>>>>>>>>>>> for "big > > > > > >>>>>>>>>>>> clusters", allowing automatic synchronization between > > > > > >>>>>>>>>>>> Karaf instances. > > > > > >>>>>>>>>>>> It seems to be really nice in the presentation :) ! > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> On the other hand, the basic lock mechanism only > > > > > >>>>>>>>>>>> provides an > > > > > >>>>>>>>>>>> active/passive solution. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> What should I prefer if my need is to provide both > > > > > >>>>>>>>>>>> failover and load > > > > > >>>>>>>>>>>> balancing over a limited number of active instances, and > > > > > >>>>>>>>>>>> not a "big > > > > > >>>>>>>>>>>> cluster". Today we use 6 Fuse Karaf instances > > > > > >>>>>>>>>>>> distributed on 3 VM. Is > > > > > >>>>>>>>>>>> Cellar the right way, or did I miss something in the > > > > > >>>>>>>>>>>> presentation ? > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Another thing : for other kinds of applications my > > > > > >>>>>>>>>>>> customer manages > > > > > >>>>>>>>>>>> several Kubernetes clusters. So I suppose that if a > > > > > >>>>>>>>>>>> containerized > > > > > >>>>>>>>>>>> solution is preferred, it should be running on > > > > > >>>>>>>>>>>> Kubernetes, since all > > > > > >>>>>>>>>>>> the existing DevOps tooling is already based on it. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> The presentation focuses on Mesos/Marathon but also says > > > > > >>>>>>>>>>>> that > > > > > >>>>>>>>>>>> Kubernetes is also an alternative solution. Right ? In > > > > > >>>>>>>>>>>> this case, what > > > > > >>>>>>>>>>>> is the preferred way to package and deploy Karaf : just > > > > > >>>>>>>>>>>> create a > > > > > >>>>>>>>>>>> custom Karaf+Cellar image (the same way the presentation > > > > > >>>>>>>>>>>> shows), and > > > > > >>>>>>>>>>>> then create a Kubernetes deployment with the needed > > > > > >>>>>>>>>>>> sizing and scaled > > > > > >>>>>>>>>>>> replicas ? > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Some examples perhaps ? > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Thanks in advance for your help. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Regards. > > > > > >>>>>>>>>> -- > > > > > >>>>>>>>>> Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > > > > >>>>>>>>>> www.avast.com > > > > > >>>>>> -- > > > > > >>>>>> Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > > > > >>>>>> www.avast.com > > > > > > > > > > -- > > > > > Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > > > > www.avast.com
