Hello JB ! >From my last tests I've observed other Cellar's strange behaviors during features deployment. Using 2 Karaf nodes in docker containers, I've tested using two scripts for : - install Cellar on each instance - install features using cluster commands on one of the instance
The deployment using Jolokia instead of client CLI produces the same behavior. Adding some sleeping time between commands, it works, but I've observed that during the installation of some features (http for example), Cellar and Hazelcast are restarted, unregistering cluster commands for some seconds, and reconstructing the cluster after. With the sleeping time, all seems to be ok on the two instances, and my own features and bundles also work as expected. Nice ! Perhaps this could help you to analyze what could be improved during your future works on Cellar. So I join my scripts (a linux shell and a karaf shell), and the logs of the two instances where the changing states of the cluster appear. Not sure you need my own features source code, but I could also send them... I hope it helps you ;) Waiting for your feedback to go ahead on my tests. Thanks again. Regards. Le jeu. 13 oct. 2022 à 08:37, Jean-Baptiste Onofré <[email protected]> a écrit : > > It should work if you just add delay (sleep) between jolokia MBean calls. > > Yes, I already have some refactoring in mind to improve Cellar (I > shared on the dev mailing list while ago): > > https://issues.apache.org/jira/browse/KARAF-6187 > https://issues.apache.org/jira/browse/KARAF-6168 > https://issues.apache.org/jira/browse/KARAF-5969 > https://issues.apache.org/jira/browse/KARAF-5140 > https://issues.apache.org/jira/browse/KARAF-4195 > https://issues.apache.org/jira/browse/KARAF-4194 > > As said, I'm busy with Karaf runtime right now, I won't be able to be > back on Cellar before next week. > > Regards > JB > > On Thu, Oct 13, 2022 at 7:10 AM Ephemeris Lappis > <[email protected]> wrote: > > > > Hello > > > > So what I've observed is a known behavior. Is there any workaround to > > allow chaining commands in a safe way ? Indeed we have to take some > > orientations about our future tooling for setup, deployment and > > maintenance of our features in Karaf clusters, and chaining command in > > Jolokia requests was an attractive solution. > > > > I hope you can provide soon a safer cluster synchronization... I'll be > > happy to help testing it. > > > > A last question about Camel (perhaps not the best list for it) : I've > > tested the master component in routes with a file lock cluster service, > > since my tests rely on docker containers running on a same machine and > > can use a shared volume that accepts locks. Is there a Camel Cluster > > Service implementation that benefits of Cellar and Hazelcast (it > > provides distributed named locks) that could be more suitable on really > > distributed cases with no shared storage ? > > > > Thanks again. > > > > Regards. > > > > Ephemeris Lappis > > > > Le 12/10/2022 à 18:39, Jean-Baptiste Onofré a écrit : > > > Hi, > > > > > > I guess you are "hitting": > > > > > > https://issues.apache.org/jira/browse/KARAF-6168 > > > > > > Currently, the "cluster commands" (sent between nodes) are not > > > "ordered". So, depending of the network, you don't have the guarantee > > > that command "2" is executed after "1" on a node (some nodes can be > > > ok, some not). > > > > > > When you execute the command step by step, and let the cluster sync, > > > it's OK. But if you send a bunch of commands at same time, you might > > > have "non ordered" events. > > > > > > I propose (in the Jira) to refactore the event mechanism at follow: > > > 1. instead of sending the full state in the command (for example, > > > currently Cellar send "install feature A on cluster group 1"), the > > > command will just send "feature update" to all nodes. > > > 2. then, each node will check the cluster group status on the > > > distributed data gris (in Hazelcast) and compare its local state with > > > the cluster state, and sync. > > > > > > I'm working on Karaf runtime releases right now, I will be back on > > > Cellar during the week end. > > > > > > Regards > > > JB > > > > > > On Wed, Oct 12, 2022 at 3:39 PM Ephemeris Lappis > > > <[email protected]> wrote: > > >> Hello again. > > >> > > >> I'm still testing Cellar. I've detected strange behaviours when > > >> installing features. I've done the same tests with 2 or 3 nodes in my > > >> cluster, and using both ssh shell or jolokia to execute commands to > > >> add repositories and install new features. > > >> > > >> When commands are played "slowly", waiting some time between them, all > > >> seems to be OK. > > >> > > >> But when commands are chained, for example in a single jolokia post, > > >> the cluster seems to execute them in an unexpected order. > > >> > > >> For example, the following jolokia post (the same happens with two > > >> commands on the shell CLI), adding a repository, and installing a new > > >> feature is executed on the first node (the node I use as jolokia > > >> target), but it seems that the "installFeature" is propagated to the > > >> second node before it receives or processes the first repository > > >> command : this results with an error because the feature is unknown. > > >> > > >> [ > > >> { > > >> "type" : "exec", > > >> "mbean" : "org.apache.karaf.cellar:name=root,type=feature", > > >> "operation" : "addRepository(java.lang.String, > > >> java.lang.String)", > > >> "arguments":["default", > > >> "mvn:my.tests/my-test-26-karaf-s-jms/0.0.1-SNAPSHOT/xml/features"] > > >> }, > > >> { > > >> "type" : "exec", > > >> "mbean" : "org.apache.karaf.cellar:name=root,type=feature", > > >> "operation" : "installFeature(java.lang.String, > > >> java.lang.String)", > > >> "arguments":["default", "my-test-26-karaf-s-jms"] > > >> } > > >> ] > > >> > > >> After this kind of error the failing node seems not to be able to > > >> synchronize itself, adding again the feature does nothing, and > > >> removing the repository with -u option also leads to errors... > > >> > > >> This is the same if I try to install "standard" features like "http", > > >> the "webconsole", and so on, in a same jolokia call... > > >> > > >> During some tests, after removing the features and repositories, some > > >> bundles are still present on some nodes. The only way to clean them is > > >> to create brand new Karaf containers and create the Cellar cluster > > >> again. > > >> > > >> So, some questions... > > >> > > >> Is there some "time logic" for chaining commands on a Cellar cluster ? > > >> > > >> Is it important to have the same IDs for bundles on all the nodes ? > > >> When commands are executed one by one, and waiting some time, this is > > >> always the case, but when commands are in batches, problems seem to > > >> appear with bundle IDs differences. > > >> > > >> What do people usually do to install big numbers of features on Cellar > > >> clusters, using dev-ops tooling for example ? > > >> > > >> Thanks for your help. > > >> > > >> Le ven. 7 oct. 2022 à 18:48, Jean-Baptiste Onofré <[email protected]> a > > >> écrit : > > >>> The node ID should be still the same. > > >>> > > >>> Let me try to reproduce the "alias lost". It sounds like a bug in > > >>> alias cluster sync indeed. > > >>> > > >>> Regards > > >>> JB > > >>> > > >>> On Fri, Oct 7, 2022 at 6:25 PM Ephemeris Lappis > > >>> <[email protected]> wrote: > > >>>> Hello. > > >>>> > > >>>> Nice ! I'm waiting for the new package ;)... > > >>>> > > >>>> I'm still testing on my docker environment. I've tried deploying > > >>>> features using either the shell or jolokia, always with Cellar > > >>>> commands or MBean. For now, no problem about the features, but I had > > >>>> twice a strange issue : I've set aliases on my 3 instances, and after > > >>>> a feature install, the 3 aliases have been lost. The same when > > >>>> stopping the instances (compose stop in my case), and restarting them. > > >>>> Perhaps some kind of bug on aliases ? > > >>>> > > >>>> Thanks again. > > >>>> > > >>>> Regards. > > >>>> > > >>>> Le ven. 7 oct. 2022 à 08:24, Jean-Baptiste Onofré <[email protected]> > > >>>> a écrit : > > >>>>> Hi, > > >>>>> > > >>>>> Catcha, let me create the Jira and work on this ;) > > >>>>> > > >>>>> Thanks ! > > >>>>> Regards > > >>>>> JB > > >>>>> > > >>>>> On Thu, Oct 6, 2022 at 11:39 PM Ephemeris Lappis > > >>>>> <[email protected]> wrote: > > >>>>>> Hello ! > > >>>>>> > > >>>>>> Thanks a lot for your very detailed answer. > > >>>>>> > > >>>>>> After you explanations, I'm not sure that listeners are really > > >>>>>> needed in > > >>>>>> our case, but I'm going to enable them and test again with basic > > >>>>>> features/bundles commands. If we can use the Cellar's MBean to > > >>>>>> script > > >>>>>> our deployments playbooks with Jolokia calls, perhaps basic > > >>>>>> synchronization is enough for us. > > >>>>>> > > >>>>>> For the last point, a Karaf+Cellar "off the shelf" tarball would > > >>>>>> obviously be a nice gift. I don't know if someone may use a prebuilt > > >>>>>> image : we usually make our own Docker images based on common > > >>>>>> linux+java > > >>>>>> stacks that are elaborated and managed by our DevOps team. Anyway, > > >>>>>> working examples of configuration to build custom Karaf assemblies > > >>>>>> could > > >>>>>> really help : the few examples I've found seem to build limited > > >>>>>> features > > >>>>>> distributions, enumerating known features, adding some custom ones, > > >>>>>> but > > >>>>>> probably missing others. An explained example with all Karaf features > > >>>>>> and the addition of Cellar should be interesting for learning... > > >>>>>> > > >>>>>> So if you could provide both... very happy :) ! > > >>>>>> > > >>>>>> Thanks again. > > >>>>>> > > >>>>>> Ephemeris Lappis > > >>>>>> > > >>>>>> Le 06/10/2022 à 19:56, Jean-Baptiste Onofré a écrit : > > >>>>>>> Hi, > > >>>>>>> > > >>>>>>> 1. By default, only cluster:* commands spread the state on the > > >>>>>>> cluster > > >>>>>>> 2. If you want the "regular" non cluster commands (like > > >>>>>>> feature:install) spread also the state, you have to enable the > > >>>>>>> listeners. The listeners are all disabled by default. You can enable > > >>>>>>> them in etc/org.apache.karaf.cellar.node.cfg. You have one listener > > >>>>>>> per resource: bundle.listener, config.listener, feature.listener. If > > >>>>>>> you set true to all of them, you will have sync on regular command > > >>>>>>> and > > >>>>>>> even for local event (like changing cfg file for instance). It's > > >>>>>>> documented here: > > >>>>>>> https://karaf.apache.org/manual/cellar/latest-4/#_synchronizers_and_sync_policy > > >>>>>>> 3. As jolokia is just a JMX client, and Cellar exposes MBeans, you > > >>>>>>> can > > >>>>>>> interact with cluster using jolokia > > >>>>>>> 4. About the distribution, I should definitely provide a full > > >>>>>>> example > > >>>>>>> to create it and even push a karaf/cellar official distro and docker > > >>>>>>> image. Thoughts ? > > >>>>>>> > > >>>>>>> Regards > > >>>>>>> JB > > >>>>>>> > > >>>>>>> On Thu, Oct 6, 2022 at 5:58 PM Ephemeris Lappis > > >>>>>>> <[email protected]> wrote: > > >>>>>>>> Hello again ! > > >>>>>>>> > > >>>>>>>> I've been testing Cellar on a simple cluster of 3 karaf instances > > >>>>>>>> created with docker images and compose. > > >>>>>>>> I've seen that cluster commands provide a synchronized > > >>>>>>>> provisioning of > > >>>>>>>> features, and for example, that stopped nodes synchronization is > > >>>>>>>> done > > >>>>>>>> when restarting. > > >>>>>>>> This is clearly what we need :) ! > > >>>>>>>> > > >>>>>>>> I've also noticed that "non cluster" feature commands (repo-add, > > >>>>>>>> install, unistall) do not produce the synchronization. I suppose > > >>>>>>>> that's normal. So a new question : will it be possible to use > > >>>>>>>> jolokia > > >>>>>>>> to execute cluster commands the same way we do it with default > > >>>>>>>> features commands ? > > >>>>>>>> > > >>>>>>>> Now I'd like to go a step further before testing on real k8s > > >>>>>>>> clusters. > > >>>>>>>> > > >>>>>>>> In your presentation you said that for now there's not a > > >>>>>>>> downloadable > > >>>>>>>> karaf distribution including Cellar, but that the best way to > > >>>>>>>> deploy > > >>>>>>>> clusters is generating such a custom distribution, and then > > >>>>>>>> providing > > >>>>>>>> a docker image with it. I've not found any example of the plugin > > >>>>>>>> configuration to generate a Karaf+Cellar distribution with all the > > >>>>>>>> default Karaf features and configurations, just adding Cellar. > > >>>>>>>> > > >>>>>>>> Could you please provide any link to working examples ? This could > > >>>>>>>> be > > >>>>>>>> very nice and help a lot ;) !!! > > >>>>>>>> > > >>>>>>>> Thanks again. > > >>>>>>>> > > >>>>>>>> Regards. > > >>>>>>>> > > >>>>>>>> Le mar. 4 oct. 2022 à 07:40, Jean-Baptiste Onofré > > >>>>>>>> <[email protected]> a écrit : > > >>>>>>>>> Yes, you can mix the approaches together. For instance, you can > > >>>>>>>>> package in docker image: karaf runtime + cellar + your apps and > > >>>>>>>>> then > > >>>>>>>>> you mix Kubernetes with Cellar. It's the presentation I did while > > >>>>>>>>> ago > > >>>>>>>>> at ApacheCon. > > >>>>>>>>> > > >>>>>>>>> Regards > > >>>>>>>>> JB > > >>>>>>>>> > > >>>>>>>>> On Tue, Oct 4, 2022 at 7:12 AM Ephemeris Lappis > > >>>>>>>>> <[email protected]> wrote: > > >>>>>>>>>> Hello. > > >>>>>>>>>> > > >>>>>>>>>> Thanks for your explanations. > > >>>>>>>>>> > > >>>>>>>>>> I understand that your 3rd choice is the only one to get > > >>>>>>>>>> multiple active > > >>>>>>>>>> and synchronized instances. But can't I run them as PODs inside a > > >>>>>>>>>> Kubernetes namespace, using deployments of an image based on > > >>>>>>>>>> Karaf+Cellar, and then using the Jolokia API, for example, to > > >>>>>>>>>> deploy and > > >>>>>>>>>> update my applications as features, targeting any one of the > > >>>>>>>>>> scaled > > >>>>>>>>>> instances, and let Cellar synchronizing the other instances ? > > >>>>>>>>>> > > >>>>>>>>>> We already use Jolokia this way via Ansible playbooks to deploy > > >>>>>>>>>> applications, as profiles instead of features, on Fuse > > >>>>>>>>>> clusters... > > >>>>>>>>>> > > >>>>>>>>>> Thanks again. > > >>>>>>>>>> > > >>>>>>>>>> Regards. > > >>>>>>>>>> > > >>>>>>>>>> Ephemeris Lappis > > >>>>>>>>>> > > >>>>>>>>>> Le 03/10/2022 à 18:36, Jean-Baptiste Onofré a écrit : > > >>>>>>>>>>> Hi, > > >>>>>>>>>>> > > >>>>>>>>>>> In order: > > >>>>>>>>>>> > > >>>>>>>>>>> 1. Karaf HA Lock: you have one active, other instances are > > >>>>>>>>>>> passive > > >>>>>>>>>>> 2. Kubernetes: you can orchestrate start/stop of the Karaf > > >>>>>>>>>>> docker > > >>>>>>>>>>> image, but Kubernetes doesn't sync Karaf instances state (like > > >>>>>>>>>>> config, > > >>>>>>>>>>> feature installed, etc) > > >>>>>>>>>>> 3. Cellar: sync Karaf instances together (you install one > > >>>>>>>>>>> feature on > > >>>>>>>>>>> one Karaf instance, the feature will be installed on other Karaf > > >>>>>>>>>>> instances in the cluster) > > >>>>>>>>>>> > > >>>>>>>>>>> Regards > > >>>>>>>>>>> JB > > >>>>>>>>>>> > > >>>>>>>>>>> On Mon, Oct 3, 2022 at 5:44 PM Ephemeris Lappis > > >>>>>>>>>>> <[email protected]> wrote: > > >>>>>>>>>>>> Hello. > > >>>>>>>>>>>> > > >>>>>>>>>>>> I've just looked at the presentation of Cellar. If I > > >>>>>>>>>>>> understand it > > >>>>>>>>>>>> well, this presentation says that Cellar's main goal is for > > >>>>>>>>>>>> "big > > >>>>>>>>>>>> clusters", allowing automatic synchronization between Karaf > > >>>>>>>>>>>> instances. > > >>>>>>>>>>>> It seems to be really nice in the presentation :) ! > > >>>>>>>>>>>> > > >>>>>>>>>>>> On the other hand, the basic lock mechanism only provides an > > >>>>>>>>>>>> active/passive solution. > > >>>>>>>>>>>> > > >>>>>>>>>>>> What should I prefer if my need is to provide both failover > > >>>>>>>>>>>> and load > > >>>>>>>>>>>> balancing over a limited number of active instances, and not a > > >>>>>>>>>>>> "big > > >>>>>>>>>>>> cluster". Today we use 6 Fuse Karaf instances distributed on 3 > > >>>>>>>>>>>> VM. Is > > >>>>>>>>>>>> Cellar the right way, or did I miss something in the > > >>>>>>>>>>>> presentation ? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Another thing : for other kinds of applications my customer > > >>>>>>>>>>>> manages > > >>>>>>>>>>>> several Kubernetes clusters. So I suppose that if a > > >>>>>>>>>>>> containerized > > >>>>>>>>>>>> solution is preferred, it should be running on Kubernetes, > > >>>>>>>>>>>> since all > > >>>>>>>>>>>> the existing DevOps tooling is already based on it. > > >>>>>>>>>>>> > > >>>>>>>>>>>> The presentation focuses on Mesos/Marathon but also says that > > >>>>>>>>>>>> Kubernetes is also an alternative solution. Right ? In this > > >>>>>>>>>>>> case, what > > >>>>>>>>>>>> is the preferred way to package and deploy Karaf : just create > > >>>>>>>>>>>> a > > >>>>>>>>>>>> custom Karaf+Cellar image (the same way the presentation > > >>>>>>>>>>>> shows), and > > >>>>>>>>>>>> then create a Kubernetes deployment with the needed sizing and > > >>>>>>>>>>>> scaled > > >>>>>>>>>>>> replicas ? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Some examples perhaps ? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Thanks in advance for your help. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Regards. > > >>>>>>>>>> -- > > >>>>>>>>>> Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > >>>>>>>>>> www.avast.com > > >>>>>> -- > > >>>>>> Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > >>>>>> www.avast.com > > > > -- > > Cet e-mail a été vérifié par le logiciel antivirus d'Avast. > > www.avast.com
karaf-logs.7z
Description: Binary data
deployment
Description: Binary data
docker exec -it karaf1_karaf1_1 /opt/apache-karaf-4.4.1/bin/client -- "feature:repo-add cellar; feature:install cellar" docker exec -it karaf1_karaf2_1 /opt/apache-karaf-4.4.1/bin/client -- "feature:repo-add cellar; feature:install cellar" sleep 60 cp deployment /FIFI/docker-volumes/COMPOSE_KARAF1/share/ docker exec -it karaf1_karaf1_1 /opt/apache-karaf-4.4.1/bin/client -f /SHARE/deployment
