Hey Shubham, Yes I did that and it did not work, once I scale it by starting it from 1. The website no longer loads.
On Sat, 20 Jul 2019 at 10:00 AM, Shubham Jain <jainshubha...@gmail.com> wrote: > That’s interesting. > > Were you able to figure out the problems when you scale up from 1 Pod? If > not, share the logs. Let’s try to figure out problem. > > Best, > Shubham. > > > On 20-Jul-2019, at 6:36 AM, Ashish Sharma <sharma.ashish...@gmail.com> > wrote: > > > > Hey Shubham, > > > > I have tried initialising with 1 pod and scaling it up to multiple pods, > > also tried session affinity, and some other cases but none of them > worked. > > > > Thanks, > > Ashish > > On Fri, 19 Jul 2019 at 4:51 PM, Shubham Jain <jainshubha...@gmail.com> > > wrote: > > > >> Hey Ashish, > >> > >> I don’t understand XWiki well enough to fully understand the > implications > >> of running a clustered setup by sharing file system. > >> > >> Can you try keeping the deployment size initially 1, and then later try > to > >> increase the size? Does that also cause problems? > >> > >> Best, > >> Shubham. > >> > >>> On 19-Jul-2019, at 4:03 PM, Ashish Sharma <sharma.ashish...@gmail.com> > >> wrote: > >>> > >>> Hey Shubham, > >>> > >>> I have moved the solr folder as exposed by the docker container to the > >> file > >>> system. I think in-memory cache is creating problem. > >>> > >>> I started 3 container of xwiki with shared file storage, but all three > >> were > >>> trying to initiate the postgress database, that is causing problem. > >>> > >>> I want to know few things about xwiki:- > >>> > >>> 1. Do we use java threads ? -> if we do then filesystem locking should > >> have > >>> been implemented for that? > >>> > >>> 2. Other stateful things in our application like caching or other > >>> filestorage that should be externalise? > >>> > >>> > >>> Thanks, > >>> Ashish > >>> > >>> On Fri, 19 Jul 2019 at 3:59 AM, Shubham Jain <jainshubha...@gmail.com> > >>> wrote: > >>> > >>>> Hey Ashish, > >>>> > >>>> I think for now you could stick to deployment. Have you been ever to > >>>> discover the data you need to move to the shared file system? Is the > >>>> in-memory caching mechanism causing problems? > >>>> > >>>> @Thomas/Vincent - Can you please help us understand the implications > of > >>>> running XWiki in a clustered mode by sharing data volumes? > >>>> > >>>> Best, > >>>> Shubham. > >>>> > >>>> > >>>> Best, > >>>> Shubham? > >>>> > >>>>> On 19-Jul-2019, at 5:35 AM, Ashish Sharma < > sharma.ashish...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Resent the email at the mail thread. > >>>>> > >>>>> On Fri, Jul 19, 2019 at 2:01 AM Ashish Sharma < > >>>> sharma.ashish...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hello everyone, > >>>>>> > >>>>>> I am trying to implement xwiki helm chart in High availability but > >>>> facing > >>>>>> some problems. > >>>>>> > >>>>>> How we can implement High Availability(HA) in Kubernetes:- > >>>>>> > >>>>>> 1. Through Deployment:- In this approach we would use a shared file > >>>>>> storage, that would get attached to multiple pods(or containers) > >>>>>> 2. Through StatefulSet:- In this approach every pod(or container) > >> would > >>>>>> have it’s own different volume and state, which should be > synchronised > >>>> by > >>>>>> an consensus algorithm > >>>>>> > >>>>>> Problem faced in xwiki case:- > >>>>>> > >>>>>> 1. In deployment case we need a locking mechanism for our > filesystem, > >>>>>> moreover caching and states should also be maintained outside the > >>>> container. > >>>>>> 2. In StatefulSet the consensus algorithm is not present so it is > >>>>>> currently not possible. > >>>>>> > >>>>>> Could anyone help me through it. > >>>>>> > >>>>>> Thanks, > >>>>>> Ashish > >>>>>> > >>>>>> On Wed, 10 Jul 2019 at 10:32 AM, Shubham Jain < > >> jainshubha...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Hey Ashish, > >>>>>>> > >>>>>>> I think Vincent/Thomas can help with the PR on the xwiki-docker > >>>> project. > >>>>>>> > >>>>>>> Regarding adding docs for High-Availability setup using shared data > >>>>>>> volumes, I think you can use could create a docs folder in the > GitHub > >>>> repo, > >>>>>>> and add the docs in Markdown format. I think you can also document > it > >>>> in > >>>>>>> XWiki blog <https://www.xwiki.org/xwiki/bin/view/Blog/>. The chart > >>>>>>> should support enabling HA, and then passing a storage class which > >>>> supports > >>>>>>> shareable volumes to Pods, and then you can document examples for > >> using > >>>>>>> this with something like Rook+GKE setup. > >>>>>>> > >>>>>>> Best, > >>>>>>> Shubham. > >>>>>>> > >>>>>>> > >>>>>>>> On 08-Jul-2019, at 1:57 PM, Ashish Sharma < > >> sharma.ashish...@gmail.com > >>>>> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Hello everyone, > >>>>>>>> > >>>>>>>> I require the xwiki solr docker container to do progress with my > >>>> work. I > >>>>>>>> have made a pull request > >>>>>>>> <https://github.com/xwiki-contrib/docker-xwiki/pull/22>, for the > >>>> same. > >>>>>>>> Please look into it. > >>>>>>>> > >>>>>>>> Moreover, I was working with Rook(Ceph Operator) for a shared file > >>>>>>> system > >>>>>>>> which would be a better approach to deploy our app on production. > >> So I > >>>>>>>> think we should document it somewhere. Can anyone help me where > >> should > >>>>>>> I do > >>>>>>>> that. It may contain the details to configure rook-ceph, and then > >>>>>>>> connecting it. > >>>>>>>> > >>>>>>>> P.S. I am writing a blog at Medium on how to configure rook on > >> Google > >>>>>>>> Kubernetes Engine. We could also use that? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Ashish > >>>>>>>> > >>>>>>>> On Mon, Jun 24, 2019 at 8:31 PM Ashish Sharma < > >>>>>>> sharma.ashish...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Shubham, > >>>>>>>>> > >>>>>>>>> Thank you Shubham, I will start working on adding test cases. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Ashish > >>>>>>>>> > >>>>>>>>> On Mon, 24 Jun 2019 at 3:32 PM, Shubham Jain < > >>>> jainshubha...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hey Ashish, > >>>>>>>>>> > >>>>>>>>>> Thanks for the update. > >>>>>>>>>> > >>>>>>>>>> Adding our chart to incubator section of helm community charts > >>>>>>> repository > >>>>>>>>>> sounds great. Although (unlike Dockerhub) this repo is for > >> community > >>>>>>> charts > >>>>>>>>>> and not really official charts of the respective installations, > >> but > >>>>>>>>>> supporting this chart in the community chart repo should help > with > >>>> the > >>>>>>>>>> visibility. > >>>>>>>>>> > >>>>>>>>>> Lets also work on adding tests to the helm chart to ensure > things > >>>> are > >>>>>>>>>> going fine. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Shubham. > >>>>>>>>>> > >>>>>>>>>>> On 21-Jun-2019, at 4:37 AM, Ashish Sharma < > >>>>>>> sharma.ashish...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi everyone, > >>>>>>>>>>> > >>>>>>>>>>> Update on HELM project:- > >>>>>>>>>>> > >>>>>>>>>>> We have added some features to the chart that includes:- > >>>>>>>>>>> * Making chart configurable with both mysql and postgres > database > >>>>>>>>>>> * We are now using configmaps and secrets for storing database > >>>>>>>>>> credentials > >>>>>>>>>>> * We use helm dependency to manage the dependencies of the > chart > >>>>>>>>>>> > >>>>>>>>>>> I think our chart is ready to be deployed in the incubator > >> section > >>>> of > >>>>>>>>>> helm > >>>>>>>>>>> community repo. <https://github.com/helm/charts> If you guys > >>>> agree I > >>>>>>>>>> would > >>>>>>>>>>> make the necessary changes required to deploy it there. > >>>>>>>>>>> > >>>>>>>>>>> Thanks > >>>>>>>>>>> - Ashish > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Jun 5, 2019 at 7:40 PM Vincent Massol < > >> vinc...@massol.net> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Ashish, > >>>>>>>>>>>> > >>>>>>>>>>>>> On 5 Jun 2019, at 15:07, Ashish Sharma < > >>>> sharma.ashish...@gmail.com > >>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi everyone, > >>>>>>>>>>>>> > >>>>>>>>>>>>> The first week update of helm chart xwiki project. We > currently > >>>>>>> have a > >>>>>>>>>>>>> chart that would deploy xwiki mysql stateful service with a > >>>>>>> persistent > >>>>>>>>>>>>> volume. The link to the repo:- > >>>>>>>>>>>>> > >>>>>>>>>>>>> https://github.com/xwiki-contrib/xwiki-helm > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for the update. > >>>>>>>>>>>> > >>>>>>>>>>>> Would be great if you could update the README file to explain > >> who > >>>> to > >>>>>>>>>> use > >>>>>>>>>>>> it. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> -Vincent > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thank You > >>>>>>>>>>>>> Ashish > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 23 May 2019 at 7:27 PM, Ashish Sharma < > >>>>>>>>>>>> sharma.ashish...@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Wed, May 22, 2019 at 12:42 PM Vincent Massol < > >>>>>>> vinc...@massol.net> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Ashish, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 21 May 2019, at 19:37, Ashish Sharma < > >>>>>>>>>> sharma.ashish...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi everyone, > >>>>>>>>>>>>>>>> Thanks for having me here > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> About Me > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I am Ashish Sharma, selected as a student for Google > Summer > >> of > >>>>>>>>>> Code. I > >>>>>>>>>>>>>>> am > >>>>>>>>>>>>>>>> final year student enrolled in Guru Gobind Singh > >> Indraprastha > >>>>>>>>>>>>>>> University, > >>>>>>>>>>>>>>>> Delhi. I am a resident of India. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Profiles > >>>>>>>>>>>>>>>> GitHub - https://github.com/ashish932/xwiki-helm-chart/ > >>>>>>>>>>>>>>>> LinkedIn - https://www.linkedin.com/in/ashish932/ > >>>>>>>>>>>>>>>> Riot - @ashish932:matrix.org > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I will be presenting my project "Helm Chart for XWiki" to > >> all > >>>> of > >>>>>>>>>> you. > >>>>>>>>>>>>>>>> Following > >>>>>>>>>>>>>>>> are the relevant details. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Helm Chart for XWiki > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Mentors: Shubham Jain, Neha Gupta > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Technologies: Kubernetes, Docker, other if required > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Overview > >>>>>>>>>>>>>>>> The proposed project is a helm chart that would deploy > xwiki > >>>> as > >>>>>>>>>> highly > >>>>>>>>>>>>>>>> available and reliable. It should be configurable with > >>>> different > >>>>>>>>>>>>>>>> databases(either a standalone database or a clustered one) > >>>> that > >>>>>>> are > >>>>>>>>>>>>>>>> configurable with xwiki. It would give the option to > either > >>>>>>>>>> configure > >>>>>>>>>>>>>>> solr > >>>>>>>>>>>>>>>> externally (standalone or clustered) or managed within the > >>>>>>>>>> container. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It also needs to be deployed on various Servlet Containers > >> and > >>>>>>> with > >>>>>>>>>> a > >>>>>>>>>>>>>>> LibreOffice server configured. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In short, it should support all the configurations > supported > >> by > >>>>>>> the > >>>>>>>>>>>>>>> official XWiki Docker images :) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> It > >>>>>>>>>>>>>>>> should deploy the app on a shared file system like a rook. > >> It > >>>>>>>>>> should > >>>>>>>>>>>>>>>> support Istio virtual services, istio matrix, and istio > >>>>>>> distributed > >>>>>>>>>>>>>>> tracing > >>>>>>>>>>>>>>>> and should be a secured system with RBAC and security > >>>> credential > >>>>>>>>>>>>>>> rotation. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ok I have no knowledge about this… The shared file system > >>>> sounds > >>>>>>>>>> good > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> support clustering XWiki for the permanent directory > sharing. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The chart should be easily deployed on GKE and amazon EKS. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On locally on minikube for testing I guess? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Yes we can test using minikube one > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Features > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -> Support for different Databases > >>>>>>>>>>>>>>>> -> Choice between using an external database, a single > node > >>>> DB > >>>>>>> or > >>>>>>>>>> a > >>>>>>>>>>>>>>>> multi-cluster DB setup > >>>>>>>>>>>>>>>> -> Support for shared file system > >>>>>>>>>>>>>>>> -> Support for istio and it's services > >>>>>>>>>>>>>>>> -> RBAC, SSL and other security methods > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> If you have any features in mind that should be added > please > >>>>>>> feel > >>>>>>>>>> free > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> reply to this mail. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Some Design Questions? > >>>>>>>>>>>>>>>> -> Which Databases should be supported? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> See > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>> > >> > https://dev.xwiki.org/xwiki/bin/view/Community/SupportStrategy/DatabaseSupportStrategy > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> More generally see the XWiki Docker image documentation and > >>>>>>>>>>>>>>> > >> https://dev.xwiki.org/xwiki/bin/view/Community/SupportStrategy > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -> As we have to detach solr out of the docker > container(run > >>>> it > >>>>>>> in > >>>>>>>>>> an > >>>>>>>>>>>>>>>> independent container) would be there a requirement for a > >> code > >>>>>>>>>> change, > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> we should approach it? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> See the doc for Solr which explains how to configure an > >>>> external > >>>>>>>>>> SOLR. > >>>>>>>>>>>>>>> There are some XWiki code/config to deploy in the external > >> SOLR > >>>>>>>>>> AFAIR. > >>>>>>>>>>>>>>> comunity controls chart > >>>>>>>>>>>>>>>> -> Apart from solr is there any other stateful service > that > >>>>>>> could > >>>>>>>>>> or > >>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>> be detached from the docker container? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Which docker container are you referring to? The Servlet > >>>>>>> Container > >>>>>>>>>> one? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> xwiki:lts-mysql-tomcat > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Here is my current repository which deploys XWiki for > MySQL > >>>>>>>>>> database > >>>>>>>>>>>>>>> using > >>>>>>>>>>>>>>>> official XWiki docker container:- > >>>>>>>>>>>>>>>> https://github.com/ashish932/xwiki-helm-chart/ > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Will need to have a look. Some additional questions: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1) Is there an official place for publishing helm charts, > >>>>>>> similar to > >>>>>>>>>>>>>>> dockerhub for docker images? How do you provide the helm > >> charts > >>>>>>> to > >>>>>>>>>>>> XWiki > >>>>>>>>>>>>>>> users in term of packaging? In other words, what will > we/you > >>>>>>>>>> document > >>>>>>>>>>>> on > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>> > >> > https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/ > >>>>>>>>>>>>>>> ? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yes, There is a centralized community control chart > >> management > >>>>>>>>>>>>>> repository. Helm charts are hosted there. Organizations like > >>>>>>> mysql, > >>>>>>>>>>>>>> wordpress uses it. > >>>>>>>>>>>>>> https://github.com/helm/charts > >>>>>>>>>>>>>> But in addition to that, we can build our own charts > >> repository > >>>>>>>>>> locally, > >>>>>>>>>>>>>> and host it. Similar to docker run command, we have helm > >> update > >>>>>>>>>> command > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2) How do you plan to have smoke tests for the XWiki Helm > >>>>>>> charts, as > >>>>>>>>>>>> part > >>>>>>>>>>>>>>> of the build, to verify that they work fine? How can we > >> execute > >>>>>>>>>> XWik’s > >>>>>>>>>>>>>>> functional tests based on TestContainers using Helm Charts? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There are some tools available > >>>>>>>>>>>>>> chart-testing(started by helm) -98 commits > >>>>>>>>>>>>>> terratest - 1039 commits (It looks a good tool to use) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 3) Do the helm charts for XWiki need to be built? If so, > can > >> it > >>>>>>> be > >>>>>>>>>> done > >>>>>>>>>>>>>>> with Maven? Since we’ll be offering several I guess, how do > >> you > >>>>>>>>>> avoid > >>>>>>>>>>>> code > >>>>>>>>>>>>>>> duplication and make maintenance of them as simple as > >> possible? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> No, the charts don’t need to be built externally, though it > >> gets > >>>>>>>>>>>>>> compressed when pushed f we build to chart repository(this > is > >> an > >>>>>>>>>>>> automatic > >>>>>>>>>>>>>> step in case of official chart repo and we can automate > using > >> ci > >>>>>>> our > >>>>>>>>>> own > >>>>>>>>>>>>>> chart repo) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 4) Could you provide a roadmap with various milestones and > >>>> dates? > >>>>>>> See > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> last roadmap email I sent on the devs list and please reply > >> to > >>>>>>> it. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Okay I would work on that and update the progress on the > mail > >>>>>>> thread > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 5) It seems you missed the bonding period so you’ll need to > >>>> catch > >>>>>>>>>> up on > >>>>>>>>>>>>>>> that. Could you pick some jira issue and work on them > >> quickly? > >>>>>>> This > >>>>>>>>>> is > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> make sure you use the proper xwiki communication channels, > >>>> learn > >>>>>>> to > >>>>>>>>>>>> work > >>>>>>>>>>>>>>> with the xwiki team and start understanding the xwiki > >>>>>>> architecture, > >>>>>>>>>>>> and how > >>>>>>>>>>>>>>> extensions are developed for XWiki. Since you’re a bit > late, > >>>>>>> you’ll > >>>>>>>>>>>> need to > >>>>>>>>>>>>>>> catch up and achieve this very quickly now and be present > >> full > >>>>>>> time > >>>>>>>>>> on > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> xwiki chat :) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> I would start doing that > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 6) Idea: please review the issues on the XWiki Docker > >> project’s > >>>>>>>>>> jira. > >>>>>>>>>>>>>>> Maybe you can help with those too as part of > >>>> onboarding/bonding. > >>>>>>> It > >>>>>>>>>>>> should > >>>>>>>>>>>>>>> also give you additional use cases for the XWiki Helm > Charts > >>>> IMO. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks! I’m eager to follow your work and see xwiki helm > >> charts > >>>>>>> in > >>>>>>>>>>>> action. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -Vincent > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thank You > >>>>>>>>>>>>>>>> Ashish Sharma > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thank You > >>>>>>>>>>>>>> Ashish > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>> > >>>> > >> >