On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote: > On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote: >> Hello folks >> >> I wanted to get your thoughts on the idea to move the whole Bitergia >> grimoire engine, scripts, database, etc to the openstack common >> infrastructure. I'm sure everybody will want this so the question is >> more about resources as in people willing to help Bitergia get their >> machinery puppetized the "OpenStack-CI way". >> >> At the moment the various spiders and scripts run on a machine on >> Bitergia's end and they drop the results of the elaboration (json, html, >> css files and sql dumps) on activity.openstack.org/dash/. >> >> While this setup is convenient and has been working so far, I think >> we've outgrown it. One of the areas that we want to do more with is the >> datawarehouse built by Bitergia, enable it to serve other purposes too >> and allow more collaboration from the community. For example, once the >> tools are all out in one place, interested parties could build some sort >> of service on top of your datawarehouse to export the data about the >> affiliation. Others could build tools to 'fix' such affiliation, pulling >> the various mailmap files used by gitdm and stackalytics. >> >> The question for OpenStack-CI is then: in the next weeks/months, is >> there going to be someone free, from the CI team or someone willing to >> join it (Dan?), to help Bitergia's team get their tools on our >> infrastructure? > > Thanks for the introduction of the issue, Stefano. > > Just to clarify a bit the needs to move all the Grimoire machinery to a > vm under openstack-ci, the software is split in three main parts:
Awesome. > * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.), > and store the data into a MySQL database (well, in fact, one schema per > kind of repository). > > * The MySQL database itself. > > * The vizGrimoire tools, that run the analysis, produce JSON files, and > the HTML/CSS/JavaScript files needed to serve the dashboard. All of this > is for now static, which means that you only need to serve those files > via HTTP, and you're done: no live queries to the database once the JSON > files are produced. > > Right now, we produce the JSON files once a day. That means that > MetricsGrimoire is first run once a day (the tools know how to get > incremental information from repositories), data is updated in the > database, and then vizGrimoire analysis is run. All of this is > controlled by automator, the tool that is configured with the list of > repos to analyze, the analysis to run, etc. > > MetricsGrimoire tools are written in Python. There are some Python > dependencies beyond Python 2.7, but I guess all of them are > straightforward from pypy. Where are these stored/installed from? Is it "pip install MetricsGrimoire"? Is it in a git repo? Does this git repo want to move into openstack tooling or does it want to stay where it is and have openstack-infra consume releases? > The MySQL is a plain MySQL. It would be great having it in SSD, because > that speeds up queries a lot. But probably it will work with regular > disks. We use cloud-based databases from Rackspace's Trove service. I have no idea what sort of disk is behind it - but MySQL databases are no problem. > Most of vizGrimoire is also Python, but there is still some R code > (we're currently moving towards a pure-Python implementation). Python > and R dependencies are easy too (from pypy or CRAN). Same questions as above on MetricsGrimoire. Do you have docs anywhere on installing these two things? > We're usually deploying on Debian or Ubuntu, but we have some experience > with other Linux-based OSs too. Great. All of our services run on Ubuntu Precise right now. We'll be starting to think about an upgrade to trusty. > We could do the whole deployment, if you can provide us with a > Debian/Ubuntu vm, including proper documentation to reproduce it if > needed. All of our deployments are done in a fully automated manner driven from our puppet repo: https://git.openstack.org/cgit/openstack-infra/config. However - we'd be more than happy for you to write the puppet! (and happy to help with pointers) I just wrote the puppet to run stackalytics in Infra: https://review.openstack.org/98656 It's probably a great place to start in terms of looking at puppetizing activity board. If someone can write up something similar to: https://wiki.openstack.org/wiki/Stackalytics/HowToRun I'll be more likely to do what I did today which is wake up on a Sunday and lazily write some puppet while watching Rafael Nadal win his ninth French Open. :) > Please, let us know which kind of other details you may need. > > Jesus. > > _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
