Hi! El jue, 12-07-2012 a las 22:31 -0400, Dave Brondsema escribió: > On Wed, Jul 11, 2012 at 4:07 AM, Alvaro del Castillo <[email protected]>wrote: > > > Hi guys, > > > > My name is Alvaro del Castillo and I work as a software engineer in > > Bitergia company (http://bitergia.com). > > > > > Hi Alvaro! > > > > First of all, thank you for Allura. It is a very nice piece of software, > > with a solid and recent architecture, based in Python so very agile to > > develop with it, so modular and last but not least, Open Source! > > > > Bitergia company is specialized in metrics in software projects. It is a > > very young company. And we are working close with Allura for some months > > now. We think that the best place to integrate our software metrics is > > inside forges so it has been a very natural decision, because Bitergia > > is an Open Source soul company. > > > > We have some experience with Fusion Forge but when we discover Allura > > the adoption of it was a natural choice. > > > > Great :D > > > > In our first prototype for creating metrics inside Allura we hace > > created a new module for Allura (AlluraBitergiaMetrics is the actual > > name, but probably we should change it) for displaying metrics. We have > > an Allura server with the module deployed at: > > > > http://bitergia.com:8080/p/joomla/metrics/commits_commiters/ > > > > > That looks very nice.
Thank you. > > > > It is just a first try of the technology and possibilities. And now it > > is time to define next steps, cleanup code and work with Allura > > community in order to decide the best way to implement things. > > > > The source code for the module is in: > > > > https://github.com/Bitergia/AlluraBitergiaMetrics > > > > > I've looked at the code briefly, but rather than read through all of it, > perhaps you can inform me. How do you gather the data for the reports? Right now we are following our general steps: 1. Identify the information sources: Source Code Management (SCM: svn, git, bazar ...), ITS (allura, launchpad, github) ... 2. Use Grimoire Metrics Tools (https://github.com/MetricsGrimoire) to get the information and store them in a database. 3. Using SQL queries, and R for advanced statistics, we generate JSON and CVS files with the information for the graphs. 4. From the web client, we download the JSON/CSV files and in the client we generate the graphs using right now Envision/Flotr2 (we are working also with other frameworks like D3). But this process could be improved if we are inside a forge and we have direct access to the information. This is something really interesting for real time info for example. > There is repository refresh code that is triggered after each commit. > Does it hook into that? Would it be useful to do so? The tools could be configured to be launched triggered after commits, but right now, the analysis is done in general downloading the full repository (source code, issues, mailing list archives) and analysing it. We have some incremental support in some of the tools, but not in all of them. > > What about scalability? For us at SourceForge, we have hundreds of > thousands of projects, so it is of some concern to us. Thinking in the client side the current approach pregenerating the JSON/CSV data files and doing all the graphs with javascript in the client is very scalable. For the server, the more demanding activity will be to analyse the project information and generate the JSON/CSV files. Using Allura MongoDB for storing the date, JSON friendly, this process could be optimized. With the current prototype we have integrated in Allura the needed javascript files for plotting (flotr2/envision), the static json files and the AlluraBitergiaMetrics is a very "static" html application. Our idea is to advance in the integration in some ways: * Program the execution of the tools using taskd. Is that a good idea? I have created it but the use of taskd in other modules is low. * Configure the metrics tools from the AlluraBitergiaMetrics module so they use directly the Allura backends for projects hosted in the forge. This Allura backends are using right now for issues the JSON API. For example: 1. Get all the issues for a project: http://sourceforge.net/rest/p/allura/tickets 2. Download each issue: http://sourceforge.net/rest/p/allura/tickets/1/ If the module is running inside Allura platform, maybe using local URLs could be quick enough: http://localhost/rest/p/allura/tickets but we can also adapt the tools to access directly to MongoDB database. For realtime processing I understand this is the best option. * Evolve the metrics visualization in a configurable dashboard: this is our mid term plan (some months). We have more ideas for Allura integration but these are the priority right now for us. > > > > I need to remove some testing code. > > > > So next steps, how do you see this initiative? do you find interesting > > including this kind of information in the forge? Is it the best way to > > create a new module for Allura? > > > > Of course the nice thing about Allura is that the tools/apps can be > independent, like you have developed AlluraBitergiaMetrics already. This > does seem like it would be something nice to integrate into the core SCM > tools (git/hg/svn) that come with Allura. Great! > > At SourceForge we have some SCM statistics in our legacy system (e.g. > http://sourceforge.net/projects/sqlitebrowser/stats/scm?repo=SVNRepository) > for which we would like to have an equivalent in Allura, at some point. Nice. > Actually the recording of the SCM activity is in our legacy system, but > the stats system is independent. We use Zarkov > http://sourceforge.net/p/zarkov/home/Home/ for these stats and others on > SourceForge. So we would likely implement something in Allura that works > with Zarkov. The Grimoire Metrics tools are all of them programmed in Python as Zarkov, so it is something natural for us. "We're using Zarkov to watch Allura events and create pretty graphs for internal use." So it seems is something similar to the Grimoire Metrics Tools. Dave, do you have some example of graphs using Zarkov information? I can read that Rick has played with flot and Zarkov. > > Anyway, the types of stats you have here are beyond that, and could be in > addition to anything we implement for our legacy data. Both could be > useful. > > If a new feature like this were to be added to the core tools, it would be > best controlled by a configuration option to enable it or not. We do this > for some of our features (e.g. Zarkov is not a required dependency for > Allura). Config flags are very useful so that different deployments of > Allura can control whether or not they want to use a feature. We have also > learned (the hard way) that it is useful during development so that > something new can be added to Allura if it has basic functionality but is > not ready to be enabled in production. That keeps the code in sync with > the main 'dev' branch. Having tons of work in a branch until it is > completely polished and done is likely to have conflicts and merge issues. > A config flag is better. Ok. So next steps: - We are expanding the kind of graphs we support and also the metrics. We will integrate them in our Allura module. - Control metrics tools execution from taskd. - Allura project configuration to enable graphs and auto configure the metrics tools to use the Allura backends. - Study Zarkov to understand the kind of information it has and how you are using it to create graphs. Kind regards! > > -Dave > > -- > Dave Brondsema > Principal Software Engineer - sf.net > Geeknet > -- |\_____/| Alvaro del Castillo San Félix [o] [o] [email protected] - Software Engineer | V | http://www.bitergia.com | | "Bridging the gap between developers and stakeholders" -ooo-ooo-
