Re: dashboarding incubator
After awaiting for feedback about my proposal, I understand there are three different aspects that should be discussed: * Cost: As Ross pointed, the potential prize is important to evaluate a solution. Although I'd love to use the professional services of the company, the toolkit is open/free software and be freely used, which moves more attention to the next point. * Infrastructure requirements: Specially in the case we decide to provide all by ourselves, such service would have some infrastructure requirements that need to be studied, as David correctly pointed. * Technical proposition: In the end the first two aspect should not be critical if the proposition brings some value, to the project-level, Incubator or ASF. I really see strong arguments against the proposal regarding the first two aspects. The third is not that easy, since I do not see how such metrics should be used for evaluating projects, rather than just bringing some indicators. Before taking the discussion to the next level, where costs and resources need to be evaluated, I opened this discussion proposing my time and personal resources to provide a simple proof of concept. Then we should have more arguments (how much resources are actually required, how useful are the indicators the dashboard provides, etc...) to move the discussion to the next level. But of course I'd like to have the good pleasure before investing time. So I'd like to ask the following question: is there already any argument to say that inevitably the answer of the proof of concept will be negative? Cheers, On 21/11/14 21:27, Ross Gardler (MS OPEN TECH) wrote: We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Sergio Fernández Senior Researcher Knowledge and Media Technologies Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernan...@salzburgresearch.at http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
On 23 November 2014 at 20:37, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: After awaiting for feedback about my proposal, I understand there are three different aspects that should be discussed: * Cost: As Ross pointed, the potential prize is important to evaluate a solution. Although I'd love to use the professional services of the company, the toolkit is open/free software and be freely used, which moves more attention to the next point. * Infrastructure requirements: Specially in the case we decide to provide all by ourselves, such service would have some infrastructure requirements that need to be studied, as David correctly pointed. * Technical proposition: In the end the first two aspect should not be critical if the proposition brings some value, to the project-level, Incubator or ASF. I really see strong arguments against the proposal regarding the first two aspects. The third is not that easy, since I do not see how such metrics should be used for evaluating projects, rather than just bringing some indicators. Before taking the discussion to the next level, where costs and resources need to be evaluated, I opened this discussion proposing my time and personal resources to provide a simple proof of concept. Then we should have more arguments (how much resources are actually required, how useful are the indicators the dashboard provides, etc...) to move the discussion to the next level. But of course I'd like to have the good pleasure before investing time. So I'd like to ask the following question: is there already any argument to say that inevitably the answer of the proof of concept will be negative? I personally think a proof of concept would be beneficial, and might help put some of the problems raised in perspective. Infra are (with full right) concerned about new services which they potentially need to support if used by many and abandoned by the original supporter. I believe a proof of concept might end up showing this could be very simple. I dont have spare cycles to help with this, but I am available anytime for questions/test etc. rgds jan I. Cheers, On 21/11/14 21:27, Ross Gardler (MS OPEN TECH) wrote: We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Sergio Fernández Senior Researcher Knowledge and Media Technologies Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernan...@salzburgresearch.at http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: dashboarding incubator
Thanks for your roundup here (very useful). You are making it clear that this is something you might want to spend time on - I'll try and answer your final question (is there already any argument to say that inevitably the answer of the proof of concept will be negative?). The short answer is yes and no. Below I spend most of the time explaining the no, if you want the short version skip to the yes in the last para. Speaking entirely personally, I have always argued against using numbers to judge the health of a project (which is the only natural outcome from collecting such numbers). For an ASF project it's not absolute activity that is important. It's the strength and behavior of the community and its governance that is important. Metrics do not provide this information, and indeed can detract from the community health issues. For example, many years ago we had a podling that didn't graduate for a long time because we had evolved into having a hard metric on what diversity meant. It did graduate in the end and today is a thriving and productive TLP. The significant expansion of the community didn't happen until after graduation, a time when newcomers feel it is safe to invest in the project. We've since scrapped the diversity metric and reverted to the original intent of requiring a project to behave in a way that is welcoming to diverse interests (and thus capable of satisfying a diversity metric given time). My point is that while some metrics can provide indicators of something that needs looking into we need to ensure that the metrics do not become more important than community health. Personally, I do use metrics when evaluating a project, but I use ones that are readily available already through a number of other services. These are not official or sanctioned and therefore say nothing, from an ASF perspective, about the health of a project. The danger I see is that providing official metrics a) provides a level of authority to the metrics which most newcomers are ill-equipped to evaluate and b) could lead to shortcut rules like the metrics must show there is X level of diversity/activity/volume/foobar replacing proper evaluation of the project and its community. In summary, I'm not against metrics per se, I'm cautious about them becoming more important than they should be. I can imagine the tools being somewhat useful *internally* where we can ensure that expectations are properly managed. I am very cautious about using such metrics externally where they can be quoted out of context and/or misrepresented. All that being said, sponsors are increasingly asking us for metrics. For this reason I'm vary interested in cross-foundational statistics rather than statistics about specific projects. That is if you were to roll up the data from across the projects into valuable data about the foundation as a whole I can see real value with minimal risk. Sally, over on press@ is currently looking at the kind of data that would be useful to report. Ross -Original Message- From: Sergio Fernández [mailto:sergio.fernan...@salzburgresearch.at] Sent: Sunday, November 23, 2014 11:37 AM To: general@incubator.apache.org Subject: Re: dashboarding incubator After awaiting for feedback about my proposal, I understand there are three different aspects that should be discussed: * Cost: As Ross pointed, the potential prize is important to evaluate a solution. Although I'd love to use the professional services of the company, the toolkit is open/free software and be freely used, which moves more attention to the next point. * Infrastructure requirements: Specially in the case we decide to provide all by ourselves, such service would have some infrastructure requirements that need to be studied, as David correctly pointed. * Technical proposition: In the end the first two aspect should not be critical if the proposition brings some value, to the project-level, Incubator or ASF. I really see strong arguments against the proposal regarding the first two aspects. The third is not that easy, since I do not see how such metrics should be used for evaluating projects, rather than just bringing some indicators. Before taking the discussion to the next level, where costs and resources need to be evaluated, I opened this discussion proposing my time and personal resources to provide a simple proof of concept. Then we should have more arguments (how much resources are actually required, how useful are the indicators the dashboard provides, etc...) to move the discussion to the next level. But of course I'd like to have the good pleasure before investing time. So I'd like to ask the following question: is there already any argument to say that inevitably the answer of the proof of concept will be negative? Cheers, On 21/11/14 21:27, Ross Gardler (MS OPEN TECH) wrote: We already evaluated the Bitergia offering - it is expensive and does not provide sufficient
Re: dashboarding incubator
On 23/11/14 20:41, Ross Gardler (MS OPEN TECH) wrote: Metrics do not provide this information, and indeed can detract from the community health issues. In looking at http://projects.bitergia.com/apache-cloudstack/browser/, I'm wondering if any meaningful metrics can be provided. Take for example, Mailing List metrics. 1,690 thread initiators 1,467 first replies 1,973 participants; Does that mean almost 10% of the threads did not receive a response? Or, as is more likely, those are part of existing threads, but due to defective email clients, appear to be new threads? My point is that while some metrics can provide indicators of something that needs looking into we need to ensure that the metrics do not become more important than community health. +1 Going back to that Mailing List metric, is it useful to know that roughly 1.4 people participate in each thread, when those metrics give no indication of whether or not the responses help the person initiating the thread? The danger I see is that providing official metrics a) provides a level of authority to the metrics which most newcomers are ill-equipped to evaluate an b) could lead to shortcut rules like the metrics must show there is X level of diversity/activity/volume/foobar replacing proper evaluation of the project and its community. Using user support for Apache OpenOffice as an example. My sense is that the general user population uses https://forum.openoffice.org/en/forum/ rather than us...@openoffice.apache.org. I'm aware of a non-Apache project where the only reliable support is IRC. The web-forum, mailing list, and other channels don't provide any indication of that factoid, though. A metric that just looks at mailing list activity falls short for projects where most interaction occurs on either web forums, or IRC channels. Metrics are useful indicators, but only when: * What they measure is clearly indicated; * What they don't measure is clearly indicated; * What their blind points are, is clearly indicated; * They measure what they purport to measure; Even with all those criteria, metrics will be misquoted, and otherwise abused, to push a point. jonathon * English - detected * English * English javascript:void(0); signature.asc Description: OpenPGP digital signature
dashboarding incubator
Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. What do you think? Cheers, [1] http://apacheconeu2014.sched.org/event/50a7178fa1d812e50e8c13e9bca8c7c1#.VG8vQnU3eis [2] http://projects.bitergia.com/apache-cloudstack/browser/ -- Sergio Fernández Senior Researcher Knowledge and Media Technologies Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernan...@salzburgresearch.at http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart. Additionally, there are free services that begin monitoring projects either on request or when they are noticed, and I am curious what we really hope to gain by doing it ourselves that a site like open hub doesn't provide. In example: https://www.openhub.net/p/npanday I do agree that things like bitergia provide much deeper sense of analytics, but wonderful if it's really necessary for our purposes. --David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
I agree with David. For example, many projects are now mirrored to github, and github has very good analytics: https://github.com/apache/incubator-calcite/pulse https://github.com/apache/incubator-calcite/graphs/contributors And, I don't have to log an INFRA JIRA to get them. Which makes both me and David happy. :) Julian On Fri, Nov 21, 2014 at 6:35 AM, David Nalley da...@gnsa.us wrote: On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart. Additionally, there are free services that begin monitoring projects either on request or when they are noticed, and I am curious what we really hope to gain by doing it ourselves that a site like open hub doesn't provide. In example: https://www.openhub.net/p/npanday I do agree that things like bitergia provide much deeper sense of analytics, but wonderful if it's really necessary for our purposes. --David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
One down side to the github provided analytics is that projects may be restricted in how they can make use of those analytics in e.g. presentations. Github has previously asserted that those analytic outputs are subject to a license grant from github and requested sign off in the form of a user agreement. I don't doubt that many many people make reference to them without bothering to check on the legal status, nor that many uses are probably within the bounds of fair use. But the ASF should probably keep the matter in mind. On Fri, Nov 21, 2014 at 11:54 AM, Julian Hyde julianh...@gmail.com wrote: I agree with David. For example, many projects are now mirrored to github, and github has very good analytics: https://github.com/apache/incubator-calcite/pulse https://github.com/apache/incubator-calcite/graphs/contributors And, I don't have to log an INFRA JIRA to get them. Which makes both me and David happy. :) Julian On Fri, Nov 21, 2014 at 6:35 AM, David Nalley da...@gnsa.us wrote: On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart. Additionally, there are free services that begin monitoring projects either on request or when they are noticed, and I am curious what we really hope to gain by doing it ourselves that a site like open hub doesn't provide. In example: https://www.openhub.net/p/npanday I do agree that things like bitergia provide much deeper sense of analytics, but wonderful if it's really necessary for our purposes. --David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Sean
Re: dashboarding incubator
Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
On 21 November 2014 20:44, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: dashboarding incubator
We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: dashboarding incubator
Note - I should say my comment below is in the context of a not-for-profit software foundation using their services. People in different use cases should look at their specific circumstances - obviously :-D Ross -Original Message- From: Ross Gardler (MS OPEN TECH) [mailto:ross.gard...@microsoft.com] Sent: Friday, November 21, 2014 12:27 PM To: general@incubator.apache.org Subject: RE: dashboarding incubator We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley da...@gnsa.us wrote: ...I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
On Fri, Nov 21, 2014 at 6:35 AM, David Nalley da...@gnsa.us wrote: On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. I am generally against us standing up our own service that does this. Wait! Who's talking about 'our own services' (AKA stuff that INFRA manages)? The conversation is very clearly around Sergio offering his help managing whatever needs to be managed. I do agree that things like bitergia provide much deeper sense of analytics, but wonderful if it's really necessary for our purposes. The only way to find out is to give it a try. Honestly, I'm still a little bit miffed by our inability to do deeper metrics. I do of course understand that I'm the first one to blame -- ain't got no cycles for that. This is precisely why Segio's offer is so useful. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
Sergio, I think there's no need to ask for permission in this particular case, unless you'd need to have INFRA help you obtaining information that is not publicly available. But then again, knowing what this information is would be useful anyway ;-) So yeah... go ahead! I'd like to help as much as I can, but I clearly can't drive it. Thanks, Roman. On Fri, Nov 21, 2014 at 4:28 AM, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. What do you think? Cheers, [1] http://apacheconeu2014.sched.org/event/50a7178fa1d812e50e8c13e9bca8c7c1#.VG8vQnU3eis [2] http://projects.bitergia.com/apache-cloudstack/browser/ -- Sergio Fernández Senior Researcher Knowledge and Media Technologies Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernan...@salzburgresearch.at http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org