Theoratically every penny spent should be measured wheter it was spent well, or was wasted, including salaries. Ideally you could find out all the direct and indirect investment in a given program. Since the latter is extremely hard (or impossible) to measure, it is always an estimated extra cost.
The emphasis here is not on the programs a meetup may trigger, but the meetup itself; its effectiveness. (it is a program itself) Regarding evalulation of programs through a meetup's perspective (if I understood your words correctly), there is no point going any deeper than the name of it, the type of it, and a note that it was"done", "in progress" , "failed" or "postponed" as of the date of publishing. Details should be carefully adjusted for all questions you wish to answer. 90% of evaluating an event is actually done through surveys collected from participants on site (!) and after a given period (somewhere between three to six month), by evaluating the answers and feedbacks you collect _proactively_ (a.k.a. "by asking") The rest 10% is an evaluation summary of the catering, the venue and the executing staff. Six month later a summary was published and an evaluation meetup was held before we started organizing the next similar event (what's content was heavily influenced by that that report). Cheers, Balázs 2013/5/14 Lodewijk <[email protected]> > I totally understand (or rather, I think I do). However, just like with any > project, also such calculations would be subject to cost/benefit ratios. > What information do you expect to win, and how much overhead does it > create? It sounds great to be able to put a number on things, but up to a > certain size, things will be mostly intangible anyway (enthusiasm, > inspiration etc.) so you will have little valuable output of such > measurements, while the costs are relatively high (they need to learn speak > your measure language, they need to understand concepts, document > thoroughly, measure etc.) and for small events that seems unreasonable to > me to expect it from them - such as with the article writing contests in > 'new' countries. > > However, this is of course different for large international events with > many participants. Anyway, I am glad to hear you're not trying to compare > different categories of events with each other. > > Lodewijk > > > 2013/5/14 Balázs Viczián <[email protected]> > > > I see a few misunderstanding here: I do not wish to compare them to each > > other, but to the previous one(s). At least I hope this is what I've > > written because this is what I was ment. So compare GLAM 2013 to GLAM > 2012, > > and GLAM 2011, ..., compare Brussels meetup with previous similar > thematic > > meetups and so forth. Create a breakdown about the costs and the > benefits. > > Aimed at GLAM? then the main thing to measue how did it boost GLAM > > cooperations. Is there any initiative that had been picked up or created > > upon the inspiration they got there? If yes, how much? What types? What > are > > their outcomes? Any other side goals (like international cooperations - > > like a new WLM participant who decided to join after discussions there)? > > Any other unexpected benefits? (like an out of the blue content donation > > from a participating G,L,A or M representative) etc. Easy to identify a > > dozen questions, plus if you start thinking it over, you'll find a dozen > > more that are as well pretty important, just not that much visible or in > > front. Count them and divide the total number with the total costs, and > > you'll get how much was invested in a single initiative there. > > > > If an idea was brought home by a chapter from that event (or the local > > ideas have been influenced by it), note even if locally it would cost > > almost nothing, on an absolute level, there is this cost/benefit ratio, > > both the chapter's and the event's, what you have to further divide it > > locally amongst the number of projects. If an example helps you, lets say > > you brought home 2 article writing contest ideas, both of them conducted > > and did cost nothing, except a few merchandise (total of 200USD) your > > travel and stay there should be added to the total costs, since without > it, > > these would likely never been conducted at you. Hence the absolute > numbers > > would include your flight and stay in London, divided equally in between > > the projects that meeting inspired. Got me? > > > > Balázs > > > > > > 2013/5/14 Lodewijk <[email protected]> > > > > > I think this is very true, and we could perhaps improve our procedures > > and > > > documentation in many ways. However, I do think it is important to > > realize > > > that you're comparing apples with oranges here. The meeting in Brussels > > for > > > example was on the topic of influencing legislative processes - a topic > > by > > > definition long term and hard to measure. The goals there would be > better > > > (copyright) legislation than without these initiatives - and rather > > > strategic. The bootcamp in DC was intended as a boost for volunteers, > > where > > > they would get a lot of information and experience in a short time: > > > capacity building. This is also long term, but much better to measure. > > And > > > finally there's the hackathon, which is much more short-term focused > (at > > > least partially), and will have very concrete results (but probably > zero > > > measured in new articles created). > > > > > > What I think is most important, is that for every meeting we have, we > > > identify desired outcomes. These outcomes can be very tangible, or > > > intangible - but they should always be defined, and the agenda should > be > > > built around it. > > > Based on these desired outcomes, you can estimate up front if the > meeting > > > is likely to be 'worth it'. If not, you can reduce the costs, increase > > the > > > outcomes (different setup) or cancel all together. > > > Finally, after the meeting a report (private or public - there should > > > always be a report or documentation) should be produced and if possible > > > published. In that report you should always return to these desired > > > outcomes, and see if they were met - and how/why not. > > > > > > But even then on the strategy side of things meetings are always hard > to > > > estimate. > > > > > > Of course you're totally right that there are expectations towards > > > participants of meetings. Not only because of the money invested, but > > also > > > because of the attention of the other participants. I definitely have > > been > > > in meetings where people literally fell asleep or played games during > the > > > meeting - and that is simply insulting to the other people in the room. > > So > > > yeah, if that is your mindset, perhaps it is better not to go at all. > But > > > then I am assuming good faith, and think that everyone will be going to > > > meetings with the best of intentions, and not simply to play tourist. > > > > > > Lodewijk > > > > > > 2013/5/14 Balázs Viczián <[email protected]> > > > > > > > 40k for a single meetup can be justified if the results worth that > much > > > > money. (I've already argued about this regarding Milan chapters > meetup > > > > btw). Providing a "more than basic" travel and accomodation can be a > > way > > > of > > > > appreciation as well for their work, what they do as volunteers, that > > > > should be calculated into the costs. > > > > > > > > Ain't these intrenational events' (not just this meetup's, but all > > > events') > > > > "success ratio" being measured by some way already? > > > > > > > > The cost/benefit ratio [1] is a pretty basic (and extreme important) > > > thing > > > > we like to calculate with here in Hungary about all of our > activities. > > > For > > > > example the number of articles created divided with the total costs > of > > > the > > > > article writing contest they've been created within gives a number we > > can > > > > work a lot with to improve cost-effectiveness. Seemingly very few > > > chapters > > > > doing anything similar (or not in a visible way) > > > > > > > > AffCom already measures itself in some ways in their reports, but > > > regarding > > > > other meetups, I've barely seen at least a basic followup or > aftercare > > > and > > > > especially not a detailed overall (measurement) report what is > usually > > > /at > > > > least in those commercial events I was involved with/ being published > > > after > > > > about six month of the last day of the event, and gives a detailed > > > summary > > > > of its pros and cons, dos and don'ts, successes and fails, overall > > impact > > > > (upon proactively collected feedbacks), etc. > > > > > > > > There were plenty of international events already this year (Brussels > > > > meeting about EU policies, Milan, London glam, Amsterdam hackathon > that > > > are > > > > coming in my mind right now from 2013, and this is just the first 4 > > month > > > > (only 1/3rd of the year) and not the full list!) most of them with no > > or > > > > very low visible results yet (ok they need some time to evolve, but > > > > regarding events in 2012, I barely read anything to remember nor any > > > > followups or summares, reports). Compared to the "GLAM camp" in the > US > > > > recently, it seems definitely true. The latter looks like a very good > > > > example of a beneficial meetup, having a lot of potential and it > seems > > > > there is a chance that it will be followed up by WMDC and other > > > > participating parties well after the event. Why to have a long term > > > > followup? To give a definite answer wheter those potentials actually > > > > resulted in anything at all (was there any "real benefit") or it was > > > just a > > > > very good mooded, fun and positive, but totally fruitless event > (wasted > > > > money from the movement's POV) > > > > > > > > I see compared to 2012 costs going up without any visible rise in > > > > effectiveness or more worse, a decline in it. This is solely based > upon > > > > what I can (or can not) read about them on meta and other places, > like > > > the > > > > comments here. Wikimania 2014 was the first event ever where this > thing > > > was > > > > taken seriously, but rather on the cost cutting side, than on the > > > > effectiveness improving side > > > > > > > > It would be great to see detailed measurements of these events, like > > how > > > > many new projects or international cooperations (or whatever it aims) > > > were > > > > boosted/inspired by the given thematic meetup up until the next > similar > > > > meetup. If that number is X, while the costs were Y, and X/Y does not > > > look > > > > good, than you can start thinking how can you improve X without > > > expanding > > > > Y (or even lowering it) to get a much friendly ratio, thus creating > an > > > even > > > > more fruitful (better quality) event next time. The best would be a > > > > detailed breakdown, like main goals, side goals, unexpected or > "extra" > > > > things that that meeting had inspired/boosted/hosted/etc. > > > > > > > > Note, there ain't no such thing as free lunch [2] > > > > > > > > Cheers, > > > > Balázs > > > > > > > > PS: WMHU has 68k budget for 2013. > > > > > > > > [1] http://en.wikipedia.org/wiki/Benefit%E2%80%93cost_ratio > > > > [2] > > > > > > http://en.wikipedia.org/wiki/There_ain%27t_no_such_thing_as_a_free_lunch > > > > > > > > > > > > > > > > 2013/5/14 Lodewijk <[email protected]> > > > > > > > > > I don't think it is trivial either, and having a discussion is > fine. > > > > > However, the bigger discussion is perhaps more relevant - because > > > AffCom > > > > is > > > > > simply following WMF policy here, which is in place for many > > employees, > > > > > board members (and others? not sure). > > > > > > > > > > So I think there are two relevant discussions to be had: First of > > all, > > > > > whether there should be an AffCom meeting at all. Fair question of > > > > course. > > > > > (analogous you could wonder if certain chapters should really send > a > > > > > representative to Wikimania from their budget, whether certain > > > employees > > > > > really should be there and whether all chapters should be present > at > > > the > > > > > Chapters meeting - all fair discussions to be had in their time, > > too). > > > > The > > > > > second relevant question is, in my opinion, whether the WMF travel > > > policy > > > > > is good, proportional etc. This policy has mostly received > criticism > > > from > > > > > two sides - some think it is too elaborate, and WMF should get > 'less > > > > > luxury' (again fair discussion, but we should then focus on the > whole > > > > > question) - another criticism is that it should be equalized for > all > > > > > Wikimedia-sponsored trips, including individual engagement grants, > > > trips > > > > to > > > > > the chapters meeting etc. When this question was brought up last > > time, > > > I > > > > > believe it was Sue who mentioned that this would simply result in > > much > > > > less > > > > > travel and participation. Again, a fair question. > > > > > > > > > > Personally, I feel that WMF Committee (and board) members should > not > > be > > > > > treated to a lower standard than staff members, simply because they > > are > > > > not > > > > > being paid for their work. But maybe I'm the only one in thát > opinion > > > > > though... > > > > > > > > > > Lodewijk > > > > > > > > > > > > > > > 2013/5/14 Andrea Zanni <[email protected]> > > > > > > > > > > > On Tue, May 14, 2013 at 12:30 AM, David Gerard < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > https://en.wikipedia.org/wiki/Parkinson%27s_law_of_triviality > > > > > > > > > > > > > > > > > > I don't really think this is a triviality. > > > > > > Cents and single dollars of chapters are weighted and analyzed, > > > > > > FDC and GACs nitpicks and their goal is to assure that donor > moneys > > > is > > > > > not > > > > > > wasted. > > > > > > We all know that there are issues on that and a lot of chapters > had > > > to > > > > > work > > > > > > hard to be able to draw their budget, and receive their money. > > > > > > > > > > > > 40'000 $ for Wikimania travels are a lot, especially compared to > > the > > > > > travel > > > > > > budget of chapters. Period. > > > > > > I would assume that WMF and the Board budgets should be > reasonably > > > > > > proportional to chapters ones. > > > > > > Maybe I'm the only one, though. > > > > > > _______________________________________________ > > > > > > Wikimedia-l mailing list > > > > > > [email protected] > > > > > > Unsubscribe: > > > https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > > > > > > > > > _______________________________________________ > > > > > Wikimedia-l mailing list > > > > > [email protected] > > > > > Unsubscribe: > > https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > > > > > > > _______________________________________________ > > > > Wikimedia-l mailing list > > > > [email protected] > > > > Unsubscribe: > https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > > > > > _______________________________________________ > > > Wikimedia-l mailing list > > > [email protected] > > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > > > _______________________________________________ > > Wikimedia-l mailing list > > [email protected] > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > _______________________________________________ > Wikimedia-l mailing list > [email protected] > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > _______________________________________________ Wikimedia-l mailing list [email protected] Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
