Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello, Here's my new report. I've finally published my code and created some samples for all of you. Youhou! :) Any feedback is very welcome. I’m happy to report that the first version of the software is now public. I have set up the app with some graph samples here [1], and hosted the code in Launchpad [2]. Above each graph, there is the code snippet that is being used to query the database. There are many syntax rules (most of them are already working) that allow a rich vocabulary for the expression of queries. You may see the BNF language description in the section ‘How to use them’ in the documentation [3]. The JS library that renders the graphs is jqplot. It’s not difficult (even for someone with minor experience to Django) to use another plotting library. The data structure returned by the extraction tag is very flexible and can be used in any manner. See the section ‘Configuring the output’ in the documentation [3]. My next step will be to implement the coalescor which (optionally) merges database entries in order to limit the growth of the number of entries stored in the database Thanks for reading and testing. I would highly appreciate any comments. [1]: http://sophron.latthi.com/metrics/ [2]: https://code.launchpad.net/~sophron/mm3-metrics/trunk [3]: http://bazaar.launchpad.net/~sophron/mm3-metrics/trunk/view/head:/MM3_metrics_display/templatetags/__init__.py -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] Implementing the IArchiver
Hello everyone, For my metrics app (GSoC project), I'm currently using a MM3 simulator for testing purposes. Proceeding with the development, i think it's time to implement a connection with a real MM3 environment. For the current version of the app, the only information i need is who posted what to which list on what date. So, as we have already discussed, the IArchiver should be the origin of the interface. But i know nothing on how to implement this. How will i add my app to the active IA modules? Is there any code that extends the IArchiver and will help me understand how it works? (I guess HK extends it to inject the messages into the archiver.) Thanks in advance, -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Greetings, Following a new report. I've finished a coalecor daemon and i wrote some generators for KittyStore too. I'm now into establishing a real MM3 connection. The first version of coalescor is ready! Coalescor is a custom Django admin command that is responsible for maintaining the number of entries in the database at a reasonable level. If the number of entries in the database is not a concern, its use is completely optional. There are five parameters which determine the behavior of the coalesence: DAILY_DETAIL_RETAINED, WEEKLY_DETAIL_RETAINED, MONTHLY_DETAIL_RETAINED and ANNUAL_DETAIL_RETAINED. Each one of them represents the number of grains for which that level of detail is maintained. The daemon coalesces only the new data from its last operation date and it was designed in a manner not to be memory intensive. The last two days i also worked on generators. I had already created a script that generates metrics from a mailbox. There is a KittyStore API availabe, so it wasn’t hard to also generate metrics from SQL or MongoDB. -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] Event notifications
Hello everyone, There is a mechanism that is not in place yet and thus some features of my metrics app can't be used. I would like to mention it as feature request. Of' course i'd like to help on its implementation. It looks like the only notification the core provides is that of new postings (through the IArchiver interface). It would be great if core could provide notifications for more events (like someone subscribed or a new thread started) that also carry more information (for postings: number of recipients and message size). I think most of this information would be helpful for other plugins (like HyperKitty) as well. So, what i suggest is basically a message bus that components can register for notifications. I'm not sure if this has been discussed before and is already in the plans. Thanks for your time, -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Implementing the IArchiver
Hi, I was wondering what is the best way to test an IArchive implementation. Are there any built in test modes that will allow me to send sample messages? I only care about the processing of the messages through my IArchive implementation rather than actually sending them out. Thanks again, -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello, This is probably my last report. Any comments about the project are very appreciated. :) GSoC is almost over and i just pushed all the work i’ve done these three months. It was one of the greatest experiences and i’ve learned a ton of stuff. I feel proud of my work and i can’t wait to see it used in Mailman working environments. For those who don’t know what is all about, a live demo of the app exists here [1]. The code is hosted in Launchpad [2] and if you are interested to learn more, i would suggest you start reading the documentation, from the installation [3], to metrics collector [4] and metrics display [5]. To learn even more about the functionality of the app, please check the tests that lie in the ‘test app’ directory for each submodule. I would like to thank Wacky (a great mentor) and the rest of Mailman developers for their help. Mailman has a great community and of course i’ll stay here for anything regarding improvements of my app or anything else Mailman-related. [1]: http://sophron.latthi.com/metrics/ [2]: https://code.launchpad.net/~sophron/mm3-metrics/trunk [3]: http://bazaar.launchpad.net/~sophron/mm3-metrics/trunk/view/head:/installation.rst [4]: http://bazaar.launchpad.net/~sophron/mm3-metrics/trunk/view/head:/MM3_metrics_collector/__init__.py [5]: http://bazaar.launchpad.net/~sophron/mm3-metrics/trunk/view/head:/MM3_metrics_display/templatetags/__init__.py -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Event notifications
On Wed, Sep 05, 2012 at 09:38:00PM -0400, Barry Warsaw wrote: > I've taken a mostly organic approach to adding new events, adding them only > when needed internally. I'm open to adding new event notifications as > necessary to support add-ons or new internal functionality. The way to > request this is via the Launchpad bug tracker. Thanks for the response. I requested this via Launchpad [1]. [1]: https://bugs.launchpad.net/mailman/+bug/1047286 -- George Chatzisofroniou sophron.latthi.com signature.asc Description: Digital signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] [GSoC 2012] Candidate on 'Integration of (existing) search code into Mailman archives'
Hello Mailman Developers, My name is George Chatzisofroniou, i'm 20 years old and i'm an undergraduate student in the Department of Informatics at the University of Piraeus (Greece). Ι have really good previous experience with Mailman. This is because i use it for managing mailing lists for almost three years. I have also developed, with a friend of mine, MailmanStats [1], a Python software that outputs statistics for a mailing list based on Mailman. I think this implements the 'metric' idea in some way. I would like to know your opinion about MailmanStats. I'm sending this mail to inform you about my will to be part of Mailman Development team starting by Google Summer of Code 2012. The idea that excites me more is the 'Integration of (existing) search code into Mailman archives'. I think it is better to be developed on Mailman v3 rather than v2. I realize the significance of a feature like this. Many times before, i've got through the archives to search for a specific thread, so an addition like this would be great! As another student mentioned this idea is kinda small for the whole summer, so if there is time left i could integrate my MailmanStats [1] software into Mailman and/or build CSS styles for the web UI. Please tell me what you think. I'm also on IRC by the name sophron. Thanks, [1]: http://mailmanstats.latthi.com/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Candidate on 'Integration of (existing) search code into Mailman archives'
2012/4/3 Terri Oda : > Hi George, > > Your MailmanStats project looks great and would totally fit with what we > have in mind for stats, though I'm guessing the hyperkitty team has some > much more extensive work in mind making use of post ratings, tags, etc. > > If you're putting together your proposal now, do feel free to mention both > projects as sources of interest. Since you already have the stats code > available, it might be possible to toss the integration in there after doing > some other work. Normally I worry about students biting off more than they > can chew, but given your prior experience with Mailman and the fact that you > already have the basic code, you can make a case for being able to package > up that code and contribute it in a week or two our of your summer if you're > ready for a code review. > > > Terri > > PS - For further advice regarding search projects, see my previous post to > mailman-developers. > > Thanks for your respond Terri, I thought about it quite a lot. Eventually, I think it is better to implement only the Metric idea (since i already have the base code) by integrating my software into Mailman. My previous experience with Mailman and the fact i have done some work already will make a more awesome result. I'll send my proposal in the next hours. I'll appreciate any feedback. Thanks, -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Congratulations to our GSoC Students!
Hello everyone, My name is George Chatzisofroniou, i'm 20 years old and study Computer Science. I'm a volunteer sysadmin in my university's labs and i love to code. In my free time, i like to walk or play extreme sounds with my band. In this summer, i'm going to implement Metrics on Mailman 3.0. Right now, i'm going through the Django documentation and waiting for the appropriate mechanisms to be in place so my app can collect the right data. I keep a GSoC diary here [1]. Any advises are very welcome! Thanks, [1]: http://sophron.latthi.com/gsoc-mailman/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] [GSoC 2012] Metrics
Hello, I'm sending a report of my progress for the last two weeks (containing an outline of my project). The following report is also posted in my blog [1]. -- The last two weeks i’m into learning the Django web framework in more detail. I’m also thinking to rearrange the GSoC schedule a bit. I’ll start writing code on the bonding period, so i can have more time during my university’s exams (starting on half of June). Finally, i came up with an outline of my project. Here it is: Settings The Django app should handle some configuration parameters, like: - Maximum number of the subscribers of a mailing list the user wants to be shown in the charts - A multi-state variable (None, abbreviated, full) for masking email addresses at the results (we don’t want the emails to be spammed) Apart from these, some parameters to establish a connection to the message store (such as host or port) will be added. Models In order to store statistical data, the app will use some Django models: - Author – This model will store specific information for each subscriber such as total posts, total threads started, date of last message, subscription date, average of posts sent per day - MailingList – General information about the mailing list will be stored here such as total posts, total threads and two foreign keys pointing the month and year models - Month – This model will store total posts and threads for each month - Year – Similar to month model, this one will store total posts and threads for each year Views To display the metrics some views will be used: - A generic view for the mailing list and the top subscribers - A more detailed page about the participation of a specific subscriber The graphs will be generated using the PyChart library. (Custom tags are the way that the app will embed the graphs) Interface to the MM core The app needs to interact with the Mailman core. I think the best idea is to implement a message bus that will send a notification every time a message is sent (the same way an archiver works). Based on this notification -which will carry the information about the sent message- the app will be ready to update the counts. Although, in some cases (eg the app is installed after an existing archive or there was an unexpected crash), the message bus should deliver (triggered by a button) more than one notifications (for the whole archive) to the app in order to initialize/recover the metrics. In those cases, the models will be initialized and the generation of the metrics will start over. -- Any comments are highly appreciated. [1]: http://sophron.latthi.com/gsoc-mailman/?p=12 -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello Stephen, > On Thu, May 3, 2012 at 8:04 AM, George Chatzisofroniou > wrote: > >> I’m also thinking to rearrange the GSoC schedule a bit. I’ll start >> writing code on the bonding period, > > It's up to your coding mentors, but it's generally not a good idea to > try to move up the coding stage by too much. Make sure you have a > clear spec before you start coding anything, and at least a rough > sketch of a design. Without those two pieces, there's no standard to > evaluate progress, or whether your code is doing the right thing. > As wacky mentioned i'm not planning to skip any step of the process. The idea is to press myself a bit more now, so i can have some more time to study later. I should have explained myself better. >> so i can have more time during my >> university’s exams (starting on half of June). > > I'd say just take the time as needed, after negotiating with your mentors. > >> Settings >> >> The Django app should handle some configuration parameters, like: >> >> - Maximum number of the subscribers of a mailing list the user wants >> to be shown in the charts > > I don't know how feasible it would be to implement, but if you're > willing to use AJAX, you could simply build up bar and line charts on > the fly, adding "slices" of say 5 users at a time on the right side of > the chart in progress. That might be cool. It could be set in a > "viewport" and scrolled if it gets too big for the viewport. Yes, AJAX can do the job here. I will implement the AJAX bar as you described, although i think it's better to primarily have a non-Javascript output. > > >> - Month – This model will store total posts and threads for each month >> - Year – Similar to month model, this one will store total posts and >> threads for each year > > It's not clear to me why year views can't be generated as an aggregate > of monthly data? This would allow years to start with arbitrary > months without too much redundancy. Generating the year views from monthly data is some more calculations while displaying the metrics. That's why they should be stored somewhere. > >> Interface to the MM core >> >> The app needs to interact with the Mailman core. I think the best idea >> is to implement a message bus that will send a notification every time >> a message is sent (the same way an archiver works). > > Why not just use the iArchiver interface? > > I think it should be possible to have this be a standalone app serving > the list itself, or be hosted by a particular archive, which wouldn't > need to be the list host (or even run by the same organization). Yes, i agree. The iArchiver should be the origin of the interface. Thank you for your feedback, -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
On Fri, May 4, 2012 at 4:06 PM, Stephen J. Turnbull wrote: > OK, but these are pretty cheap calculations if I understand Django's > design correctly. Yes, these are cheap calculations. Although, the number of calculations is based on the years of list's existence (e.g. a 10 year list would need 10 calculations). > Which design do you think is simpler, in the context of your overall > plan? I think it's simpler to just add a year model. I would like to hear more opinions on this issue. -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello Geoff, On Sat, May 5, 2012 at 12:17 AM, Geoff Shang wrote: > > I would just like to restate my plea for the ability to manage Mailman > without needing javascript. Note that I'm not saying don't use any, I'm > merely asking for it to be possible to use without it. > > I use lynx as my primary browser, and while general web surfing can be a bit > tricky these days, it's very quick and useful for doing admin-type things, > and I for one would like to be able to continue to do so. Certainly I have > no problems using Mailman 2'.1's interface with lynx, and have been doing so > for many years. > > If my memory serves me correctly, this was generally felt to be a reasonable > request when I first mentioned it. I'm sure it's in the archives somewhere. > Yes, i also agree with you. That's why i mentioned a "primarily non-Javascript output". -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
On Sat, May 5, 2012 at 11:26 PM, Barry Warsaw wrote: > Do you know exactly what kind of information you need? With my current implementation plan i only need author and date of the message to export my metrics. > The suggestion to use the IArchiver interface is convenient, but might not > tell you much other than who posted what to which list on what date. It won't > tell you things like the number of recipients, how long that message took to > deliver, if there were any failures, etc. Some of the things you mentioned sound pretty interesting and it would great to be on the metrics. > It would certainly be possible to hook in a zope.event notification with those > metrics for each successfully posted message. From there, a plugin could > register a subscriber that put the event data on a message bus. Or you could > grep the logs. :) Hooking the zope.event notification sounds good. > Anyway, I think it would be useful to improve the support for this in the > core. I'd like that! I'll come up with another more-detailed report this week. Thanks Barry! -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello, I'm posting some additional details of my outline for this week. For an easier reading please read it in my blog [1]. -- Following a “top-down design” methodology, i came up with additional detail of my design concept. What information is stored? As i have already mentioned the information is stored in Django models. Author This model represents an author of the mailing list. It mostly keeps track of the number of postings and number of threads started. It has the following fields: mail – CharField (Mail address of the author) totalmails – IntegerField totalthreads – IntegerField firstmsgdate – DateTimeField lastmsgdate – DateTimeField MailingList This model counts the total number of postings and threads started. totalmails – IntegerField totalthreads – IntegerField Month This model associates the author and the mailing list with each month. author – ForeignKey month – CharField postscount – IntegerField threadscount – IntegerField mailinglist – Boolean (if this is true it corresponds to the whole mailing list) Year This model is similar to month. It has a year field instead of a month field. What happens when a new email is posted to the list? - The IArchiver interface will grab the email and send it to the Metrics class. - The Metrics class will update the models: * If the author is sending to the mailing list for the first time, A new Author row will be created to the database. Otherwise, the Posts field will increase by one. - Also, the corresponding row of the two foreign fields (Month and Year) will also increase by one. * The Posts field of the Mailing List model (as well as the the related rows on the Month and Year models) will increase by one. What happens when a user requests the Metrics page? Django will generate a page based on the current models that outputs: - On top, general metrics about total authors, total mails and total threads, based on the MailingList model and below some charts (AJAX based) that represents this information - The authors of the mailing list ordered by the number of their total posts, after the general metrics. Each user will correspond to a more specific page with information like date of first and last message on top and a chart on his overall activity below. What will happen if the app is installed after an existing archive or something crashes on the system? How do the metrics are being recovered? For this case there should be a special ‘Generate’ button in the admin page. When this button is pressed, the class ‘Generate’ will be triggered and: - The Django models will be initialized (the metrics will go back to zero). A progress bar will inform the administrator that the operation is being held. - All the messages of the archive will be parsed by performing a direct Python call to the IArchiver. Another instance to the IArchiver will grab any mails sent while the parsing is going on and store them to a text file. - The metrics will be generated from scratch. - The mails stored to the text file will be added to the metrics - The administrator will be informed with a success message when the process is over. -- [1]: http://sophron.latthi.com/gsoc-mailman/?p=27 -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
The following document is the lowest level of my design concept. You may also read it in my blog [1]. Of course, comments are very welcome. -- Models In order to store statistical data, the app will use some Django models: Author This model represents an author of the mailing list. It mostly keeps track of the number of postings and number of threads started. It has the following fields: - authorid – IntegerField - authormail – CharField - totalmails – IntegerField - totalthreads – IntegerField - firstmsgdate – DateTimeField - lastmsgdate – DateTimeField MailingList This model counts the total number of postings and threads started. - totalmails – IntegerField - totalthreads – IntegerField Month This model associates the author and the mailing list with each month. - author – ForeignKey - month – CharField - postscount – IntegerField - threadscount – IntegerField - mailinglist – Boolean (if this is true it corresponds to the whole mailing list) Year This model is similar to month. It has a year field instead of a month field. Views To display the metrics the Django template system will be used. To output the charts i will create some custom tags. The three following views will be used: - General page – On top, there will be general metrics about total authors, total mails and total threads and below three charts (AJAX based) that represent number of posts per author, number of threads per author and mailing list’s yearly usage. Even below there will be a number of charts (equal to the number of years of list’s existence) that output monthly usage. At the end, there will be tabular data representing the authors of the mailing list along with their number of posts, number of threads started and the date of their last post. The user will be able to order the tabular data (alphabetically, ascending/descending on number of posts, number of threads, date of last message) by clicking on the table’s headings (Mail, Mails Sent, Threads Started, Last Message). - Author page – Each user will have his own page with his own metrics. On top, there will be the email of the author, number of posts, number of threads started and the dates of first and last message. Below there will be monthly usage charts for each year the user is subscribed to the mailing list. Django Admin page – A ‘Generate’ button will be added to the Django admin page. Settings The Django app should handle the following configuration parameters: - Host – Message store data host - Port – Message store data port - Masking – A multi-state variable (None, abbreviated, full) for masking email addresses at the results (we don’t want the emails to be spammed) Interface to the Mailman core - Metrics class – When a new post is sent, the Metrics class will receive it through the IArchiver interface. The Posts field of the Mailing List model (as well as the the related rows on the Month and Year models) will increase by one. If the author’s email is not in the database, it will query the mailman core database with the email, grab the author’s id and a new Author row will be created. Otherwise if the author is already in the database, the Posts field and the two foreign fields (Month and Year) will increase by one - Generate class – When the ‘Generate’ button on the Admin page is pressed: * The Django models will be initialized (the metrics will go back to zero). A progress bar will inform the administrator that the operation is being processed. * All the messages of the archive will be parsed by performing a direct Python call to the IArchiver. Another instance to the IArchiver will grab any mails sent while the parsing is going on. * The metrics will be generated from scratch. * The administrator will be informed with a success message when the process is over. [1]: http://sophron.latthi.com/gsoc-mailman/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello Patrick, On Fri, May 18, 2012 at 12:09 AM, Patrick Ben Koetter wrote: > Like with most statistical data I mostly see the figures being used to give > statements on quantity - top poster, number of threads etc. Do you think it > would be possible to also make some statements on quality? The metrics will primarily extract the activity of a mailing list and its users. It is possible to emphasize on the quality of the posts but i think this is a different app (but it interacts with the Metrics one). If the users were able to rank the posts through the archiver it wouldn't be hard to indicate one user's offering to his community. This looks like the first idea on this page [1]. > P.S. > Do you also plan to deliver a tool that analyzes a mailing list archive in > order to gather your statistical data? Having the statistical data might be a > good reason for people to upgrade their MMx installation to MM3. Yes, there will be a special Generate button in case of an existing archive or in case of a system crash. [1]: http://blog.linuxgrrl.com/2012/03/13/mailman-brainstorm/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hello Stephen, On Sat, May 19, 2012 at 09:12:03PM +0900, Stephen J. Turnbull wrote: > I don't see why. I would think quality metrics would be usefully > presented via the same application as quantity metrics. It would be > interesting to correlate quality and quantity, for example. What i was trying to say is that the quality metrics need some designing on a "posts rating system" first. Of course they will be presented by my app. > Do you mean to say "this is out of scope of my project?" As much as > I'd like to see quality metrics provided, I'd have to agree with you > that it's out of scope of your project (maybe you could do one or more > quality metrics on a time-permitting basis at the end of the period). Yes, i think it is out of my GSoC project, but i would like to implement this after finish with quantity metrics this summer. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hi, Here's my report. As always, you can also read it in my blog [1]. -- The coding period has started. I’ve already learned a lot of stuff! Although, I intended to have more work done by now. That’s why i’m going to speed up in the following days. Here’s what i’ve done so far: - I documented the components of the app. I’ve documented storage of counts, parsing of custom tags and generation of HTML output, basic interfacing with mailman, coalescor (a daemon that cleans up the database) and generator (that generates the metrics from scratch). - I coded the most of my models using polymorphism solutions. - I’ve described the syntax of my custom tags. The syntax is defined by the BNF Language description below: ::= ‘{%’ ‘GRAPH’ ‘%}’ ‘{%’ ‘ENDGRAPH’ ‘%}’ := ‘{%’ ‘TABULATE’ ‘%}’ ‘{%’ ‘ENDTABULATE’ ‘%}’ ::= | ‘, ‘ ::= | ‘BY’ ::= | ::= ‘DAILY’ | ‘MONTHLY’ | ‘ANNUAL’ | (* Omitted. Default value: DAILY *) ::= ‘posts’ (* The number of total posts *) | ‘threads’ (* The number of total threads started *) ::= ‘subscribers’ (* The number of total subscribers *) ::= ‘FOR’ | ‘FROM’ ‘TO’ | ‘SINCE’ ::= ‘LAST MONTH’ | ‘LAST DAY’ | ‘YESTERDAY’ | ‘TODAY’ | ‘THIS MONTH’ | ‘THIS YEAR’ | DAYS | MONTHS | YEARS ::= ‘YESTERDAY’ | ‘START OF THE MONTH’ | ‘START OF THE YEAR’ | ::= ‘TODAY’ | ‘NOW’ ::= -MM-DD (* International format defined by ISO (ISO 8601) *) ::= | (* Omitted. Default value: Ommited *) ::= ‘WHERE’ | ‘AND’ ::= ‘STARTSWITH’ | ‘IS’ | ‘ENDSIN’ ::= ::= -- [1]: http://sophron.latthi.com/gsoc-mailman/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [GSoC 2012] Metrics
Hi everyone, Here's my new report [1] just before the midterm evaluation. --- Ι am before the midterm evaluation and i’m having fun coding and learning. The project has reached the first version and achieves most of the targets set. Specifically: Metrics store is ready and it is designed in a way to make it easy to add new metrics. Every metric is either an event counter (eg posts sent) or a measured level (eg number of total subscribers) and it is stored in the database representing an interval of a granularity. Special methods that retrieve counts for the specified interval or tally the events with their effects (like a message sent or a new subscription) are also completed. Most of the graph tags are completed. In order to make it easy for the designer to output the metrics he wishes, i have created a language and its parser. For example, if the designer wishes to output the data of monthly number of posts for the last two years, he can easily extract these counts as follows: “ {% EXTRACT AS graph1 %} posts MONTHLY FOR 2 YEARS {% ENDEXTRACT %} “. The whole syntax of tag’s language is defined by a BNF Language description (you may find an earlier version of this language in a previous post [2]). Usually, the designer will add a template specification for the graph tag and the returned values will be placed in this context, but is is also possible to customize the way this context is presented or create his own template (For example, he can easily change the date format or publish the metrics through a table), since the ’EXTRACT’ tag actually returns a complex data structure that can be used in any manner. However, some templates are already created for the designer, with the most significant to be the one that outputs the graphs. I used jqplot library [3] (minimal and fast rendered graphs) to render the graphs. So, in the previous example, to build the graph from the extracted data, the designer simply has to add the following to his template: “ {% include “MM3/line_graph.js” with dataset=graph1 title=’List Activity’ %}”, after loading the appropriate plotting libraries (there are ready templates that can be included to do that). For both metrics store and graph tags, i have already created a number of tests. But i still need to add even more. There are some things that have not need be done yet and i’ll work on them in the following days. I need to complete the implementation of the language (there are some rules that my parser does not handle yet). I also want to create a template for bar graphs as well, design a coalescor that will maintain the number of entries in the database at a reasonable level and finish my generator that will generate the metrics from scratch (a very first version is done). Last, i need to use IArchiver to make a real connection with MM3 core (i currently use a simulator for testing purposes). --- [1]: http://sophron.latthi.com/gsoc-mailman/?p=70 [2]: http://sophron.latthi.com/gsoc-mailman/?p=62 [3]: http://www.jqplot.com/ -- George Chatzisofroniou sophron.latthi.com ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9