Re: [Server-devel] [XSCE] sugarstats and xovis
On Thu, Jul 3, 2014 at 2:26 AM, Martin Dluhos mar...@gnu.org wrote: On 06/27/2014 02:10 PM, Martin Abente wrote: Hello Martin! Do you have documentation on how XOvis pipeline works? I would like to see how it works after the data is store and the visualization is generated. Here is a simple graphic that illustrates the workflow: http://i.imgur.com/Xhy1Vx3.jpg XOvis is a Couch application, which means that it's also stored in the db along with the journal metadata. It queries the db using map-reduce queries, which you can see here: https://github.com/martasd/xovis/blob/master/lib/views.js Writing a tool to export the metadata from Harvest-server to a compatible subset in the visualization-ready format of XOvis, would be cheaper in development cost, but more expensive in running time (with large datasets). The other way around, would be to allow XOvis read directly from Harvest-server MySQL format. CouchDB has two distinct advantages for XOVis. 1) It allows for eventual consistency which helps in syncing data from the school to a central location using XS or XSCE. I believe this isn't relevant for AU/Harvest because they don't use a school server, and the XOs send metadata directly. 2) CouchDB does the aggregation through mapreduce. This is powerful, and withour couchDB (or mongodb etc) you'll have to write the aggregation yourself and/or get MySQL to do it. The visualization may or may not be stored in the DB. It just happens to be one more convenience of CouchDB. cheers, Sameer Since transferring metadata into the db needs to be done only once, running time has never been a major concern. How large are the datasets you've collected on the Harvest-server? Cheers, Martin ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
On 06/22/2014 08:33 PM, Sameer Verma wrote: On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody t...@timmoody.com wrote: does xovis supersede sugarstats? From what I understand, sugar-stats runs at the dbus level as opposed to the Journal metadata, which pulls data from the Journal backups. xovis relies on data that it can get as json, pushed into couchdb. So, while right now xovis uses json from Journal backups, it could be modified to do so from the RRD files that sugar-stats produces. That said, the data collected by sugar-stats is different, and needs to be filtered accordingly before it can be used sensibly by xovis. That's exactly right. Using sugar-stats data in XOvis was actually one of the things I began exploring while in Nepal, but didn't implement prior to my departure. I think adding that capability to XOvis would be very very useful. Martin ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
On 06/24/2014 07:20 PM, Sameer Verma wrote: On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente martin.abente.lah...@gmail.com wrote: Hello Sameer, The harvest-client serializes (using json) a subset of journal-objects metadata and then sends it to the harvest-server, which is basically a web-service plus a MySQL back-end. Great! Then we can perhaps add a method to process_journal_stats.py in XOVis to support Harvest. XOvis visualization relies on journal metadata being stored in couchdb in json, so it should be relatively straightforward to add a method to process_journal_stats.py to read data generated by harvest-client and store it in the db. Martin ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
Hello Martin! Do you have documentation on how XOvis pipeline works? I would like to see how it works after the data is store and the visualization is generated. Writing a tool to export the metadata from Harvest-server to a compatible subset in the visualization-ready format of XOvis, would be cheaper in development cost, but more expensive in running time (with large datasets). The other way around, would be to allow XOvis read directly from Harvest-server MySQL format. There are other options for integration too, but ultimately depends on how XOvis pipeline works :) On Fri, Jun 27, 2014 at 5:09 AM, Martin Dluhos mar...@gnu.org wrote: On 06/24/2014 07:20 PM, Sameer Verma wrote: On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente martin.abente.lah...@gmail.com wrote: Hello Sameer, The harvest-client serializes (using json) a subset of journal-objects metadata and then sends it to the harvest-server, which is basically a web-service plus a MySQL back-end. Great! Then we can perhaps add a method to process_journal_stats.py in XOVis to support Harvest. XOvis visualization relies on journal metadata being stored in couchdb in json, so it should be relatively straightforward to add a method to process_journal_stats.py to read data generated by harvest-client and store it in the db. Martin ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente martin.abente.lah...@gmail.com wrote: On Sun, Jun 22, 2014 at 9:43 PM, Sameer Verma sve...@sfsu.edu wrote: On Sun, Jun 22, 2014 at 6:12 PM, Walter Bender walter.ben...@gmail.com wrote: On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma sve...@sfsu.edu wrote: On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody t...@timmoody.com wrote: does xovis supersede sugarstats? From what I understand, sugar-stats runs at the dbus level as opposed to the Journal metadata, which pulls data from the Journal backups. xovis relies on data that it can get as json, pushed into couchdb. So, while right now xovis uses json from Journal backups, it could be modified to do so from the RRD files that sugar-stats produces. That said, the data collected by sugar-stats is different, and needs to be filtered accordingly before it can be used sensibly by xovis. And there is the Harvest system, which for the most part relies on data in Journal. Yes, in fact we had conversations with Martin Abente about Harvest, and the possibility that the Harvest data could be serialized (json) and pushed into xovis. Are there any samples of Harvest data that we can look at some place? Hello Sameer, The harvest-client serializes (using json) a subset of journal-objects metadata and then sends it to the harvest-server, which is basically a web-service plus a MySQL back-end. Great! Then we can perhaps add a method to process_journal_stats.py in XOVis to support Harvest. I need to update this documentation [1] (will do this tomorrow early), but you can get an idea of what fields are used. If you need an example of the serialized format for the client-server communication you can look at [2]. The bits that handle the serialization process can be found here [3], which is meant to be read. OK. Will take a look. Sameer Regards, Martin. Refs: 1. http://wiki.sugarlabs.org/go/Harvest 2. https://github.com/tchx84/harvest-server/blob/master/test/data/data.json 3. https://github.com/tchx84/harvest-client/blob/master/extensions/webservice/harvest/harvest/crop.py Sameer -walter We have RRD files from Bhagmalpur, India, but we haven't done much with it as yet. I understand that AU may have a boatload of sugar-stats data as well, but not sure if they got around to doing anything with it. cheers, Sameer -- Sameer Verma, Ph.D. Professor, Information Systems San Francisco State University http://verma.sfsu.edu/ http://commons.sfsu.edu/ http://olpcsf.org/ http://olpcjamaica.org.jm/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel -- Walter Bender Sugar Labs http://www.sugarlabs.org -- Sameer Verma, Ph.D. Professor, Information Systems San Francisco State University http://verma.sfsu.edu/ http://commons.sfsu.edu/ http://olpcsf.org/ http://olpcjamaica.org.jm/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody t...@timmoody.com wrote: does xovis supersede sugarstats? From what I understand, sugar-stats runs at the dbus level as opposed to the Journal metadata, which pulls data from the Journal backups. xovis relies on data that it can get as json, pushed into couchdb. So, while right now xovis uses json from Journal backups, it could be modified to do so from the RRD files that sugar-stats produces. That said, the data collected by sugar-stats is different, and needs to be filtered accordingly before it can be used sensibly by xovis. We have RRD files from Bhagmalpur, India, but we haven't done much with it as yet. I understand that AU may have a boatload of sugar-stats data as well, but not sure if they got around to doing anything with it. cheers, Sameer -- Sameer Verma, Ph.D. Professor, Information Systems San Francisco State University http://verma.sfsu.edu/ http://commons.sfsu.edu/ http://olpcsf.org/ http://olpcjamaica.org.jm/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [XSCE] sugarstats and xovis
On Sun, Jun 22, 2014 at 6:12 PM, Walter Bender walter.ben...@gmail.com wrote: On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma sve...@sfsu.edu wrote: On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody t...@timmoody.com wrote: does xovis supersede sugarstats? From what I understand, sugar-stats runs at the dbus level as opposed to the Journal metadata, which pulls data from the Journal backups. xovis relies on data that it can get as json, pushed into couchdb. So, while right now xovis uses json from Journal backups, it could be modified to do so from the RRD files that sugar-stats produces. That said, the data collected by sugar-stats is different, and needs to be filtered accordingly before it can be used sensibly by xovis. And there is the Harvest system, which for the most part relies on data in Journal. Yes, in fact we had conversations with Martin Abente about Harvest, and the possibility that the Harvest data could be serialized (json) and pushed into xovis. Are there any samples of Harvest data that we can look at some place? Sameer -walter We have RRD files from Bhagmalpur, India, but we haven't done much with it as yet. I understand that AU may have a boatload of sugar-stats data as well, but not sure if they got around to doing anything with it. cheers, Sameer -- Sameer Verma, Ph.D. Professor, Information Systems San Francisco State University http://verma.sfsu.edu/ http://commons.sfsu.edu/ http://olpcsf.org/ http://olpcjamaica.org.jm/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel -- Walter Bender Sugar Labs http://www.sugarlabs.org -- Sameer Verma, Ph.D. Professor, Information Systems San Francisco State University http://verma.sfsu.edu/ http://commons.sfsu.edu/ http://olpcsf.org/ http://olpcjamaica.org.jm/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
