Re: [Server-devel] [XSCE] sugarstats and xovis

2014-07-09 Thread Sameer Verma
On Thu, Jul 3, 2014 at 2:26 AM, Martin Dluhos  wrote:
> On 06/27/2014 02:10 PM, Martin Abente wrote:
>> Hello Martin!
>>
>> Do you have documentation on how XOvis pipeline works? I would like to see 
>> how
>> it works after the data is store and the visualization is generated.
>
>
> Here is a simple graphic that illustrates the workflow:
>
> http://i.imgur.com/Xhy1Vx3.jpg
>
> XOvis is a Couch application, which means that it's also stored in the db 
> along
> with the journal metadata. It queries the db using map-reduce queries, which 
> you
> can see here:
>
> https://github.com/martasd/xovis/blob/master/lib/views.js
>
>> Writing a tool to export the metadata from Harvest-server to a compatible 
>> subset
>> in the visualization-ready format of XOvis, would be cheaper in development
>> cost, but more expensive in running time (with large datasets). The other way
>> around, would be to allow XOvis read directly from Harvest-server MySQL 
>> format.
>

CouchDB has two distinct advantages for XOVis. 1) It allows for
"eventual consistency" which helps in syncing data from the school to
a central location using XS or XSCE. I believe this isn't relevant for
AU/Harvest because they don't use a school server, and the XOs send
metadata directly. 2) CouchDB does the aggregation through mapreduce.
This is powerful, and withour couchDB (or mongodb etc) you'll have to
write the aggregation yourself and/or get MySQL to do it.

The visualization may or may not be stored in the DB. It just happens
to be one more convenience of CouchDB.

cheers,
Sameer

> Since transferring metadata into the db needs to be done only once, running 
> time
> has never been a major concern. How large are the datasets you've collected on
> the Harvest-server?
>
> Cheers,
> Martin
>
>
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-07-03 Thread Martin Dluhos
On 06/27/2014 02:10 PM, Martin Abente wrote:
> Hello Martin!
> 
> Do you have documentation on how XOvis pipeline works? I would like to see how
> it works after the data is store and the visualization is generated.


Here is a simple graphic that illustrates the workflow:

http://i.imgur.com/Xhy1Vx3.jpg

XOvis is a Couch application, which means that it's also stored in the db along
with the journal metadata. It queries the db using map-reduce queries, which you
can see here:

https://github.com/martasd/xovis/blob/master/lib/views.js

> Writing a tool to export the metadata from Harvest-server to a compatible 
> subset
> in the visualization-ready format of XOvis, would be cheaper in development
> cost, but more expensive in running time (with large datasets). The other way
> around, would be to allow XOvis read directly from Harvest-server MySQL 
> format.

Since transferring metadata into the db needs to be done only once, running time
has never been a major concern. How large are the datasets you've collected on
the Harvest-server?

Cheers,
Martin

___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-27 Thread Martin Abente
Hello Martin!

Do you have documentation on how XOvis pipeline works? I would like to see
how it works after the data is store and the visualization is generated.

Writing a tool to export the metadata from Harvest-server to a compatible
subset in the visualization-ready format of XOvis, would be cheaper in
development cost, but more expensive in running time (with large datasets).
The other way around, would be to allow XOvis read directly from
Harvest-server MySQL format.

There are other options for integration too, but ultimately depends on how
XOvis pipeline works :)





On Fri, Jun 27, 2014 at 5:09 AM, Martin Dluhos  wrote:

> On 06/24/2014 07:20 PM, Sameer Verma wrote:
> > On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente
> >  wrote:
> >> Hello Sameer,
> >>
> >> The harvest-client serializes (using json) a subset of journal-objects
> >> metadata and then sends it to the harvest-server, which is basically a
> >> web-service plus a MySQL back-end.
> >>
> >
> > Great! Then we can perhaps add a method to process_journal_stats.py in
> > XOVis to support Harvest.
>
> XOvis visualization relies on journal metadata being stored in couchdb in
> json,
> so it should be relatively straightforward to add a method to
> process_journal_stats.py to read data generated by harvest-client and
> store it
> in the db.
>
> Martin
>
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-27 Thread Martin Dluhos
On 06/24/2014 07:20 PM, Sameer Verma wrote:
> On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente
>  wrote:
>> Hello Sameer,
>>
>> The harvest-client serializes (using json) a subset of journal-objects
>> metadata and then sends it to the harvest-server, which is basically a
>> web-service plus a MySQL back-end.
>>
> 
> Great! Then we can perhaps add a method to process_journal_stats.py in
> XOVis to support Harvest.

XOvis visualization relies on journal metadata being stored in couchdb in json,
so it should be relatively straightforward to add a method to
process_journal_stats.py to read data generated by harvest-client and store it
in the db.

Martin
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-27 Thread Martin Dluhos
On 06/22/2014 08:33 PM, Sameer Verma wrote:
> On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
>> does xovis supersede sugarstats?
> 
> 
>>From what I understand, sugar-stats runs at the dbus level as opposed
> to the Journal metadata, which pulls data from the Journal backups.
> xovis relies on data that it can get as json, pushed into couchdb. So,
> while right now xovis uses json from Journal backups, it could be
> modified to do so from the RRD files that sugar-stats produces. That
> said, the data collected by sugar-stats is different, and needs to be
> filtered accordingly before it can be used sensibly by xovis.

That's exactly right. Using sugar-stats data in XOvis was actually one of the
things I began exploring while in Nepal, but didn't implement prior to my
departure. I think adding that capability to XOvis would be very very useful.

Martin

___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-24 Thread Sameer Verma
On Mon, Jun 23, 2014 at 9:32 PM, Martin Abente
 wrote:
>
>
>
> On Sun, Jun 22, 2014 at 9:43 PM, Sameer Verma  wrote:
>>
>> On Sun, Jun 22, 2014 at 6:12 PM, Walter Bender 
>> wrote:
>> >
>> >
>> > On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma  wrote:
>> >>
>> >> On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
>> >> > does xovis supersede sugarstats?
>> >>
>> >>
>> >> From what I understand, sugar-stats runs at the dbus level as opposed
>> >> to the Journal metadata, which pulls data from the Journal backups.
>> >> xovis relies on data that it can get as json, pushed into couchdb. So,
>> >> while right now xovis uses json from Journal backups, it could be
>> >> modified to do so from the RRD files that sugar-stats produces. That
>> >> said, the data collected by sugar-stats is different, and needs to be
>> >> filtered accordingly before it can be used sensibly by xovis.
>> >
>> >
>> > And there is the Harvest system, which for the most part relies on data
>> > in
>> > Journal.
>> >
>>
>> Yes, in fact we had conversations with Martin Abente about Harvest,
>> and the possibility that the Harvest data could be serialized (json)
>> and pushed into xovis. Are there any samples of Harvest data that we
>> can look at some place?
>
>
>
> Hello Sameer,
>
> The harvest-client serializes (using json) a subset of journal-objects
> metadata and then sends it to the harvest-server, which is basically a
> web-service plus a MySQL back-end.
>

Great! Then we can perhaps add a method to process_journal_stats.py in
XOVis to support Harvest.

> I need to update this documentation [1] (will do this tomorrow early), but
> you can get an idea of what fields are used. If you need an example of the
> serialized format for the client-server communication you can look at [2].
> The bits that handle the serialization process can be found here [3], which
> is meant to be read.
>

OK. Will take a look.

Sameer

> Regards,
> Martin.
>
> Refs:
> 1. http://wiki.sugarlabs.org/go/Harvest
> 2. https://github.com/tchx84/harvest-server/blob/master/test/data/data.json
> 3.
> https://github.com/tchx84/harvest-client/blob/master/extensions/webservice/harvest/harvest/crop.py
>
>>
>>
>> Sameer
>>
>> > -walter
>> >>
>> >>
>> >> We have RRD files from Bhagmalpur, India, but we haven't done much
>> >> with it as yet. I understand that AU may have a boatload of
>> >> sugar-stats data as well, but not sure if they got around to doing
>> >> anything with it.
>> >>
>> >> cheers,
>> >> Sameer
>> >> --
>> >> Sameer Verma, Ph.D.
>> >> Professor, Information Systems
>> >> San Francisco State University
>> >> http://verma.sfsu.edu/
>> >> http://commons.sfsu.edu/
>> >> http://olpcsf.org/
>> >> http://olpcjamaica.org.jm/
>> >> ___
>> >> Server-devel mailing list
>> >> Server-devel@lists.laptop.org
>> >> http://lists.laptop.org/listinfo/server-devel
>> >
>> >
>> >
>> >
>> > --
>> > Walter Bender
>> > Sugar Labs
>> > http://www.sugarlabs.org
>>
>>
>>
>> --
>> Sameer Verma, Ph.D.
>> Professor, Information Systems
>> San Francisco State University
>> http://verma.sfsu.edu/
>> http://commons.sfsu.edu/
>> http://olpcsf.org/
>> http://olpcjamaica.org.jm/
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-23 Thread Martin Abente
On Sun, Jun 22, 2014 at 9:43 PM, Sameer Verma  wrote:

> On Sun, Jun 22, 2014 at 6:12 PM, Walter Bender 
> wrote:
> >
> >
> > On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma  wrote:
> >>
> >> On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
> >> > does xovis supersede sugarstats?
> >>
> >>
> >> From what I understand, sugar-stats runs at the dbus level as opposed
> >> to the Journal metadata, which pulls data from the Journal backups.
> >> xovis relies on data that it can get as json, pushed into couchdb. So,
> >> while right now xovis uses json from Journal backups, it could be
> >> modified to do so from the RRD files that sugar-stats produces. That
> >> said, the data collected by sugar-stats is different, and needs to be
> >> filtered accordingly before it can be used sensibly by xovis.
> >
> >
> > And there is the Harvest system, which for the most part relies on data
> in
> > Journal.
> >
>
> Yes, in fact we had conversations with Martin Abente about Harvest,
> and the possibility that the Harvest data could be serialized (json)
> and pushed into xovis. Are there any samples of Harvest data that we
> can look at some place?
>


Hello Sameer,

The harvest-client serializes (using json) a subset of journal-objects
metadata and then sends it to the harvest-server, which is basically a
web-service plus a MySQL back-end.

I need to update this documentation [1] (will do this tomorrow early), but
you can get an idea of what fields are used. If you need an example of the
serialized format for the client-server communication you can look at [2].
The bits that handle the serialization process can be found here [3], which
is meant to be read.

Regards,
Martin.

Refs:
1. http://wiki.sugarlabs.org/go/Harvest
2. https://github.com/tchx84/harvest-server/blob/master/test/data/data.json
3.
https://github.com/tchx84/harvest-client/blob/master/extensions/webservice/harvest/harvest/crop.py


>
> Sameer
>
> > -walter
> >>
> >>
> >> We have RRD files from Bhagmalpur, India, but we haven't done much
> >> with it as yet. I understand that AU may have a boatload of
> >> sugar-stats data as well, but not sure if they got around to doing
> >> anything with it.
> >>
> >> cheers,
> >> Sameer
> >> --
> >> Sameer Verma, Ph.D.
> >> Professor, Information Systems
> >> San Francisco State University
> >> http://verma.sfsu.edu/
> >> http://commons.sfsu.edu/
> >> http://olpcsf.org/
> >> http://olpcjamaica.org.jm/
> >> ___
> >> Server-devel mailing list
> >> Server-devel@lists.laptop.org
> >> http://lists.laptop.org/listinfo/server-devel
> >
> >
> >
> >
> > --
> > Walter Bender
> > Sugar Labs
> > http://www.sugarlabs.org
>
>
>
> --
> Sameer Verma, Ph.D.
> Professor, Information Systems
> San Francisco State University
> http://verma.sfsu.edu/
> http://commons.sfsu.edu/
> http://olpcsf.org/
> http://olpcjamaica.org.jm/
>
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-22 Thread Sameer Verma
On Sun, Jun 22, 2014 at 6:12 PM, Walter Bender  wrote:
>
>
> On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma  wrote:
>>
>> On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
>> > does xovis supersede sugarstats?
>>
>>
>> From what I understand, sugar-stats runs at the dbus level as opposed
>> to the Journal metadata, which pulls data from the Journal backups.
>> xovis relies on data that it can get as json, pushed into couchdb. So,
>> while right now xovis uses json from Journal backups, it could be
>> modified to do so from the RRD files that sugar-stats produces. That
>> said, the data collected by sugar-stats is different, and needs to be
>> filtered accordingly before it can be used sensibly by xovis.
>
>
> And there is the Harvest system, which for the most part relies on data in
> Journal.
>

Yes, in fact we had conversations with Martin Abente about Harvest,
and the possibility that the Harvest data could be serialized (json)
and pushed into xovis. Are there any samples of Harvest data that we
can look at some place?

Sameer

> -walter
>>
>>
>> We have RRD files from Bhagmalpur, India, but we haven't done much
>> with it as yet. I understand that AU may have a boatload of
>> sugar-stats data as well, but not sure if they got around to doing
>> anything with it.
>>
>> cheers,
>> Sameer
>> --
>> Sameer Verma, Ph.D.
>> Professor, Information Systems
>> San Francisco State University
>> http://verma.sfsu.edu/
>> http://commons.sfsu.edu/
>> http://olpcsf.org/
>> http://olpcjamaica.org.jm/
>> ___
>> Server-devel mailing list
>> Server-devel@lists.laptop.org
>> http://lists.laptop.org/listinfo/server-devel
>
>
>
>
> --
> Walter Bender
> Sugar Labs
> http://www.sugarlabs.org



-- 
Sameer Verma, Ph.D.
Professor, Information Systems
San Francisco State University
http://verma.sfsu.edu/
http://commons.sfsu.edu/
http://olpcsf.org/
http://olpcjamaica.org.jm/
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-22 Thread Walter Bender
On Sun, Jun 22, 2014 at 6:33 PM, Sameer Verma  wrote:

> On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
> > does xovis supersede sugarstats?
>
>
> From what I understand, sugar-stats runs at the dbus level as opposed
> to the Journal metadata, which pulls data from the Journal backups.
> xovis relies on data that it can get as json, pushed into couchdb. So,
> while right now xovis uses json from Journal backups, it could be
> modified to do so from the RRD files that sugar-stats produces. That
> said, the data collected by sugar-stats is different, and needs to be
> filtered accordingly before it can be used sensibly by xovis.
>

And there is the Harvest system, which for the most part relies on data in
Journal.

-walter

>
> We have RRD files from Bhagmalpur, India, but we haven't done much
> with it as yet. I understand that AU may have a boatload of
> sugar-stats data as well, but not sure if they got around to doing
> anything with it.
>
> cheers,
> Sameer
> --
> Sameer Verma, Ph.D.
> Professor, Information Systems
> San Francisco State University
> http://verma.sfsu.edu/
> http://commons.sfsu.edu/
> http://olpcsf.org/
> http://olpcjamaica.org.jm/
> ___
> Server-devel mailing list
> Server-devel@lists.laptop.org
> http://lists.laptop.org/listinfo/server-devel
>



-- 
Walter Bender
Sugar Labs
http://www.sugarlabs.org
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel


Re: [Server-devel] [XSCE] sugarstats and xovis

2014-06-22 Thread Sameer Verma
On Sun, Jun 22, 2014 at 7:41 AM, Tim Moody  wrote:
> does xovis supersede sugarstats?


>From what I understand, sugar-stats runs at the dbus level as opposed
to the Journal metadata, which pulls data from the Journal backups.
xovis relies on data that it can get as json, pushed into couchdb. So,
while right now xovis uses json from Journal backups, it could be
modified to do so from the RRD files that sugar-stats produces. That
said, the data collected by sugar-stats is different, and needs to be
filtered accordingly before it can be used sensibly by xovis.

We have RRD files from Bhagmalpur, India, but we haven't done much
with it as yet. I understand that AU may have a boatload of
sugar-stats data as well, but not sure if they got around to doing
anything with it.

cheers,
Sameer
-- 
Sameer Verma, Ph.D.
Professor, Information Systems
San Francisco State University
http://verma.sfsu.edu/
http://commons.sfsu.edu/
http://olpcsf.org/
http://olpcjamaica.org.jm/
___
Server-devel mailing list
Server-devel@lists.laptop.org
http://lists.laptop.org/listinfo/server-devel