Re: [Xen-devel] Updates on the project

[Jesus M. Gonzalez-Barahona]

On Mon, 2017-04-17 at 18:54 +0530, Gayathri Menakath wrote:
> Hello Jesus,
> 
> I would like to thank you for the comments. I will look into the part
> where it uploads the data to the Elasticsearch index and the
> jwzthreading.py. I believe that I had mentioned in one of the IRC
> chats that I would be reusing the jwzthreading.py. I am sorry if I
> hadn't mentioned it. However, should I be making any changes to it?

No, that's fine. Maybe I just don't remember. Thanks in any case.

Jesus

> On Mon, Apr 17, 2017 at 4:48 AM, Jesus M. Gonzalez-Barahona
> <j...@bitergia.com> wrote:
> > On Sat, 2017-04-15 at 20:08 +0530, Gayathri Menakath wrote:
> > > Hello Jesus,
> > >
> > > As my periodical exams were going on I could not spend much time
> > on
> > > writing the tests (2nd microtask). I will resume the work soon
> > and
> > > will send the updates. Along with my proposal, I have uploaded an
> > > official letter from my university which states that I would not
> > be
> > > having any academic commitments for at least 8 weeks during the
> > > coding period. I hope with that I would be able to meet the
> > > eligibility criteria for Outreachy. I had sent a copy of the
> > letter
> > > to the Outreachy coordinators and Lars too.
> > 
> > Thanks a lot for the update.
> > 
> > > Meanwhile, may I know if you had reviewed the first microtask's
> > code?
> > 
> > Yes. I did. Some of comments:
> > 
> > * I've tested it with some mboxes, and seem to work pretty well. A
> > bit
> > weird that you have to produce a JSON file, and then upload it to
> > ES,
> > instead of just uploading it to ES directly. But otherwise, it
> > seems to
> > work with the tests I did.
> > 
> > * However, you had hardwired a path in jwzthreading.py, with (I
> > presume) the directory where you store the mboxes. After changing
> > it to
> > mind, worked like a charm.
> > 
> > * BTW, I don't remember that you commented that you were using
> > jwzthreading.py. That's not bad (reusing code which works is always
> > a
> > good option to consider), but makes the exercise different, since
> > the
> > implementation of the threading algorithm is in it.
> > 
> > * The readme.md explains well how to run the scripts.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > -- 
> > > Yours Sincerely,
> > > Gayathri.P.Menakath
> > > B-Tech 3rd year,
> > > Amrita University
> > > blog | Github 
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
> 
> 
> 
> -- 
> Yours Sincerely,
> Gayathri.P.Menakath
> B-Tech 3rd year,
> Amrita University
> blog | Github 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-16 at 21:26 -0700, Heather Booker wrote:
> Hi Jesus!
> 
> I appreciate the info on the unicode error. I might have missed it,
> but I also asked about the general microtask specifications. Here
> was my original inquiry:
> > And to clarify, my understanding is that the final result of
> this task
> > is an index of Xen data, with two types: commits and messages.
> > Each commit document should contain its original information
> > from git, plus the name of the branch it was developed in. And
> > should only the mbox messages which appear to be associated
> > with a specific commit exist in the final index? Is there some
> > key information in messages that is supposed to indicate the
> > association of a given commit with a git branch? I would be
> > grateful if you could specify the end goal a little more. :D
> 
> Yeah, so overall I'm not sure I understand the relationship of
> branches to the mailing list messages. Is this to be a simple
> string parsing task wherein I should scan the message body
> for the word "branch"? (I am guessing not ;P)

I'm sorry, I understood that text was about the project, not about the
microtask. The microtask is about either:

* Producing an ES index with messages labeled by thread (by applying a
threading algorithm to messages retrieved from archives), or

* Producing an ES index with commits labeled by branch (by following
refes, and parents information in the output produced by Perceval).

In the complete project, both will be used to produce the final indexes
that power the code review dashboard.

> I will be happy to get back on developing once I better grasp
> the goal! :)

More clear now?

If you want, let's schedule some IRC slot for clarifying whatever is
not clear.

Jesus.

> Thanks!
> 
> Heather
> 
> On Sun, Apr 16, 2017 at 4:23 PM, Jesus M. Gonzalez-Barahona <jgb@bite
> rgia.com> wrote:
> > On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> > > Hi,
> > >
> > > I submitted an application for this code review dashboard and
> > > would love to keep working on the microtask once I get some
> > > more info. :)
> > 
> > Great! I answered your message, could you progress with the task?
> > 
> > > I also came up with a general idea of how the project might be
> > > split up - any feedback on this would be welcome! I wrote:
> > >
> > > "As said by Jesus, the big picture of this project will be
> > porting
> > > everything behind the current code review dashboard to use
> > > Grimoire Lab tools, from the current state of using
> > > MetricsGrimoire and custom scripts. I expect this would involve
> > > Perceval for analyzing data, and Grimoire Elk may be useful in
> > > further stages, or may be too general - this is something I would
> > > wish to explore.
> > > This project will also involve a migration from SQL to
> > Elasticsearch
> > > - because I believe the relevant data is mostly / all available
> > in
> > > places online, I am unsure whether this would need to be a direct
> > > migration. However, looking at the current SQL setup would be
> > > beneficial to understanding the desired format of the
> > Elasticsearch
> > > indexes.
> > > I would love to dive into this project and have 3 main parts -
> > > getting
> > > data into ES, turning it into dashboard displays, and then fine
> > > tuning
> > > and perhaps augmenting the dashboard to improve its usefulness.
> > > Getting data into ES may seem simple but I believe that once it
> > > needs to be used for the dashboard, many realizations will pop up
> > > - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> > > weeks
> > > for the visualizations (which will include querying the data),
> > and
> > > the
> > > final 3 weeks for touch ups and improvements."
> > 
> > The plan could be sound, but would need some tweaks, once your
> > skills
> > in Python are clear, which could be the main blocker for the first
> > stages.
> > 
> > > Does this sound like an accurate summary and reasonable
> > timeline? 
> > > And I am guessing that from Jesus's involvement with the threads
> > > that Jesus would be the mentor, is that correct? :)
> > 
> > Yes, I would be ;-)
> > 
> >         Jesus.
> > 
> > > Thanks!
> > >
> > > Heather
> > >
> > >
> > > On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker <heather.j.booker@
> > gmai
> > > l.com&

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> Hi,
> 
> I submitted an application for this code review dashboard and
> would love to keep working on the microtask once I get some
> more info. :)

Great! I answered your message, could you progress with the task?

> I also came up with a general idea of how the project might be
> split up - any feedback on this would be welcome! I wrote:
> 
> "As said by Jesus, the big picture of this project will be porting
> everything behind the current code review dashboard to use
> Grimoire Lab tools, from the current state of using
> MetricsGrimoire and custom scripts. I expect this would involve
> Perceval for analyzing data, and Grimoire Elk may be useful in
> further stages, or may be too general - this is something I would
> wish to explore.
> This project will also involve a migration from SQL to Elasticsearch
> - because I believe the relevant data is mostly / all available in
> places online, I am unsure whether this would need to be a direct
> migration. However, looking at the current SQL setup would be
> beneficial to understanding the desired format of the Elasticsearch
> indexes.
> I would love to dive into this project and have 3 main parts -
> getting
> data into ES, turning it into dashboard displays, and then fine
> tuning
> and perhaps augmenting the dashboard to improve its usefulness.
> Getting data into ES may seem simple but I believe that once it
> needs to be used for the dashboard, many realizations will pop up
> - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> weeks
> for the visualizations (which will include querying the data), and
> the
> final 3 weeks for touch ups and improvements."

The plan could be sound, but would need some tweaks, once your skills
in Python are clear, which could be the main blocker for the first
stages.

> Does this sound like an accurate summary and reasonable timeline? 
> And I am guessing that from Jesus's involvement with the threads
> that Jesus would be the mentor, is that correct? :)

Yes, I would be ;-)

Jesus.

> Thanks!
> 
> Heather
> 
> 
> On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker <heather.j.booker@gmai
> l.com> wrote:
> > Hi Jesus,
> > 
> > While using the Elasticsearch python library
> > (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> > messages to an index, I would get a UnicodeEncodeError:
> > "'utf-8' codec can't encode character '\udca0' in position 767:
> > surrogates not allowed".
> > 
> > Investigating in Grimoire elk https://github.com/grim
> > oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> > 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that 
> > perhaps that tool instead uses Latin-1 encoding, but I found that
> > to then produce a serialization error (their custom error message:
> > "Unable to serialize %r (type: %s)"). I suppose this is because
> > now it's bytes; of course, converting back to string after encoding
> > just cycles back to the first error.
> > 
> > As somewhat of a Python newbie I don't really know how to tackle
> > this! My thought atm is to splice the offending character out
> > of the message. 
> > 
> > And to clarify, my understanding is that the final result of this
> > task
> > is an index of Xen data, with two types: commits and messages.
> > Each commit document should contain its original information
> > from git, plus the name of the branch it was developed in. And
> > should only the mbox messages which appear to be associated
> > with a specific commit exist in the final index? Is there some
> > key information in messages that is supposed to indicate the
> > association of a given commit with a git branch? I would be
> > grateful if you could specify the end goal a little more. :D
> > 
> > Thanks so much!
> > 
> > Heather
> > 
> > 
> > 
> > On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona <jgb@bi
> > tergia.com> wrote:
> > > On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> > > > Hi Jesus, 
> > > >
> > > > Thanks for your reply!
> > > >
> > > > So about the task, instructions say after analyzing mboxes with
> > > > Perceval to
> > > > "store the resulting raw index in ElasticSearch" - what does
> > > raw
> > > > index mean?
> > > 
> > > In this context, I mean "storing the JSON documents produced by
> > > Perceval in an ElasticSearch index, as such". ElasticSearch
> > > stores JSON
> >

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-09 at 21:50 -0700, Heather Booker wrote:
> Hi Jesus,
> 
> While using the Elasticsearch python library
> (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> messages to an index, I would get a UnicodeEncodeError:
> "'utf-8' codec can't encode character '\udca0' in position 767:
> surrogates not allowed".
> 

What happens here is that Perceval has some assumptions about character
encoding, when reading messages (to convert them to Unicode strings).
If they are not fulfilled, it converts the character as "surrogate".
When trying to produce utf8 from those, that cannnot be done, since the
space for "surrogate" Unicode is thought to convert back to the
original encoding. But JSON expects the encoding to be utf8, so no luck
here.

The trick is to provide a serializer which either skips those messages,
or produces a "escaped" encoding for them.

See http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/ for
a detailed explanation.

Please, let me know if you can work from here...

Jesus.

> Investigating in Grimoire elk https://github.com/grim
> oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that 
> perhaps that tool instead uses Latin-1 encoding, but I found that
> to then produce a serialization error (their custom error message:
> "Unable to serialize %r (type: %s)"). I suppose this is because
> now it's bytes; of course, converting back to string after encoding
> just cycles back to the first error.
> 
> As somewhat of a Python newbie I don't really know how to tackle
> this! My thought atm is to splice the offending character out
> of the message. 
> 
> And to clarify, my understanding is that the final result of this
> task
> is an index of Xen data, with two types: commits and messages.
> Each commit document should contain its original information
> from git, plus the name of the branch it was developed in. And
> should only the mbox messages which appear to be associated
> with a specific commit exist in the final index? Is there some
> key information in messages that is supposed to indicate the
> association of a given commit with a git branch? I would be
> grateful if you could specify the end goal a little more. :D
> 
> Thanks so much!
> 
> Heather
> 
> 
> 
> On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona <jgb@bite
> rgia.com> wrote:
> > On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> > > Hi Jesus, 
> > >
> > > Thanks for your reply!
> > >
> > > So about the task, instructions say after analyzing mboxes with
> > > Perceval to
> > > "store the resulting raw index in ElasticSearch" - what does raw
> > > index mean?
> > 
> > In this context, I mean "storing the JSON documents produced by
> > Perceval in an ElasticSearch index, as such". ElasticSearch stores
> > JSON
> > documents, so it is just uploading the output of Perceval to it.
> > 
> > > In terms of figuring out the elasticsearch structure, do I want
> > an
> > > index
> > > (xen-devel mbox) with a type (message) and each object from the
> > > perceval
> > > output to be one document? Or should it be more fine-grained?
> > 
> > Exactly.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Cheers,
> > >
> > > Heather
> > >
> > > On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona <jgb@b
> > iter
> > > gia.com> wrote:
> > > > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > > > Hi!
> > > > >
> > > > > I'd love to work on the Code Review Dashboard project for
> > this
> > > > round
> > > > > of Outreachy.
> > > >
> > > > Great!!
> > > >
> > > > > Are the steps outlined
> > > > > here http://markmail.org/message/7adkmords3imkswd still the
> > first
> > > > > contribution you'd like to see?
> > > >
> > > > Yes.
> > > >
> > > > > So is this a project that has been worked on in previous
> > rounds
> > > > of
> > > > > GSOC/Outreachy also?
> > > > > If so is there a place to find links to the previous
> > participants
> > > > > blogs? :)
> > > >
> > > > No. We had one participation at some point, but couldn't even
> > start
> > > > for
> > > > personal reasons. There are some people consid

Re: [Xen-devel] [Outreachy] Interested in Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Mon, 2017-04-10 at 14:50 -0700, Candida Haynes wrote:
> Hi,
> 
> I apologize to anyone who receives this twice - I received an
> error/bounce message. I am writing because I am interested in
> applying to Outreachy and contributing to the Xen Code Review
> Dashboard. My most formal experience with open source was in 2014
> when I participated in the Ascend Project. 
> 
> I know I need to make a small code contribution so I am here to find
> out what to do. I have used Mercurial and Git, and have been studying
> JavaScript and Python so I am comfortable switching contexts and
> learning how different software works. I've also been exposed to Go,
> Ruby, Elm, and Swift through tutorials and immersive weekend
> workshops. My JavaScript stack is Node, Express, and PostgreSQL. I've
> interacted with Bugzilla and (briefly) with Try-Server. I enjoy
> working with data and have participated in PyData, but I want to
> learn more. I think this project lends itself to that. Can anyone
> advise on how to get started on the "small contribution" or if there
> is anyone else I need to add to this e-mail? Thanks a lot!

Hi, Candida,

The stack for this project is more like Python and ElasticSearch (I see
you are already familiar with Python, which is great).

A microtask is defined in

http://markmail.org/message/7adkmords3imkswd

And you have some other messages recently posted to this mailing list
with some more info about the project, since some other people
expressed their interest on it. Let me know if you need something else,
please.

Jesus.

> Candida Haynes / Didit
> 
> 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

Oooops. I think I missed your message. Sorry about that. Please, let me
know about your ideas. In any case, in other messages about the matter,
to this mailing list, you can also find some more details.

Important notice: please have in mind that the project is more about
Python, ElasticSearch and Kibana than anything else.

Jesus.

On Mon, 2017-04-03 at 20:38 -0500, Todd Hendricks wrote:
> Team,
> 
> I am interested in the project and would like to begin contributing.
> It would be good to speak with a team lead about the project at a
> high level, and to identify areas where I can provide quick wins. In
> addition to CSS/HTML, my languages are JS, PHP, and SQL. 
> 
> Thank you for your consideration.
> 
> Regards,
> 
> Todd Hendricks
> 281-702-1156
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> Hi Jesus, 
> 
> Thanks for your reply!
> 
> So about the task, instructions say after analyzing mboxes with
> Perceval to
> "store the resulting raw index in ElasticSearch" - what does raw
> index mean?

In this context, I mean "storing the JSON documents produced by
Perceval in an ElasticSearch index, as such". ElasticSearch stores JSON
documents, so it is just uploading the output of Perceval to it.

> In terms of figuring out the elasticsearch structure, do I want an
> index
> (xen-devel mbox) with a type (message) and each object from the
> perceval
> output to be one document? Or should it be more fine-grained?

Exactly.

Saludos,

Jesus.

> Cheers,
> 
> Heather
> 
> On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona <jgb@biter
> gia.com> wrote:
> > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > Hi!
> > >
> > > I'd love to work on the Code Review Dashboard project for this
> > round
> > > of Outreachy.
> > 
> > Great!!
> > 
> > > Are the steps outlined
> > > here http://markmail.org/message/7adkmords3imkswd still the first
> > > contribution you'd like to see?
> > 
> > Yes.
> > 
> > > So is this a project that has been worked on in previous rounds
> > of
> > > GSOC/Outreachy also?
> > > If so is there a place to find links to the previous participants
> > > blogs? :)
> > 
> > No. We had one participation at some point, but couldn't even start
> > for
> > personal reasons. There are some people considering working on this
> > for
> > this next round of Outreachy, however. You'll see their messages in
> > this mailing list.
> > 
> > > Should questions about how the specifications/completion of the
> > > microtask be addressed to
> > > IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> > > grimoire? On that note, I'm 
> > > curious why #metrics-grimoire is the listed channel on the
> > project
> > > page - are main contributors
> > > involved in both projects? Or is it just because the Xen
> > dashboard
> > > doesn't have a channel?
> > 
> > The code review is for the Xen project, but it is done with (I
> > mean,
> > the ssoftware used for it is) GrimoireLab, which for historical
> > reasons
> > uses the #metrics-grimoire channel. That's why it is likely that
> > you
> > find somebody from the project there.
> > 
> > If you have questions, and find me around in IRC, please ping me.
> > If
> > I'm not available, please send an email message.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Thanks!
> > >
> > > Heather
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > https://lists.xen.org/xen-devel
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> Hi!
> 
> I'd love to work on the Code Review Dashboard project for this round
> of Outreachy.

Great!!

> Are the steps outlined
> here http://markmail.org/message/7adkmords3imkswd still the first
> contribution you'd like to see?

Yes.

> So is this a project that has been worked on in previous rounds of
> GSOC/Outreachy also?
> If so is there a place to find links to the previous participants
> blogs? :)

No. We had one participation at some point, but couldn't even start for
personal reasons. There are some people considering working on this for
this next round of Outreachy, however. You'll see their messages in
this mailing list.

> Should questions about how the specifications/completion of the
> microtask be addressed to
> IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> grimoire? On that note, I'm 
> curious why #metrics-grimoire is the listed channel on the project
> page - are main contributors
> involved in both projects? Or is it just because the Xen dashboard
> doesn't have a channel?

The code review is for the Xen project, but it is done with (I mean,
the ssoftware used for it is) GrimoireLab, which for historical reasons
uses the #metrics-grimoire channel. That's why it is likely that you
find somebody from the project there.

If you have questions, and find me around in IRC, please ping me. If
I'm not available, please send an email message.

Saludos,

Jesus.

> Thanks!
> 
> Heather
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Outreachy] Interested to Work on Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Tue, 2017-04-04 at 05:07 +0530, Pooja wrote:
> Hello mentors,
> 
> I'd like to work on the new Xen Code Dashboard extensions project as
> part of applying to Outreachy Rd 14, 2017.
> 
> I've been working in front end development since 2013, and am quite
> comfortable in JavaScript, jQuery, JSON, CSS frameworks, SQL besides
> HTML5/CSS3.
> 
> I'd really appreciate if you could guide me about beginning the
> project with bite-sized tasks.

Hi, Pooja,

First of all, thanks for your interest.

And now, a warning notice: this project will require mainly Python and
noSQL (ElasticSearch, in particular) knowledge. If you're not familiar
with them, that could be a big problem.

[Lars, as I just commented in another message, I now notice this is
wrong in the project description at
https://wiki.xenproject.org/wiki/Outreach_Program_Projects
sorry about that. Could we change it?]

If you're still interested, I guess it would be good to have a quick
IRC chat, and discuss about next steps.

Meanwhile, you can start having a look at:

http://markmail.org/message/7adkmords3imkswd

Saludos,

Jesus.

> Thanks and Regards,
> Pooja G
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-02 at 13:09 -0700, vaishnavi.ur...@gmail.com wrote:
> Hi Jesus,
> I understand that the Perceval scripts are written in Python. I am
> familiar with Python and comfortable working in it. So does the micro
> task remain the same? To run Perceval on xen-devel list and save
> output to elastic search ?

Yes, you can start that way, a the link suggested by Lars suggests.

> Also, when are you generally online on IRC? 

At different times, feel free to ping me (jgbarah at #metrics-grimoire
on Freenode) if you see me. But maybe it is better to set some slot.
I'm in CEST timezone. What timezone are you?

Saludos,

Jesus.

> Thanks a lot!
> 
> Vaishnavi
> 
> > On Apr 2, 2017, at 11:20 AM, Jesus M. Gonzalez-Barahona <jgb@biterg
> > ia.com> wrote:
> > 
> > > On Thu, 2017-03-30 at 18:48 -0700, Vaishnavi Ramesh Jayaraman
> > > wrote:
> > > Hi,
> > > I am Vaishnavi, interested in contributing to the Xen Project as
> > > part
> > > of the Outreachy Program. I am particularly interested in working
> > > on
> > > the Xen Code Review Dashboard.
> > > 
> > > I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
> > > previously and am comfortable with Javascript.
> > > 
> > > It would be great if you could give me pointers on how to get
> > > started!
> > > 
> > > Also, I am unable to join the mailing list for this project -
> > > metrics
> > > -grimo...@lists.libresoft.es 
> > 
> > Hi, Vaishnavi,
> > 
> > First of all, thanks for your interest.
> > 
> > And now, a warning notice: this project will require mainly Python
> > and
> > noSQL (ElasticSearch, in particular) knowledge. I see you're
> > familiar
> > with ELK, what about Python?
> > 
> > [Lars, as I just commented in another message, I now notice this is
> > wrong in the project description at
> > https://wiki.xenproject.org/wiki/Outreach_Program_Projects
> > sorry about that. Could we change it?]
> > 
> > If you're still interested, I guess it would be good to have a
> > quick
> > IRC chat, and discuss about next steps.
> > 
> > Saludos,
> > 
> >    Jesus.
> > 
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > https://lists.xen.org/xen-devel
> > 
> > -- 
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2017-03-30 at 18:48 -0700, Vaishnavi Ramesh Jayaraman wrote:
> Hi,
> I am Vaishnavi, interested in contributing to the Xen Project as part
> of the Outreachy Program. I am particularly interested in working on
> the Xen Code Review Dashboard.
> 
> I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
> previously and am comfortable with Javascript.
> 
> It would be great if you could give me pointers on how to get
> started!
> 
> Also, I am unable to join the mailing list for this project - metrics
> -grimo...@lists.libresoft.es 

Hi, Vaishnavi,

First of all, thanks for your interest.

And now, a warning notice: this project will require mainly Python and
noSQL (ElasticSearch, in particular) knowledge. I see you're familiar
with ELK, what about Python?

[Lars, as I just commented in another message, I now notice this is
wrong in the project description at
https://wiki.xenproject.org/wiki/Outreach_Program_Projects
sorry about that. Could we change it?]

If you're still interested, I guess it would be good to have a quick
IRC chat, and discuss about next steps.

Saludos,

Jesus.

> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Outreachy] - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-02 at 15:43 +0530, Iresha Perera wrote:
> Hi all,
>  
> I'm a 4th year undergraduate at University of Moratuwa, following a
> Computer Science and Engineering degree. I'm interested in Xen Code
> Review Dashboard project and I would love to participate in this
> project for Outreachy program, 2017. I have a good knowledge on  SQL,
> Java, Javascript, HTML5 and XML and I have done several projects with
> these technologies. 
>  
> To start with Outreachy program, I need to make a contribution to the
> Xen Code Review Dashboard project.I read about the Contribution
> Guidelines. It would be very grateful if you could provide me with
> the details on how I can make an initial contribution to this
> project, in order to participate for the Outreahcy program.

Hi, Iresha,

First of all, thanks for your interest.

And now, a warning notice: this project will require mainly Python and
noSQL (ElasticSearch, in particular) knowledge. If you're not familiar
with them, that could be a big problem.

[Lars, I now notice this is wrong in the project description at
https://wiki.xenproject.org/wiki/Outreach_Program_Projects
sorry about that. Could we change it?]

If you're still interested, I guess it would be good to have a quick
IRC chat, and discuss about the initial contribution.

Saludos,

Jesus.

> Thank you very much.
>  
> Sincerely,
> -- 
> Anne Iresha Perera
> Department of Computer Science Engineering
> University of Moratuwa
> Sri Lanka
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [GSoC] Progress in the microtask

[Jesus M. Gonzalez-Barahona]

Thanks a lot, Gayathri. I will be looking at that asap.

Saludos,

Jesus.

On Sun, 2017-03-26 at 23:18 +0530, Gayathri Menakath wrote:
> Hello Jesus and Lars,
> 
> I have completed the micro task and now I am working on writing tests
> fro my code. The micro task was to write a Python script to use the
> Perceval email backend to retrieve data from the xen-devel mailing
> list, identify the same threads and feed the result to an
> ElasticSearch database. In order to identify the same threads, jwz
> threading algorithm was implemented on the data retrieved from Xen-
> devel mailing list’s archive.
> 
> The result of the script is an ElasticSearch search index, with one
> JSON document per message, with a value which is same for all the
> messages in the same thread. The value could be the message-id of the
> first message in the thread. Each month’s archive will have a
> separate index.
> 
> A script to perform a search has also been written. The data is
> retrieved from the Elasticsearch index and a search will be performed
> on it according to the specified field of the message. The task has
> been completed and has helped me to gain knowledge about Perceval and
> Elasticsearch. The code has been uploaded to my Github account [1].
> 
> 
> On Sun, Mar 26, 2017 at 12:07 PM, Gayathri Menakath  @gmail.com> wrote:
> > Hello Jesus and Lars,
> > 
> > I have uploaded the data to the Elasticsearch index. As a part of
> > testing it, I have uploaded only some of the fields in the JSON
> > file (Sender, From and Message ID). I have also written a python
> > script to perform a search on the data in Elasticsearch index. The
> > code has been uploaded to my Github account [1]. Please review it.
> > 
> > If you are available this week, could we have an IRC meeting so
> > that I will get to know whether there are any changes to be made to
> > the current code and what next is to be done? Looking forward to
> > your response.
> > 
> > [1]: https://github.com/Gayathrimenakath/Xen-project
> > 
> > 
> > On Thu, Mar 23, 2017 at 4:04 AM, Jesus M. Gonzalez-Barahona <jgb@bi
> > tergia.com> wrote:
> > > On Wed, 2017-03-22 at 23:24 +0530, Gayathri Menakath wrote:
> > > > Hello Jesus and Lars,
> > > >
> > > > As I had mentioned earlier, I was facing and error in the
> > > threading
> > > > algorithm. I have fixed it and now I am able to get the
> > > threaded
> > > > output. I have uploaded the code to my github account[1]. Soon
> > > I will
> > > > upload the data to the Elasticsearch index.
> > > 
> > > Thanks a lot, I will review it asap.
> > > 
> > > > Due to Outreachy's strict time constraints, I came to know that
> > > I
> > > > will not be able to apply for this round. However I am eligible
> > > to
> > > > apply for GSoC and will be applying for the same.
> > > >
> > > > [1]: https://github.com/Gayathrimenakath/Xen-project
> > > 
> > > OK. Thanks for reporting.
> > > 
> > >         Jesus.
> > > 
> > > --
> > > Bitergia: http://bitergia.com
> > > /me at Twitter: https://twitter.com/jgbarah
> > > 
> > > 
> > 
> > 
> > 
> > -- 
> > Yours Sincerely,
> > Gayathri.P.Menakath
> > B-Tech 3rd year,
> > Amrita University
> > blog | Github 
> > 
> 
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Debugging environment Update

[Jesus M. Gonzalez-Barahona]

Great! Please, let me know if I can help you.

BTW, I hope you formally applied for Outreachy.

Saludos,

Jesus.

On Mon, 2016-10-17 at 14:54 -0400, Tevin Mallory wrote:
Hi Jesus,
    How are you today? I hope you are doing well. Since running perceval on 
windows was giving me to much trouble, I decided to change my OS to Ubuntu and 
haven't had any troubles with it. I am currently working on getting the data 
from the Xen git repo and should be finishing that up soon. Honestly looking 
back I should have switched to Ubuntu sooner and saved us both some time. If I 
run into any other issues I'll keep you posted, so far I am doing pretty well. 

Have a great day!
Tevin K. Mallory 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Debugging your environment...

[Jesus M. Gonzalez-Barahona]

C:\Users\Tevin\AppData\Local\Programs\Python\Pytho
> n35-
> 32\Scripts\;C:\Users\Tevin\AppData\Local\Programs\Python\Python35-
> 32\;C:\Users\Tevin\AppData\Local\Microsoft\WindowsApps;C:\Windows\Sys
> tem32;C:\Users\Tevin\Documents\project\mtask\Lib\site-packages
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>echo %PYTHONPATH
> %PYTHONPATH
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>path
> PATH=C:\Users\Tevin\Desktop\project\perceval\Scripts;C:\ProgramData\O
> racle\Java\javapath;C:\Program Files (x86)\Intel\TXE
> Components\TCS\;C:\Program Files\Intel\TXE
> Components\TCS\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wb
> em;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files
> (x86)\Common Files\Adobe\AGL;C:\Program Files
> (x86)\DevDesktop\drush;C:\Program Files\Java\jdk\bin;C:\Program
> Files\Java\jre\bin;C:\Users\Tevin\AppData\Local\Programs\Python\Pytho
> n35-
> 32\Scripts\;C:\Users\Tevin\AppData\Local\Programs\Python\Python35-
> 32\;C:\Users\Tevin\AppData\Local\Microsoft\WindowsApps;C:\Windows\Sys
> tem32;C:\Users\Tevin\Documents\project\mtask\Lib\site-packages
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>perceval
> 'perceval' is not recognized as an internal or external command,
> operable program or batch file.
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>perceval --help
> 'perceval' is not recognized as an internal or external command,
> operable program or batch file.
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>python
> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900
> 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more
> information.
> >>> exit()
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>Scripts\perceval
> 'Scripts\perceval' is not recognized as an internal or external
> command,
> operable program or batch file.
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>scripts\perceval
> 'scripts\perceval' is not recognized as an internal or external
> command,
> operable program or batch file.
>  
> (perceval) C:\Users\Tevin\Desktop\project\perceval>
>  
> Sent from Mail for Windows 10
>  
> From: Jesus M. Gonzalez-Barahona
> Sent: Friday, October 14, 2016 11:39 AM
> To: tevin.k.mall...@gmail.com
> Subject: Debugging your environment...
>  
> Hi, Tevin,
>  
> After our conversation via irc, please do the following:
>  
> * Let's assume you have the "C:\Users\Tevin\Desktop\project"
> directory 
> created already, with no "perceval" subdirectory in it.
>  
> * Run "venv C:\Users\Tevin\Desktop\project\perceval"
>  
> Now you should have a "C:\Users\Tevin\Desktop\project\perceval"
> directory. Change to it:
>  
> * "cd C:\Users\Tevin\Desktop\project\perceval"
>  
> * Get a listing of its contents, and in particular of its bin
> directory:
>  
> "dir ."
> "dir bin"
>  
> * Now, activate the environment.
>  
> * Once the environment is activated, run
>  
> "pip install perceval"
>  
> And then again
>  
> "dir ."
> "dir bin"
>  
> * and let's get some environment variables:
>  
> "echo %PATH%"
> "echo %PYTHONPATH"
>  
> * Then, try to run perceval:
>  
> "perceval"
>  
> * and python:
>  
> "python --version"
>  
> (of course, after activation, everything from the same console).
>  
> Plese, copy and paste in a message all what you typed and the answer
> by
> the system. And let's see if we can progress from that...
>  
> Saludos,
>  
>     Jesus.
>  
>  
>  
> --
> http://twitter.com/jgbarah  http://gsyc.es/~jgb
>  
>  
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IRC Meeting Logs Mon October 3, 2016

[Jesus M. Gonzalez-Barahona]

No problem with file extensions. Text is text ;-) Thanks!

[Well, maybe utf-8 encoding would be nice, but our conversations being
English, old good ASCII would be perfect as well]

Jesus.

On Fri, 2016-10-07 at 13:30 -0400, tevin.k.mall...@gmail.com wrote:
> Sorry about that Jesus. Is there a file type you prefer to have the
> logs in? I have it attached to this email in .txt . If you like for
> me to change it let me know.
>  
> Sent from Mail for Windows 10
>  
> From: Jesus M. Gonzalez-Barahona
> Sent: Friday, October 7, 2016 11:59 AM
> To: tevin.k.mall...@gmail.com; Lars Kurth
> Cc: Xen-Devel@Lists. Xenproject. Org
> Subject: Re: IRC Meeting Logs Mon October 3, 2016
>  
> On Fri, 2016-10-07 at 11:24 -0400, tevin.k.mall...@gmail.com wrote:
> > Hello Jesus and Lars, as requested here are the logs from today's
> IRC
> > meeting. It was really helpful to have your input Jesus, since you
> > have a lot of experience. I'm confident things will run smoothly as
> I
> > continue my micro-task. I really appreciate your guidance, and the
> > time you spent with me today.
> >  
> > -Tevin Mallory
>  
> Thanks, Tevin!
>  
> Usually, a text version is more useful. Please find one for our
> session
> below. If your IM program doesn't allow for saving sessions, usually
> you can just copy & paste from it to get this text version.
>  
> Saludos,
>  
>     Jesus.
>  
> ---
> (04:33:18 PM) tmallory: Hello Everyone, I am Tevin Mallory. How are
> you
> all doing today?
> (04:34:13 PM) jgbarah: Hi Tevin!
> (04:34:50 PM) jgbarah: tmallory: How do you do? Is the hurricane
> being
> a problem?
> (04:35:41 PM) jgbarah: tmallory: Are you there?
> (04:36:02 PM) tmallory: I am doing well so far, the hurrican hasn't
> hit
> my area yet.
> (04:37:09 PM) jgbarah: ok. I hope it doesn't become a big problem for
> you...
> (04:37:24 PM) jgbarah: Anyway, could you have a look a the microtask
> description?
> (04:37:55 PM) tmallory: Thank you. I hope so as well. I have the
> microtask description in fornt me.
> (04:38:51 PM) tmallory: I started on it yesterday and have been
> installing the requirements for perceval
> (04:39:14 PM) jgbarah: ok. I have to leave in 10 min, but will be
> back
> in 30, so let's start, and if needed and you can, we can follow up
> later
> (04:39:25 PM) jgbarah: Good. Any trouuble with Perceval?
> (04:39:45 PM) tmallory: Okay
> (04:40:25 PM) tmallory: I haven't started using Perceval yet, but i
> shouldn't have any trouble with it.
> (04:41:33 PM) tmallory: I just started learning python a little while
> ago, so i spent a great amount of time getting a good grasp
> (04:42:07 PM) jgbarah: OK. The other stuff that could block you is
> installing ElasticSearch and maybe Kibana. I would start with
> ElasticSearch, which is all you'll need for onow
> (04:42:37 PM) tmallory: Ok i'll work on installing that next
> (04:42:37 PM) jgbarah: ok, let me know if you need some help for
> Python. I can point you to courses and material if you happen to need
> those
> (04:43:03 PM) jgbarah: For elasticSearch, you'll mainly need a
> working
> Java vm on your computer, not much more than that.
> (04:43:51 PM) tmallory: Okay I'am taking note of that
> (04:44:22 PM) jgbarah: Can I help you in some way now, or you're on
> your way?
> (04:45:17 PM) tmallory: I am pretty much on my way.
> (04:45:33 PM) tmallory: no major problems
> (04:45:49 PM) jgbarah: Perfect. Then, maybe we can just schedule a
> meeting for next week
> (04:45:59 PM) jgbarah: Meanwhile you can ping me here, or via email
> (04:46:13 PM) tmallory: Ok. perfect
> (04:46:57 PM) jgbarah: For the meeting here, we can consider
> tentatively next Friday, but 45 min. later
> (04:47:03 PM) jgbarah: Is that ok with you?
> (04:47:18 PM) tmallory: yes it is great
> (04:47:20 PM) jgbarah: I guess that would be 17:15 CEST, 11:15am EST
> (04:47:26 PM) jgbarah: Good!
> (04:47:51 PM) jgbarah: There is a slight chance that I cannot make
> that
> time, in that case I will suggest meeting a bit later
> (04:47:57 PM) jgbarah: Anthing else from your side?
> (04:48:32 PM) tmallory: No I am all green lights on my side, if
> anything comes up I'll let you know
> (04:49:22 PM) tmallory: Thank you for your time.
> (04:49:38 PM) jgbarah: Perfect! It was great meeting you. See you
> next
> week! If you don't mind, please send Lars and me the log for his
> meeting via email
> (04:50:32 PM) tmallory: No problem, I'll send it right away.
> (04:50:52 PM) tmallory: It was great meeting with you as well.
> (04:51:01 PM) jgbarah: Thanks! See you. Bye for now!
>  
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>  
>  
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IRC Meeting Logs Mon October 3, 2016

[Jesus M. Gonzalez-Barahona]

On Fri, 2016-10-07 at 11:24 -0400, tevin.k.mall...@gmail.com wrote:
> Hello Jesus and Lars, as requested here are the logs from today's IRC
> meeting. It was really helpful to have your input Jesus, since you
> have a lot of experience. I'm confident things will run smoothly as I
> continue my micro-task. I really appreciate your guidance, and the
> time you spent with me today.
>  
> -Tevin Mallory

Thanks, Tevin!

Usually, a text version is more useful. Please find one for our session
below. If your IM program doesn't allow for saving sessions, usually
you can just copy & paste from it to get this text version.

Saludos,

Jesus.

---
(04:33:18 PM) tmallory: Hello Everyone, I am Tevin Mallory. How are you
all doing today?
(04:34:13 PM) jgbarah: Hi Tevin!
(04:34:50 PM) jgbarah: tmallory: How do you do? Is the hurricane being
a problem?
(04:35:41 PM) jgbarah: tmallory: Are you there?
(04:36:02 PM) tmallory: I am doing well so far, the hurrican hasn't hit
my area yet.
(04:37:09 PM) jgbarah: ok. I hope it doesn't become a big problem for
you...
(04:37:24 PM) jgbarah: Anyway, could you have a look a the microtask
description?
(04:37:55 PM) tmallory: Thank you. I hope so as well. I have the
microtask description in fornt me.
(04:38:51 PM) tmallory: I started on it yesterday and have been
installing the requirements for perceval
(04:39:14 PM) jgbarah: ok. I have to leave in 10 min, but will be back
in 30, so let's start, and if needed and you can, we can follow up
later
(04:39:25 PM) jgbarah: Good. Any trouuble with Perceval?
(04:39:45 PM) tmallory: Okay
(04:40:25 PM) tmallory: I haven't started using Perceval yet, but i
shouldn't have any trouble with it.
(04:41:33 PM) tmallory: I just started learning python a little while
ago, so i spent a great amount of time getting a good grasp
(04:42:07 PM) jgbarah: OK. The other stuff that could block you is
installing ElasticSearch and maybe Kibana. I would start with
ElasticSearch, which is all you'll need for onow
(04:42:37 PM) tmallory: Ok i'll work on installing that next
(04:42:37 PM) jgbarah: ok, let me know if you need some help for
Python. I can point you to courses and material if you happen to need
those
(04:43:03 PM) jgbarah: For elasticSearch, you'll mainly need a working
Java vm on your computer, not much more than that.
(04:43:51 PM) tmallory: Okay I'am taking note of that
(04:44:22 PM) jgbarah: Can I help you in some way now, or you're on
your way?
(04:45:17 PM) tmallory: I am pretty much on my way.
(04:45:33 PM) tmallory: no major problems
(04:45:49 PM) jgbarah: Perfect. Then, maybe we can just schedule a
meeting for next week
(04:45:59 PM) jgbarah: Meanwhile you can ping me here, or via email
(04:46:13 PM) tmallory: Ok. perfect
(04:46:57 PM) jgbarah: For the meeting here, we can consider
tentatively next Friday, but 45 min. later
(04:47:03 PM) jgbarah: Is that ok with you?
(04:47:18 PM) tmallory: yes it is great
(04:47:20 PM) jgbarah: I guess that would be 17:15 CEST, 11:15am EST
(04:47:26 PM) jgbarah: Good!
(04:47:51 PM) jgbarah: There is a slight chance that I cannot make that
time, in that case I will suggest meeting a bit later
(04:47:57 PM) jgbarah: Anthing else from your side?
(04:48:32 PM) tmallory: No I am all green lights on my side, if
anything comes up I'll let you know
(04:49:22 PM) tmallory: Thank you for your time.
(04:49:38 PM) jgbarah: Perfect! It was great meeting you. See you next
week! If you don't mind, please send Lars and me the log for his
meeting via email
(04:50:32 PM) tmallory: No problem, I'll send it right away.
(04:50:52 PM) tmallory: It was great meeting with you as well.
(04:51:01 PM) jgbarah: Thanks! See you. Bye for now!
 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen Code Review Dashboard - Outreachy Program Project

[Jesus M. Gonzalez-Barahona]

On Tue, 2016-10-04 at 20:18 -0400, tevin.k.mall...@gmail.com wrote:
> Hello Jesus!
>  
> Thank you for taking the time out of your busy schedule to responded.
> I would love a summary of the small contribution over email, as this
> will allow me to get started on the project sooner. I am located in
> Florida USA, on Eastern Daylight Time and available anytime on
> October 6th to discuss details. If you have certain times that work
> best for you I can easily adjust my schedule. Just let me know when
> you would like to chat via IRC and I will be there. For the most part
> my schedule is very flexible, I am always available on Mondays,
> Tuesdays and Thursdays at anytime. Thank you once again and I look
> forward to hearing from you.

OK, here we are. With respect to the project in general, I assume
you're familiar with

https://wiki.xenproject.org/wiki/Outreach_Program_Projects#Xen_Code_Rev
iew_Dashboard

Right?

The main aim of the project is to reproduce all the process up to
having something like https://xen.biterg.io/ But using only GrimoireLab
tools

http://grimoirelab.github.io

Now it is using a mixture of MetricsGrimoire and custom scripts

The first step is to get information both from mailing lists and git
repositories using Perceval, and storing it in ElasticSearch. Later,
there are some scripts that should be ported to use this ElasticSearch
data (instead of the SQL data they are using now). With that, produce
the ElasticSearch indexes for the dashboard. Then, if possible, improve
the dashbosrd and make it more useful for the Xen community

Now, about the microtask.

I guess you know you can start with one microtask to show that you are
likely to be the right person for this project, according to Outreachy
requirements.

In this case, the microtask would be getting data from a mailing list
with Perceval. The mailing list is xen-devel. You get its archives
analyzed by Perceval, and store the resulting raw index in
ElasticSearch.

Once you're good with this, you do the same for the Xen git repo

And once you're good with this, you write an script which produces a
new index with some information for each commit, plus the branch in
which it was committed

Mboxes are at

https://lists.xenproject.org/archives/html/mbox/

They are then named xen-devel--, e.g. xen-devel-2016-09

You can write a little script using wget or curl to downoad several of
them at once. To begin with, you can start with some of them (say 5-10)

The code contribution result of the microtask would be the
identification of the branches, based on the output of Perceval / git

The setup would be you getting all the info from some mboxes in
ElasticSearch, a git repo in ElasticSearch, and a simple index,
combined of both plus branches information, again in ElasticSearch

Once you have those in Elasticsearch, just produce the result of
querying some of the items in ElasticSearch with curl, and the code for
the identification of branches. All of this can be stored in a git
repository for verification.

I can support you via irc and email if you have any trouble.

We're compiling some information on how to use GrimoireLib in

https://jgbarah.gitbooks.io/grimoirelab-training

Maybe that's a good place to start.

Is all of this ok with you?

Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen Code Review Dashboard - Outreachy Program Project

[Jesus M. Gonzalez-Barahona]

Hi Tevin!

If you want, I can define a small contribution for you. Let me know,
and I can summarize over email, and if needed we can decide on some IRC
time to discuss details (let me know about your timezone and
availability to find some convenient slot).

Saludos,

Jesus.

On Tue, 2016-10-04 at 18:03 -0400, tevin.k.mall...@gmail.com wrote:
> Hello Xen Project Team!
> My name is Tevin Mallory, I am graduate student with a knack for data
> analytics and a deep interest in coding. I wish to take a moment to
> introduce myself to you by email to say hello. I´m enthusiastic about
> applying to join this team as an Outreachy intern and look forward to
> contacting everyone.  I have included the project I am interested in
> joining below:
>  
> 
> Skills Needed: SQL, Java/Javascript, HTML5/XML skills, basic software
> design knowledge (working with the mentors)
> 
> Description: The code review process in Xen is being analysed using
> MetricsGrimoire tools (correlating email based reviews with git
> commits in Xen Project trees to cover the entire workflow). The data
> is then stored in an SQL database and visualised using a Kibana based
> dashboard and some custom reports (e.g. [1]). The main objectives of
> this project is to extend the existing tools, to
> 
> To produce a Perceval-based script to analyse the code review
> messages in Xen (instead of the original MLStats/CVSAnalY-based
> scripts). This would include taking the output of our current
> prototype scripts, and converting them into a more mature script,
> using information produced by Perceval.
> 
> To enrich that information as is needed, based on existing custom
> reports, to produce the ElasticSearch indexes that we use for the
> dashboards.
> 
> If time, to work with the Xen Project developer community on
> extending the dashboards themselves (note that this part may not be
> needed and depends on engagement with the developer communities'
> needs).
> 
> If time, we could consider testing/extending the heuristics developed
> for Xen Project to work with other Linux-related projects, and maybe
> Linux itself.
> 
> I have coded a website for a small business using HTML5, CSS and same
> JavaScript. I plan to start a career in marketing research analytics
> and have learn to use SQL. However I am still at the beginner level
> in fully utilizing it and would love the opportunity to develop my
> skills in this project.  I would love further discuss ways in which I
> can get involved with this project, maybe I can receive a task making
> a code contribution. I know there is an IRC for this project and was
> wonder if you would prefer me to contact you all though that, instead
> of email.  Please and thank you for your time. You can reach me at
> tevin.k.mall...@gmail.com and my GitHub: https://github.com/CodeCaste
> r-MoonT .
>  
> Sincerely,
> Tevin K. Mallory
>  
>  
> Sent from Mail for Windows 10
>  
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

Ooops. Sorry for the delay in answering. Yes, this is going to be
offered in the context of Outreachy once again. If you are still
interested, we can schedule an IRC meeting to start guiding you to your
first contribution.

Saludos,

Jesus.

On Tue, 2016-09-06 at 20:57 +0530, Anubha Aggarwal wrote:
> Yes in the context of outreachy.
> On 6 Sep 2016 20:43, "Jesus M. Gonzalez-Barahona" <j...@bitergia.com>
> wrote:
> > Hi, Anubha,
> > 
> > Are you commenting about this in the context of Outreacy or
> > anything
> > else?
> > 
> >         Jesus.
> > 
> > On Tue, 2016-09-06 at 19:11 +0530, Anubha Aggarwal wrote:
> > > Hello i am Anubha . I am interested to do work in this project .
> > I am
> > > good at Java, Xml and Sql.
> > > So can you guide me how i can make my first contribution to it.
> > > Tnks
> > >
> > >
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

Hi, Anubha,

Are you commenting about this in the context of Outreacy or anything
else?

Jesus.

On Tue, 2016-09-06 at 19:11 +0530, Anubha Aggarwal wrote:
> Hello i am Anubha . I am interested to do work in this project . I am
> good at Java, Xml and Sql.
> So can you guide me how i can make my first contribution to it.
> Tnks
> 
> 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

Thanks a lot. I can run it now, with the latest version of Perceval,
and at first glance seems to work. Some improvements could be done,
but it seems to work. Since we're quite close to the evaluation for the
microtask, let's stop here, except for the tests. Please, produce some,
to validate at least a bit the code...

Saludos,

Jesus.

On Thu, 2016-04-14 at 22:41 +0530, Priya wrote:
> Hello Jesus, 
> 
> I had made changes to my code to work with the latest version of
> Perceval, you can see my latest commit [1]. Let me know if come
> across issues?
> 
> I am working on the testing part now, stuck with few issues. Hoping
> to complete by tomorrow or day after. 
> 
> 
> [1]:https://github.com/priya299/Dashboard/commit/150e259c22b36b359f79
> ea711ba4e294d0b0c9ab
> 
> 
> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> 
> On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <jgb@bite
> rgia.com> wrote:
> > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> > > Hello,
> > >
> > > I tried running the same command in new version of perceval.  I
> > found
> > > the following missing message id errors in
> > perceval_mbox_parse.log
> > > file. I am working on the testing part and I will be able to
> > finish
> > > it in one or two days.
> > >
> > > You can see the errors here [1]
> > >
> > > [1]:http://imgur.com/yVsIoCT
> > 
> > Hi, Priya. I'm not sure about what exactly is causing your
> > messages,
> > since I cannot reproduce them (see below). But I still suspect that
> > they may happen because in current versions of Perceval the data
> > parsed
> > from an mbox is no longer stored as first level key/data in the
> > dictionary returned by Perceval for each message, but in data for
> > key
> > "data", which is itself a dictionary.
> > 
> > In particular, in the code:
> > 
> > -
> >               for k in msg_json:
> > try:
> > if key == k['Message-ID'].strip('<>'):
> > k['property'] = key
> > -
> > 
> > probably you should be checking for k['data']['Message-ID'] instead
> > of
> > just k['Message-ID'].
> > 
> > Please, have a look at how recent versions of Perceval produce the
> > dictionaries for each message...
> > 
> > But as I said, I cannot reproduce your error. When running your
> > most
> > recent code right now (9a5abc47bbab3b06550) with the most recent
> > Perceval/master code (53efc14001c806f0452) I get:
> > 
> > 
> > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> > createjson.py --mbox advisory-board-2014-02 --output new.json
> > Traceback (most recent call last):
> >   File "createjson.py", line 96, in 
> > main()
> >   File "createjson.py", line 92, in main
> > mparser.create_json(args.mbox,args.output)
> >   File "createjson.py", line 59, in create_json
> > messages = th.message_details(mbox_files)
> >   File
> > "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
> > line 338, in message_details
> > urllib.request.urlretrieve(filename, 'mbox')
> >   File "/usr/lib/python3.4/urllib/request.py", line 186, in
> > urlretrieve
> > with contextlib.closing(urlopen(url, data)) as fp:
> >   File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
> > return opener.open(url, data, timeout)
> >   File "/usr/lib/python3.4/urllib/request.py", line 449, in open
> > req = Request(fullurl, data)
> >   File "/usr/lib/python3.4/urllib/request.py", line 267, in
> > __init__
> > self.full_url = url
> >   File "/usr/lib/python3.4/urllib/request.py", line 293, in
> > full_url
> > self._parse()
> >   File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
> > raise ValueError("unknown url type: %r" % self.full_url)
> > ValueError: unknown url type: 'advisory-board-2014-02'
> > -
> > 
> > Could you please try to checkout and install exactly the same
> > version
> > of Perceval I'm using, and see if you get the same error? And if
> > the
> > above problem with the format returned by Perceval persists, maybe
> > you
> > can fix that too.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> Hello,
> 
> I tried running the same command in new version of perceval.  I found
> the following missing message id errors in perceval_mbox_parse.log
> file. I am working on the testing part and I will be able to finish
> it in one or two days.
> 
> You can see the errors here [1]
> 
> [1]:http://imgur.com/yVsIoCT

Hi, Priya. I'm not sure about what exactly is causing your messages,
since I cannot reproduce them (see below). But I still suspect that
they may happen because in current versions of Perceval the data parsed
from an mbox is no longer stored as first level key/data in the
dictionary returned by Perceval for each message, but in data for key
"data", which is itself a dictionary.

In particular, in the code:

-
              for k in msg_json:
try:
if key == k['Message-ID'].strip('<>'):
k['property'] = key
-

probably you should be checking for k['data']['Message-ID'] instead of
just k['Message-ID'].

Please, have a look at how recent versions of Perceval produce the
dictionaries for each message...

But as I said, I cannot reproduce your error. When running your most
recent code right now (9a5abc47bbab3b06550) with the most recent
Perceval/master code (53efc14001c806f0452) I get:


(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox advisory-board-2014-02 --output new.json
Traceback (most recent call last):
  File "createjson.py", line 96, in 
main()
  File "createjson.py", line 92, in main
mparser.create_json(args.mbox,args.output)
  File "createjson.py", line 59, in create_json
messages = th.message_details(mbox_files)
  File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
line 338, in message_details
urllib.request.urlretrieve(filename, 'mbox')
  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 449, in open
req = Request(fullurl, data)
  File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__
self.full_url = url
  File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url
self._parse()
  File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'advisory-board-2014-02'
-

Could you please try to checkout and install exactly the same version
of Perceval I'm using, and see if you get the same error? And if the
above problem with the format returned by Perceval persists, maybe you
can fix that too.

Saludos,

Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2016-04-07 at 17:57 +0530, Priya wrote:
> Hello all,
> 
> Thanks for the suggestions. I have updated the changes as u had
> mentioned. I am sorry, but I could not find any errors while running 
> 
> $ python3 createjson.py --mbox xen-devel-2016-03 --output new.json 
> 
> command. I am wondering what is wrong with this and might be a
> problem with python3 or my perceval version. I have added licensing
> and python logging. You can see it in my github repo [1]. I will try
> upgrading perceval and adding in the tests in the coming days, and
> will update.

Priya, I guess I didn't explain the source for the error clearly
enough, sorry about that. It is very likely that you're not running the
latest version of Perceval, which changed the format for the items it
produces some days ago. That's why, if you're running a version of
Perceval of more than some days ago, it works, but if you run the
latest one, it doesn't work (at least for me). I'm not completely sure,
but that's why I asked you to upgrade to the latest version (either
master HEAD or version 0.1.0). If you don't see the error once you do
this, please let me know.

Thanks,

Jesus.

> [1]: https://github.com/priya299/Dashboard
> 
> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> 
> On Thu, Apr 7, 2016 at 3:29 AM, Jesus M. Gonzalez-Barahona <jgb@biter
> gia.com> wrote:
> > On Wed, 2016-04-06 at 17:30 +0530, Priya wrote:
> > > Hello,
> > >
> > > Thanks for your suggestions.
> > > I have made the appropriate changes as you had mentioned.
> > > It took a little time to change from python3 to python3.4 as
> > perceval
> > > supports python3.4. I have updated the changes in my github. You
> > can
> > > see my git repo [1]
> > >
> > > [1]:https://github.com/priya299/Dashboard
> > 
> > Thanks a lot, Priya. Good work. Some preliminary comments, below.
> > 
> > * When runing the script on the xen-devel-2016-03 mbox, I seen an
> > exception raised:
> > 
> > 
> > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> > createjson.py --mbox xen-devel-2016-03 --output new.json
> > Traceback (most recent call last):
> >   File "createjson.py", line 61, in 
> > create_json(args.mbox,args.output)
> >   File "createjson.py", line 43, in create_json
> > if key == k['Message-ID'].strip('<>'):
> > KeyError: 'Message-ID'
> > 
> > 
> > Maybe some message is not having a Message-ID field? I suggest that
> > you
> > capture this exception, print out the offending message, and go on
> > with
> > the next one. You can use the Python logging package for printing
> > out
> > this kind of information (you can see how to use it in the Perceval
> > package itself). But see below.
> > 
> > * Minor typo in the README:
> > 
> > Instead of 
> > 
> > eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json
> > 
> > it should be
> > 
> > eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output
> > new.json
> > 
> > * The files have no licensing info. If you agree, it could be
> > GPLv3, as
> > is Perceval itself. For that, it would be enough that you mimic the
> > header in Perceval files in your Python files (of course,
> > indicating
> > your authorship information).
> > 
> > * Which version of Perceval are you using? Some weeks ago, the
> > format
> > of the dictionary produced by Perceval for each message changed.
> > Now
> > the  actual fields of the message are in a data subdictionary.
> > Please,
> > check that: the above exception with respect to the Message-ID key
> > could be because of this... Please, try to make it work with master
> > HEAD for Perceval (I don't expect any new major change in the next
> > days/weeks, and I'll try to warn you in case some happens).
> > 
> > * Could you please write at least one unit test for your code? You
> > can
> > see examples of the testing schema we use in the tests directory in
> > Perceval, but we use vanilla unittest (the Python package for
> > tests).
> > At this stage I don't need that you produce a whole set of tests,
> > only
> > one or two to show that you know how to write unit tests, please.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Priya V
> > > Amrita University
> > > LinkedIn | GitHub | Bitbucket
> > >
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Wed, 2016-04-06 at 17:30 +0530, Priya wrote:
> Hello,
> 
> Thanks for your suggestions.
> I have made the appropriate changes as you had mentioned.
> It took a little time to change from python3 to python3.4 as perceval
> supports python3.4. I have updated the changes in my github. You can
> see my git repo [1]
> 
> [1]:https://github.com/priya299/Dashboard

Thanks a lot, Priya. Good work. Some preliminary comments, below.

* When runing the script on the xen-devel-2016-03 mbox, I seen an
exception raised:


(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox xen-devel-2016-03 --output new.json
Traceback (most recent call last):
  File "createjson.py", line 61, in 
create_json(args.mbox,args.output)
  File "createjson.py", line 43, in create_json
if key == k['Message-ID'].strip('<>'):
KeyError: 'Message-ID'


Maybe some message is not having a Message-ID field? I suggest that you
capture this exception, print out the offending message, and go on with
the next one. You can use the Python logging package for printing out
this kind of information (you can see how to use it in the Perceval
package itself). But see below.

* Minor typo in the README:

Instead of 

eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json

it should be

eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output new.json

* The files have no licensing info. If you agree, it could be GPLv3, as
is Perceval itself. For that, it would be enough that you mimic the
header in Perceval files in your Python files (of course, indicating
your authorship information).

* Which version of Perceval are you using? Some weeks ago, the format
of the dictionary produced by Perceval for each message changed. Now
the  actual fields of the message are in a data subdictionary. Please,
check that: the above exception with respect to the Message-ID key
could be because of this... Please, try to make it work with master
HEAD for Perceval (I don't expect any new major change in the next
days/weeks, and I'll try to warn you in case some happens).

* Could you please write at least one unit test for your code? You can
see examples of the testing schema we use in the tests directory in
Perceval, but we use vanilla unittest (the Python package for tests).
At this stage I don't need that you produce a whole set of tests, only
one or two to show that you know how to write unit tests, please.

Saludos,

Jesus.

> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Tue, 2016-04-05 at 22:05 +0530, Priya wrote:
> Hello all, 
> 
> I have completed coding the initial task of grouping the email thread
> using the Zawinski algorithms and then adding property entity to the
> json for the messages that belong to the same email thread. 
> 
> You can see my git repo [1]. The new.json is the output of my script
> and out.json is the output of Perceval. 
> 
> Also, I have updated the README.md file regarding the execution
> procedures in github.
> 
> Instructions
> 
> 
> git clone https://github.com/priya299/Dashboard.git
> 
> cd Dashboard
> 
> python createjson.py 'Perceval Ouputfile' 'mbox file' 'output_file'
> 
> eg: python createjson.py out.json xen-devel-2016-03 new.json
> 
> "new.json" json file will be created with each message belong to a
> single thread having an additional attribute "property". The property
> attribute will have message id of the first message in the thread.
> 
> Now, I will be pushing the new.json into the elastic search db[2].
> Please give me your valuable feedback about my progress. 
> 
> [1]:https://github.com/priya299/Dashboard
> [2]:https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html

Hi, Priya. To begin with, could you please integrate your code with the
Perceval iterator? In other words, you can run Perceval on the mailing
list archive directly from your code, which will render the use of
"out.json" void. That way, the invocation of the script would be more
like:

python createjson.py xen-devel-2016-03 new.json

In other words, create.json would use Perceval to parse the mailing
list archive. For this end, the Perceval mbox backend is a class, which
once instantiated, provides an iterator function, fetch(), that you can
run inside a loop. For each iteration of the loop, you get the
equivalent to a JSON element in out.json.

The code would be similar to:

---
import perceval

mbox_parser = perceval.backends.mbox.MBox(
  origin=mbox_url,
  dirpath=mbox_file_name
)
for item in mbox_parser.fetch():
  thread_id = find_thread(item)
  ...
-

Some details about the Perceval mbox class:

http://perceval.readthedocs.org/en/master/perceval.backends.html#module
-perceval.backends.mbox

If you have trouble running the Perceval backend as an iterator, please
let me know.

In addition, you can use argparse for reading the arguments in the
command line. It is easy and convenient.

Saludos,

Jesus.

> 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

[Jesus M. Gonzalez-Barahona]

On Thu, 2015-10-15 at 22:36 +0100, Lars Kurth wrote:
> > On 15 Oct 2015, at 10:26, Ian Campbell 
> > wrote:
> > 
> > On Thu, 2015-10-15 at 10:06 +0100, Ian Campbell wrote:
> > > On Wed, 2015-10-14 at 18:32 +0100, Lars Kurth wrote:
> > > > C1) Only 60% percent of the reviews on the mailing list could
> > > > be
> > > > matched
> > > > to commits. This can be improved going forward, but we felt
> > > > that the
> > > > dataset is big enough for statical analysis and didn't want to
> > > > spend
> > > > too
> > > > much time to get the matching perfect at this stage. See
> > > > "Coverage
> > > > analysis" for more details
> > > 
> > > How strict or fuzzy is the matching?
> > > 
> > > Does it account for e.g. spelling, grammar and clarity changes
> > > and things
> > > like adding a subsystem ("tools: libxc:") prefix, either upon
> > > commit or
> > > by
> > > the author in vN+1 based on feedback?
> > > 
> > > I often both comment on such things during review and (with the
> > > authors
> > > permission) tweak things upon commit.
> > > 
> > > If those changes are not being correlated then I expect that
> > > would skew
> > > the
> > > figures of those for whom English is not their first language
> > > (and not a
> > > small portion of native speakers even!) and newcomers who e.g.
> > > might not
> > > be
> > > aware of the need to prefix things with the subsystem.
> > > 
> > > In a (smaller) number of cases a patch is abandoned in favour of
> > > a very
> > > different approach, which I think would be essentially
> > > untrackable, at
> > > least automatically.
> > 
> > Looking at the stuff in [47] marked as last reviewed in 2014 it
> > seems the
> > majority of them (at least the ones for which I am involved as a
> > maintainer
> > etc) can be explained by one of these factors, just going from my
> > memory of
> > things having been fixed in one way or another.
> 
> I think you are right: we hardly spent any time on more intelligent
> matching.

Yes. We tried to get to a meaningful sample, assuming the skew was
small enough to draw conclusions on the duration of the review process,
which was the main target at this stage. As Lars mentions in some other
message, the nice thing is that once we improve the matching
heuristics, the rest of the analysis can be run automatically, which
means we would get more accurate results.

At this point, without more careful validation, we started to be afraid
of having false positives (relaxing the matching rules to a point where
they start matching messages and commits that are not really the same).

> > There also looks to be identical titles (e.g. "x86: Full support of
> > PAT")
> > being listed there more than once.
> 
> Will have to look at this one

Yes. Maybe some whitespace difference or something...

> [...]

Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

[Jesus M. Gonzalez-Barahona]

On Thu, 2015-10-15 at 12:58 +0100, Wei Liu wrote:
> On Wed, Oct 14, 2015 at 06:32:37PM +0100, Lars Kurth wrote:
> [...]
> > 
> > == Cycle Time [27] to [29] ==
> > This is not quite obvious from the definition in 1. For the cycle
> > time, we basically ignore the mapping to a patch or patch series
> > and just look at the cycle times between review iterations
> > *regardless* of which series it belongs to.
> > 
> > For example [29] then tells us that the median cycle of any code
> > review has been fairly static around 5 days from 2012.
> > 
> > == Backlog Analysis ==
> > This section shows us the total of patch series reviews that could
> > be modelled (60%) over the project's life-time 
> > 
> > Complete: 5641
> > Active (aka activity in the last 7 days) : 78
> > Ongoing (aka activity in the last 12 months): 403
> > Stalled (no activity for 12 months): 600
> 
> Do note that this category covers several things: patches no longer
> needed, patches superseded by newer version but with different
> titles,
> patches to be upstreamed but lost momentum, so I wouldn't use this as
> primary index for how bad the community functions. In fact I think it
> shouldn't be given the same importance as the other two.

I agree. In fact, this data was not the core for the study, which was
more focused on how long the review process is. As I commented in
another message, we would need to more clearly define "stalled" and if
possible "abandoned", but that is not easy.

> >  
> > This is an area where time based diagrams would help, such that we
> > understand whether things have become much worse than in the past.
> > On the face of it, the figures are *worrying*: in particular if you
> > consider we historically have handled an average of 513 patch
> > series a year, we have the equivalent of 
> > - a year's worth of backlog which is active
> > - a bit more than a year's backlog which is stalled - there is no
> > way to identify abandoned reviews  
> > 
> 
> I notice that you noticed the same thing. :-)

>From a "problem identification" point of view, if you had a way of
telling apart the abandoned from the stalled, it would be great.
Otherwise, it is very difficult to draw conclusions. If for those 600
reviews, 550 are for example due to superseded and no longer needed
patches, you only have 50 stalled patches. That's a very different
number than if say 500 are really stalled...

So, I guess we need better matching and some kind of labeling of
abandoned reviews to have any conclusion about the real backlog, and
its trends.

Saludos,

Jesus..

> Wei.
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

[Jesus M. Gonzalez-Barahona]

On Thu, 2015-10-15 at 22:36 +0100, Lars Kurth wrote:
> > On 15 Oct 2015, at 10:26, Ian Campbell 
> > wrote:
> > 
> > On Thu, 2015-10-15 at 10:06 +0100, Ian Campbell wrote:

> > > On Wed, 2015-10-14 at 18:32 +0100, Lars Kurth wrote:
> [...]

> That is correct and a case we need to look at if it's worthwhile and
> possible to fix it. However the fact remains that pretty much all the
> graphs cover data for completed reviews only, except for backlog
> data. We have two options
> 1) Spend some effort trying to fix it
> 2) Accept that "stalled" reviews are not that meaningful
> Not sure what the answer is at this stage

I guess there are three possible solutions here:

* To send some message to the mailing list to "abandon" or close a
review. This would allow us to detect those reviews, and everyone to
know that they are not really stalled. But this would require changes
to your policy, I assume.

* To consider that when there is no activity for a certain period, the
review is no longer going to progress, and can be considered abandoned.
The main trouble with this could be that we have seen some patch series
inactive for very long periods, and still coming back to life after
that. But being a very small fraction of the cases, for statistical
purposes those could be considered as abandoned.

* To label "by hand" the reviews that are abandoned, by some of you
knowing about the project. But this is (I assume) too time-consuming
and probably error-prone

> > I'm afraid that on that basis I think C1 has skewed the conclusion
> > that
> > there are 600 stalled series, possibly by a considerable factor.
> 
> I agree with that and should have maybe highlighted this more
> clearly.
> 
> For the other data, it is of course also possible that there is some
> skew. But given that we had 60% of a sample (and that the 60% is
> likely higher as xen-devel also is CC'ed when patches are posted to
> QEMU, Linux, ...) the sample is big enough to make significant skew
> very unlikely.

Yes, I think so. In any case, this will be improved when the heuristics
for matching improve.

Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel