Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Here is the basic structure i prepared.

https://jsfiddle.net/3r4vL4L3/5/

And i wanted to know what all am i supposed to show in installation
statistics mandatorily?

Top port Versions of a month?
Top os of that port?
Number of installations versus time ? (of that particular port)

what else?

Also in the port history table. Should we add another coloumn named os
. which will store the current os version??

Databse Spreadsheet :
*https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing
*

Thanks

On 2 April 2018 at 19:44, Vishnu  wrote:

> Hello,
>
> I would be getting back on the project from now.
> Sorry i could not deliver soon because i was having my exams going on.
> Now i would deliver my best.
>
> Thanks
>
> On 2 April 2018 at 00:36, Mojca Miklavec  wrote:
>
>> On 1 April 2018 at 20:33, Vishnu  wrote:
>> > Hello mojca
>> >
>> > I am planning on submitting the simple static form of the webpage(for
>> port).
>> >
>> > Had some doubts for the same. Is css required for that? As of now while
>> > showing you the structure?
>> >
>> > Right now its in plain html.
>> > That's  why.
>>
>> The very first step is to have a sound understanding and clear plan
>> for the content, there's no need to prepare css for that.
>>
>> Then we expect some proof of coding skills to build a simple django app.
>>
>> If you have time left, it would be nice to see a nicely looking html,
>> but you should concentrate on content for now and only add design at
>> the end.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello,

I would be getting back on the project from now.
Sorry i could not deliver soon because i was having my exams going on.
Now i would deliver my best.

Thanks

On 2 April 2018 at 00:36, Mojca Miklavec  wrote:

> On 1 April 2018 at 20:33, Vishnu  wrote:
> > Hello mojca
> >
> > I am planning on submitting the simple static form of the webpage(for
> port).
> >
> > Had some doubts for the same. Is css required for that? As of now while
> > showing you the structure?
> >
> > Right now its in plain html.
> > That's  why.
>
> The very first step is to have a sound understanding and clear plan
> for the content, there's no need to prepare css for that.
>
> Then we expect some proof of coding skills to build a simple django app.
>
> If you have time left, it would be nice to see a nicely looking html,
> but you should concentrate on content for now and only add design at
> the end.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

The database i am planning to make.
https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing

Please go through it.

On 26 March 2018 at 18:37, Mojca Miklavec  wrote:

> On 26 March 2018 at 14:47, Vishnu wrote:
> > Ok .
> > I will be writing Milestones seperately very soon.
> > What is your irc id? and on what channel are you available.i use
> freenode.
>
> You can join #macports
>
> > Also rather than irc could we use comments in docs.google to interact.And
> > simultaneously you can see the changes.That would quicken the process.
>
> The problem is that I don't get any feedback when you add a comment.
> So unless I stare in the document all the time ...
>
> > I have a good prior expeirence with sql.I had planned to use that
> instead of
> > log file.This idea had striked my mind to create table.As it would really
> > simplfy the statistics section.
> > But creating tables for so many ports would be good or not .is something
> i
> > cant decide.
>
> Databases are designed to handle terabytes of data in an efficient
> way. It's creating files for so many ports that should be questioned,
> not the other way around.
>
> > I can learn django in no time.As i mentioned i am a quick learner.
> > So i guess it would be wise to make everything in django.
> >
> > I will try to create the Static Website .If possible will submit that
> before
> > the deadline.
>
> That would be great. But again: it's not a big problem if you create a
> website a bit later. But if you fail to submit all the paperwork
> (proposal + enrolment proof) properly by tomorrow, you would not be
> eligible to participate at all.
>
> > I wanted to know if we can chat somewhere.
>
> Just join the #macports channel.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 26 March 2018 at 14:47, Vishnu wrote:
> Ok .
> I will be writing Milestones seperately very soon.
> What is your irc id? and on what channel are you available.i use freenode.

You can join #macports

> Also rather than irc could we use comments in docs.google to interact.And
> simultaneously you can see the changes.That would quicken the process.

The problem is that I don't get any feedback when you add a comment.
So unless I stare in the document all the time ...

> I have a good prior expeirence with sql.I had planned to use that instead of
> log file.This idea had striked my mind to create table.As it would really
> simplfy the statistics section.
> But creating tables for so many ports would be good or not .is something i
> cant decide.

Databases are designed to handle terabytes of data in an efficient
way. It's creating files for so many ports that should be questioned,
not the other way around.

> I can learn django in no time.As i mentioned i am a quick learner.
> So i guess it would be wise to make everything in django.
>
> I will try to create the Static Website .If possible will submit that before
> the deadline.

That would be great. But again: it's not a big problem if you create a
website a bit later. But if you fail to submit all the paperwork
(proposal + enrolment proof) properly by tomorrow, you would not be
eligible to participate at all.

> I wanted to know if we can chat somewhere.

Just join the #macports channel.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Ok .
I will be writing Milestones seperately very soon.
What is your irc id? and on what channel are you available.i use freenode.
Also rather than irc could we use comments in docs.google to interact.And
simultaneously you can see the changes.That would quicken the process.


I have a good prior expeirence with sql.I had planned to use that instead
of log file.This idea had striked my mind to create table.As it would
really simplfy the statistics section.
But creating tables for so many ports would be good or not .is something i
cant decide.

I can learn django in no time.As i mentioned i am a quick learner.
So i guess it would be wise to make everything in django.

I will try to create the Static Website .If possible will submit that
before the deadline.


I wanted to know if we can chat somewhere.

Thanks
Vishnu

On 26 March 2018 at 17:23, Mojca Miklavec  wrote:

> Dear Vishnu,
>
> First some general remarks, then the answers to your questions below.
>
> - One of the most important things that we should discuss and improve
> is the timeline. This is very little work, but it requires some
> coordination to make sure that they are defined in the most sensible
> way. One option is to discuss over IRC, but email is also fine. I'll
> write about that separately.
>
> - I suggest to write the milestones in a separate section. For example
> something along those lines:
>   May X: The first version of database design is ready, finish the
> first import of portindex.
>   May Y: The prototype website gets deployed at a temporary location
> xyz.com. It is possible to show the most basic properties of any port
> "foo" on a plain html page via xyz.com/index/port/foo and list all
> ports via xyz.com/index/list
>   ...
>   June Z: The website is accepting installation statistics submissions
> from users.
>
> - At the moment your schedule says "June 11-15: Phase 1 Evaluation"
> and no plans to code during those days and no explicit criteria that
> would define whether the project passed or not. It makes sense to put
> evaluation on the schedule, so that you see where it is, but note that
> this doesn't take any of your time, that's when mentors are busy and
> that's when you need to make sure to show results that were agreed on.
> (It can happen that there are valid reasons why the milestones would
> not be met, for example if the plan gets changed during the coding
> since a better solution or idea is found etc., so the milestones are
> not set in stone, but in case of disagreement between a student and
> mentor it helps a lot if it's straightforward to check for outsiders
> or admin as well.)
> I didn't check if those days are weekend, but don't make it sound like
> you would be spending all your time on evaluation. You only need to
> fill in a simple webform some time during the evaluation period,
> that's all.
>
> - Please do take a look at how relational databases work and get
> familiar with some basic Object-oriented programming in case you don't
> know those concepts already. This is of crucial importance for success
> of the project. In case you would work in Django (this is not the only
> option, there are other possibilities as well), you should be familiar
> with
> https://docs.djangoproject.com/en/2.0/intro/tutorial02/
> (this is part of a slightly longer tutorial). We need a single uniform
> website rather than something scraped from random chunks of scripts
> which do individual tasks without knowing about each other.
>
> - After the proposal gets submitted (because that's the highest
> priority task at the moment), we would still want to verify your
> skills with a simple coding challenge before we request slots (so
> ideally the challenge should be complete cca. one week after the
> deadline for proposal submission). Other organisations would ask for a
> pull request, but in this case it probably makes more sense to
> demonstrate a simple prototype. You may pick a slightly different
> task, but it should demonstrate comparable skillset or ability to
> learn. My suggestion would be to do these two tasks:
> (a) Create a simple (static) website for one single port of your
> choice and include nearly everything that will be part of the final
> product. You can copy some data from a Portfile to help you, but you
> can just as well make up the information, statistics etc. I would
> actually consider this to be part of the proposal. The page should
> demonstrate how the final product will look like (only with fake data)
> and will also help you while developing the actual functionality of
> the website: you would know where is it that you are heading. This
> should not take you more than a few hours to do, you basically need to
> visualize what should already be in the proposal.
> (b) Pick a framework to use and create a super simple hello-port
> application. In case you pick Django, go to
> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
> or
> 

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

First some general remarks, then the answers to your questions below.

- One of the most important things that we should discuss and improve
is the timeline. This is very little work, but it requires some
coordination to make sure that they are defined in the most sensible
way. One option is to discuss over IRC, but email is also fine. I'll
write about that separately.

- I suggest to write the milestones in a separate section. For example
something along those lines:
  May X: The first version of database design is ready, finish the
first import of portindex.
  May Y: The prototype website gets deployed at a temporary location
xyz.com. It is possible to show the most basic properties of any port
"foo" on a plain html page via xyz.com/index/port/foo and list all
ports via xyz.com/index/list
  ...
  June Z: The website is accepting installation statistics submissions
from users.

- At the moment your schedule says "June 11-15: Phase 1 Evaluation"
and no plans to code during those days and no explicit criteria that
would define whether the project passed or not. It makes sense to put
evaluation on the schedule, so that you see where it is, but note that
this doesn't take any of your time, that's when mentors are busy and
that's when you need to make sure to show results that were agreed on.
(It can happen that there are valid reasons why the milestones would
not be met, for example if the plan gets changed during the coding
since a better solution or idea is found etc., so the milestones are
not set in stone, but in case of disagreement between a student and
mentor it helps a lot if it's straightforward to check for outsiders
or admin as well.)
I didn't check if those days are weekend, but don't make it sound like
you would be spending all your time on evaluation. You only need to
fill in a simple webform some time during the evaluation period,
that's all.

- Please do take a look at how relational databases work and get
familiar with some basic Object-oriented programming in case you don't
know those concepts already. This is of crucial importance for success
of the project. In case you would work in Django (this is not the only
option, there are other possibilities as well), you should be familiar
with
https://docs.djangoproject.com/en/2.0/intro/tutorial02/
(this is part of a slightly longer tutorial). We need a single uniform
website rather than something scraped from random chunks of scripts
which do individual tasks without knowing about each other.

- After the proposal gets submitted (because that's the highest
priority task at the moment), we would still want to verify your
skills with a simple coding challenge before we request slots (so
ideally the challenge should be complete cca. one week after the
deadline for proposal submission). Other organisations would ask for a
pull request, but in this case it probably makes more sense to
demonstrate a simple prototype. You may pick a slightly different
task, but it should demonstrate comparable skillset or ability to
learn. My suggestion would be to do these two tasks:
(a) Create a simple (static) website for one single port of your
choice and include nearly everything that will be part of the final
product. You can copy some data from a Portfile to help you, but you
can just as well make up the information, statistics etc. I would
actually consider this to be part of the proposal. The page should
demonstrate how the final product will look like (only with fake data)
and will also help you while developing the actual functionality of
the website: you would know where is it that you are heading. This
should not take you more than a few hours to do, you basically need to
visualize what should already be in the proposal.
(b) Pick a framework to use and create a super simple hello-port
application. In case you pick Django, go to
https://docs.djangoproject.com/en/2.0/intro/tutorial01/
or

https://devcenter.heroku.com/articles/getting-started-with-python#introduction
or any other tutorial of your choice, install the framework and the
database, follow the steps towards creating a simple app and modify
the app in such a way that it will implement *any* super simple
functionality from the final application. You could for example import
three ports, create a listing of those three ports with symlink to the
port page which will do nothing else but print port name and
description. You may ask for help in case you are struggling with
installation of dependencies.
(c - fully optional) If you still have time and motivation, try to
make the page from (a) look at nice with basic style (you could use
any existing framework for fontend development or ask a friend to help
you with this particular one :)

On 26 March 2018 at 11:39, Vishnu wrote:
>
> Can you help me with where does that json api work from?where is the code
> that sends request to the backend.To get that data.

You mean in the planned website or on our buildbot setup?

For the 

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Also i wanted to know.Like if a new port is created .Buildbot is
compulsorily run right?
and if it is deleted.There is no need for buildbot to run.So buildbot wont
send updates to the log that port has been deleted.
In that case i want to know how / where is the port deleted from.What code ?
Because i would have to do changes there.That as soon as port is deleted
log of that port is updated.

" Buildbot master and this website will live on a different server. How
exactly do you plan to update the log file? "
In that case. I would need to make a mediator(present on same server as
that of website) of some sort. Wherein requests would be sent to that
mediator. And that mediator would inturn write in the log files.

or i think that would be hectic.
It would be best to have the logs on the Buildbot master side.Because every
time buildbot is used it can easily update the log at that time only.No
need for continuously sending stuff to mediator as i previously planned
above.
As at max the logs(history) required by the web app is 20 logs.It can make
a request to the backend to get them.

I think this would be better.

Also is backend and your website lives on different servers ?
How does the interaction happen ?

On 26 March 2018 at 15:09, Vishnu  wrote:

> " I just wanted to say that in the long term it's more important to make
> sure that new builds are stored soon after they finish, but it's
> probably much easier to start with historical data (and it's also
> super useful in the short term), so that you have immediately
> something to work with. "
>
> yes i definitely agree with you.It would be best to first work upon
> history.That will allow me to understand the code more.
>
> Can you help me with where does that json api work from?where is the code
> that sends request to the backend.To get that data.
> Once i get to know that it will simplify my job to send request to backend
> for port
>
> Also is there any existing method to show all ports of a maintainer?
> or to show all ports of a category?say python?
>
>
> It would be best if i start working gmy gsoc with 3 most important things .
> 1) Basic Port information through portindex2json
> 2) Port Build history. It would use JSON API
> 3) Buildbot code updation to send updates to the logs.
>
>
> And as you said logs would be ineffective to store .For history of builds
> of the port.I think logfile would be the bestway to store history.
> And regarding the syntax . I was thinking of storing the build history in
> json format in one line in log.txt of a particular port.
>
> Also mojca can i know your working hours.or atleast the timezone where you
> live.
> I will try to do most of the work on proposal during that time.Because
> only one day is left and a lot to change.
>
> Thanks
> Vishnu
>
>
>
>
>
> On 26 March 2018 at 05:46, Mojca Miklavec  wrote:
>
>> Dear Vishnu,
>>
>> On Mon, Mar 26, 2018 at 2:58 AM, Vishnu wrote:
>> >
>> > Hi,
>> >
>> > I wanted some help in build bot summary.
>> > I am not sure i completely understand what's happening here.
>> > Could you please explain me about buildbot summary more?
>> >
>> > what i understand is :
>> > There can be n number of builds for each port.
>> > So how am i supposed to get the information of what all builds have
>> > already been done for that particular port.Where can i find that history
>> > from? from build.macports.org ?
>>
>> Yes, that information can be obtained from build.macports.org.
>>
>> > First and foremost I think first i have to implement the Buildbot idea
>> > .That would create logs.
>> > But for that idea..how to get all the exiting build history of
>> ports?Build
>> > history such as timestamp,os,Succesful or not?
>> > Where to get exclusive information about a port.
>>
>> You cannot get that at the moment.
>> That's what the whole idea is about: to implement the functionality
>> that will let developers query a single port.
>>
>> Take a look at
>> https://build.macports.org/builders/ports-10.13_x86_64-builder
>>
>> You'll see a list of builds. The one currently being built is
>> https://build.macports.org/builders/ports-10.13_x86_64-build
>> er/builds/21976
>> for the port "sumo".
>>
>> The task is then to go back to the history by decreasing that number
>> (21976) one by one and fetching information from each individual build
>> until you reach number 1 or the first build where the information is
>> no longer available. I believe the build logs get deleted after cca.
>> 10.000 builds and we have 9 builders, so you would end up querying
>> roughly 100k entries. That could take a couple of days and might need
>> some coordination with Ryan, but you can start with as little as two
>> entries per builder.
>>
>> This information can be queried via JSON interface
>> (https://build.macports.org/json/help). You can infer OS version from
>> the build slave name. Timestamp, port name, whether the build was
>> successful etc. are all available in 

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 25 March 2018 at 11:29, Vishnu wrote:
> Hi Mojca,
>
> I have submitted a draft proposal through GSOC portal.
> Please go through it and do review and suggest changes.

I'll try to provide a bit more feedback later, but just some quick points.

1.) Stretch goals are not asking you for your career goals, but rather:

Let's assume that you finish your work 3 weeks earlier; what will you
do next? Assume that you start mentoring another student next year:
what's your vision about the project, what task should/could be
implemented after the GSOC, or during next GSOC? How do you envision
integration/interplay with other parts of MacPorts website?

2.) There is no mention about the technologies/languages/frameworks
planned for use. You should be more specific about how you plan to
implement stuff. My suggestion would be python (maybe django), but
there are also plenty of other technical decisions to be made.

3.) Log files are not going to cut it. You should draft/propose the
database schema to be used in the project. (You can provide graphs or
plain text or whatever, but something that demonstrates which objects
you would store, which properties and what would be the relations
between objects.)

4.) How and where would you deploy your work in development (before
it's finished and deployed on our server)?

5.) For the timeline: you mention, for example "coding to gather port
information". But the timeline doesn't say when the website could be
deployed (somewhere temporary) to actually display all that data. What
will you use for frontend?

6.) You have scheduled installation statistics for the end. This means
that you would hardly be able to have time to deplay anything, let
alone test and fix things. Things like links to trac tickets,
livechecks etc. are less important and can be added at any given time,
but you will want to start collecting real data from users early in
the coding period, even if display will still be suboptimal, so that
you get an impression of where things / design needs to be improved
etc. The existing statistics page has some issues and since none of us
is fluent in rails to fix those issues, the project never took off due
to some design issues (not that hard to fix, but nobody did that) and
lack of advertising. You should aim for deployment of some kind as
early as possible, and to have the most important features finished by
the second evaluation, so that you can get feedback and have
sufficient time for bugfixing, documuntation etc.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Can someone from MacPorts help me with my proposal?
There's not much time left.
I really need some suggestion/improvements in the proposal.

On 25 March 2018 at 14:59, Vishnu  wrote:

> Hi Mojca,
>
> I have submitted a draft proposal through GSOC portal.
> Please go through it and do review and suggest changes.
>
> Thanks
> Vishnu
>
> On 25 March 2018 at 12:59, Mojca Miklavec  wrote:
>
>> On 25 March 2018 at 00:42, Vishnu wrote:
>> > If i take up the project of improving the buildbot or the website.
>> > would it be considered a different project? Or we both still have to
>> > compete?
>>
>> Both :)
>>
>> > The other student could work on Option B and i could work on Option A
>> > Because those also sounded interesting.
>>
>> First of all, any projects will be first and foremost evaluated to see
>> whether the quality of proposal & and the "proof of competence" of the
>> student is above the threshold. As Google said, "only OK" proposals
>> should usually not be accepted, they need to be well above average. In
>> addition to that, even if we get two excellent proposals we may only
>> get one slot and then students will "compete" for that single slot no
>> matter whether the projects are different or not.
>>
>> But if more than one proposal for the same project meets the quality
>> standard and if we had enough slots to cover both, it's a complete
>> waste of resources if two people start coding exactly the same thing
>> and then we need to throw one of the two works away after the end of
>> the summer.
>>
>> If one works on the buildbot side and the other one on the website,
>> that's OK because the outcome would be a different code serving a
>> different purpose (even if we have an excellent website with build
>> statistics, we still hesitate to switch to buildbot 1.x just because
>> nobody in the team is currently willing to invest time into learning
>> some new javascript frameworks and figure out how to implement the new
>> views that we would need; and the code could also be useful as a demo
>> to other groups who hesitate to switch to 1.1).
>>
>> If you want to know more about the buildbot idea, you should ask Piere
>> (CC-ed in my initial reply) and CC this list.
>>
>> To answer the rest of you question:
>>
>> I don't mind if one document describes two potential project, but
>> describing one project well is already time consuming (and keep in
>> mind that you are running out of time). If you end up writing two bad
>> proposals instead of one excellent, you are doing yourself a
>> disservice. (Writing one excellent project and drafting an alternative
>> should be ok.)
>>
>> In any case don't wait for the last day to submit your draft because
>> you won't be able to get any feedback and even if you do get feedback,
>> you won't have time to improve it later.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi Mojca,

I have submitted a draft proposal through GSOC portal.
Please go through it and do review and suggest changes.

Thanks
Vishnu

On 25 March 2018 at 12:59, Mojca Miklavec  wrote:

> On 25 March 2018 at 00:42, Vishnu wrote:
> > If i take up the project of improving the buildbot or the website.
> > would it be considered a different project? Or we both still have to
> > compete?
>
> Both :)
>
> > The other student could work on Option B and i could work on Option A
> > Because those also sounded interesting.
>
> First of all, any projects will be first and foremost evaluated to see
> whether the quality of proposal & and the "proof of competence" of the
> student is above the threshold. As Google said, "only OK" proposals
> should usually not be accepted, they need to be well above average. In
> addition to that, even if we get two excellent proposals we may only
> get one slot and then students will "compete" for that single slot no
> matter whether the projects are different or not.
>
> But if more than one proposal for the same project meets the quality
> standard and if we had enough slots to cover both, it's a complete
> waste of resources if two people start coding exactly the same thing
> and then we need to throw one of the two works away after the end of
> the summer.
>
> If one works on the buildbot side and the other one on the website,
> that's OK because the outcome would be a different code serving a
> different purpose (even if we have an excellent website with build
> statistics, we still hesitate to switch to buildbot 1.x just because
> nobody in the team is currently willing to invest time into learning
> some new javascript frameworks and figure out how to implement the new
> views that we would need; and the code could also be useful as a demo
> to other groups who hesitate to switch to 1.1).
>
> If you want to know more about the buildbot idea, you should ask Piere
> (CC-ed in my initial reply) and CC this list.
>
> To answer the rest of you question:
>
> I don't mind if one document describes two potential project, but
> describing one project well is already time consuming (and keep in
> mind that you are running out of time). If you end up writing two bad
> proposals instead of one excellent, you are doing yourself a
> disservice. (Writing one excellent project and drafting an alternative
> should be ok.)
>
> In any case don't wait for the last day to submit your draft because
> you won't be able to get any feedback and even if you do get feedback,
> you won't have time to improve it later.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 25 March 2018 at 00:42, Vishnu wrote:
> If i take up the project of improving the buildbot or the website.
> would it be considered a different project? Or we both still have to
> compete?

Both :)

> The other student could work on Option B and i could work on Option A
> Because those also sounded interesting.

First of all, any projects will be first and foremost evaluated to see
whether the quality of proposal & and the "proof of competence" of the
student is above the threshold. As Google said, "only OK" proposals
should usually not be accepted, they need to be well above average. In
addition to that, even if we get two excellent proposals we may only
get one slot and then students will "compete" for that single slot no
matter whether the projects are different or not.

But if more than one proposal for the same project meets the quality
standard and if we had enough slots to cover both, it's a complete
waste of resources if two people start coding exactly the same thing
and then we need to throw one of the two works away after the end of
the summer.

If one works on the buildbot side and the other one on the website,
that's OK because the outcome would be a different code serving a
different purpose (even if we have an excellent website with build
statistics, we still hesitate to switch to buildbot 1.x just because
nobody in the team is currently willing to invest time into learning
some new javascript frameworks and figure out how to implement the new
views that we would need; and the code could also be useful as a demo
to other groups who hesitate to switch to 1.1).

If you want to know more about the buildbot idea, you should ask Piere
(CC-ed in my initial reply) and CC this list.

To answer the rest of you question:

I don't mind if one document describes two potential project, but
describing one project well is already time consuming (and keep in
mind that you are running out of time). If you end up writing two bad
proposals instead of one excellent, you are doing yourself a
disservice. (Writing one excellent project and drafting an alternative
should be ok.)

In any case don't wait for the last day to submit your draft because
you won't be able to get any feedback and even if you do get feedback,
you won't have time to improve it later.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Hi,

Please keep any technical discussion (that doesn't include
private/sensitive data) on the mailing list.

On 25 March 2018 at 00:42, Vishnu wrote:
>
> And regarding the individual port page.
> I was thinking whenever someone opens the port page.The content is generated
> at that time and displayed.

True.

> So everytime port page is opened my scraper would scrape the build summary.

No way. You would jam both your website and the buildbot that way.
The build logs get deleted after a while (I think after 10k builds and
a single revbump of perl version in a single commit might trigger more
than 1000 builds), but more importantly you have absolutely no idea
where to look for the builds (unless the buildbot idea gets
implemented) which is why we put that idea on the list in the first
place.

Also, you cannot consult each user who opted in to participate at
statistics collection to tell you whether they have a particular port
installed each time when some user clicks on the website ...

> Or are you saying to create such a framework that will scrape all the ports
> only once(in forever) and create a port log?

Before it gets deployed it needs to collect information about all
ports (once), but then it needs to keep monitoring everything, after
each commit and after each build, and update the data accordingly.

> And whenever a new build is done Base code would add to the website log of
> the port .?
>
> Also how to know about binary package status?

See an example where a package has been uploaded:

https://build.macports.org/builders/ports-10.11_x86_64-builder/builds/51073/steps/gather-archives/logs/stdio
and an example where there were "issue":

https://build.macports.org/builders/ports-10.9_x86_64-builder/builds/51949/steps/gather-archives/logs/stdio
You should of course also consult a page like
http://packages.macports.org/python27/
and make sure that both sources match. For old files on
packages.macports.org you won't have the logs, but it's helpful to
keep the information on the website. The most important piece of
information is when an old version built and the new one no longer
does, or only builds on a different os version etc.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Hi,

Just so that you know: there is now two students inquiring about the
same project.
We cannot know in advance how good the proposals will be, but we
certainly cannot accept two students for the same project.

(That doesn't mean that you should grab a different project, but you
might or might not want to keep this in mind when writing the
proposals in case the other idea(s) sound reasonable. So far there was
no interest yet in improving the buildbot part or perhaps other parts
of the website.)

In any case do not forget to demonstrate your skillset with samples
and/or prototypes. I would suggest to visually convince us with some
nice-looking (may be static) single page as well as provide some kind
of functional prototype along with references to any projects you did
before.

I also added two additional "availability" questions to the template at
https://trac.macports.org/wiki/SummerOfCodeApplicationTemplate

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Thanks Mojca for your prompt reply.
1) description, version,maintainer, homepage, variants, dependencies,dependent
ports,etc
So for information of each port, my website has to scrape data from such
pages :

https://github.com/macports/macports-ports/blob/master/python/py-amqplib/Portfile

is this correct?

2) Build Summary:

Would scraping buildbot history pose a problem?

and "  modify the buildbot setup to submit data to the server. "
isn't it aldready submitting data to the server?
if not how is the buildbot history currently generated?


On 25 March 2018 at 04:01, Mojca Miklavec  wrote:

> On 24 March 2018 at 22:50, Vishnu wrote:
> > Well that's really nice.
> > I'll be emailing you the proposal very soon.
> > I would really like to work on creating dynamic websites for each port
> with
> > one framework.
> > I think that is what you are also working on currently.Or is it something
> > else?(Umesh Sir)
> >
> > I would like to go with option B.
> >
> > What i understand is .Basically, i will be creating a dynamic
> > website/framework (use some scripting language) to generate/gather all
> the
> > following information for any port that is requested.
> >
> >
> > - all the information about the port (description, version,
> > maintainer, homepage, variants, dependencies, dependent ports, ...)
> > - build summary (as scraped from buildbot history): which version was
> > built at what time on which os, successful or not, ...
> >   - including whether or not a binary package exists
> > - installation statistics (from users who opted in and send a json
> > file every now and then) with some graphs, similar to the stats page
> > above
> > - results of "livecheck": whether there's a newer version of the port
> > available
> > - short git log with links to the last few changes to the Portfile
> > - links to trac tickets for that port
> >
> > https://trac.macports.org/query?0_port=python27&0_port_
> mode=%7E&0_status=%21closed
> > - (maybe links to pull requests for that port)
> >
> > Is my understanding of the project correct?
> >
> > Now to generate:
> > 1) description, version,maintainer, homepage, variants, dependencies,
> > dependent ports,etc:
> > I will need to update this script
> > https://github.com/g5pw/macports-port-tree made by Aljaz for the
> remaining
> > missing information.because not all information is produced by this tool.
>
> This source of information PortIndex which can be generated from
> https://github.com/macports/macports-ports (that's part of MacPorts
> installation process, it should be possible to get it under linux as
> well, but I didn't try, we also mirror it).
>
> PortIndex is some data structure in Tcl which can be converted into
> JSON with an existing script postindex2json, but ideally the Tcl
> script should be fixed to generate data in a better format.
> This data would then need to be imported at the beginning, but the
> application should keep accepting updates about new or deleted ports.
>
> > 2) Build Summary: This can be generated by scraping through buildbot
> > history.
>
> For historic data. For the future it would be better to modify the
> buildbot setup to submit data to the server.
>
> > 3) Installation Statistics: Take help from Stats web app(Gsoc 2011
> project)
> >
> > 4)Results of livecheck: I am not sure how to get that. i'll be needing
> help
> > to how to get this.
>
> Sample output from the package manager:
>
> > port -v livecheck *
> Bear seems to be up to date
> CImg seems to have been updated (port version: 1.7.9, new version: 2.2.1)
> CoreOSMakefiles seems to be up to date
> Error: cannot check if Djinn was updated (regex didn't match)
> Drizzle seems to have been updated (port version: 30, new version: 31)
> FlowUI seems to be up to date
> GASNet seems to have been updated (port version: 1.26.3, new version:
> 1.30.0)
> GitX seems to be up to date
> HaXml seems to have been updated (port version: 1.20.2, new version:
> 1.23.3)
> Error: livecheck failed for LibVNCServer: extracted version '0.4' is
> older than livecheck.version '0.9.9'
> LucenePlusPlus seems to be up to date
> Error: livecheck failed for MongoObjCDriver: extracted version 'test2'
> is older than livecheck.version '1.0.6'
> Error: cannot check if PlasmaShop was updated (regex didn't match)
> ResIL seems to have been updated (port version: 1.7.9, new version: 1.8.3)
> Error: cannot check if SilverCity was updated (regex didn't match)
> SourceKitten seems to have been updated (port version: 0.17.6, new
> version: 0.19.1)
> Error: cannot check if VillainousStyle was updated (regex didn't match)
> aap seems to be up to date
>
> > 5) Link to track tickets : Can be done using scripting,html.
> > 6)Pull request : Can be done.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 24 March 2018 at 22:50, Vishnu wrote:
> Well that's really nice.
> I'll be emailing you the proposal very soon.
> I would really like to work on creating dynamic websites for each port with
> one framework.
> I think that is what you are also working on currently.Or is it something
> else?(Umesh Sir)
>
> I would like to go with option B.
>
> What i understand is .Basically, i will be creating a dynamic
> website/framework (use some scripting language) to generate/gather all the
> following information for any port that is requested.
>
>
> - all the information about the port (description, version,
> maintainer, homepage, variants, dependencies, dependent ports, ...)
> - build summary (as scraped from buildbot history): which version was
> built at what time on which os, successful or not, ...
>   - including whether or not a binary package exists
> - installation statistics (from users who opted in and send a json
> file every now and then) with some graphs, similar to the stats page
> above
> - results of "livecheck": whether there's a newer version of the port
> available
> - short git log with links to the last few changes to the Portfile
> - links to trac tickets for that port
>
> https://trac.macports.org/query?0_port=python27&0_port_mode=%7E&0_status=%21closed
> - (maybe links to pull requests for that port)
>
> Is my understanding of the project correct?
>
> Now to generate:
> 1) description, version,maintainer, homepage, variants, dependencies,
> dependent ports,etc:
> I will need to update this script
> https://github.com/g5pw/macports-port-tree made by Aljaz for the remaining
> missing information.because not all information is produced by this tool.

This source of information PortIndex which can be generated from
https://github.com/macports/macports-ports (that's part of MacPorts
installation process, it should be possible to get it under linux as
well, but I didn't try, we also mirror it).

PortIndex is some data structure in Tcl which can be converted into
JSON with an existing script postindex2json, but ideally the Tcl
script should be fixed to generate data in a better format.
This data would then need to be imported at the beginning, but the
application should keep accepting updates about new or deleted ports.

> 2) Build Summary: This can be generated by scraping through buildbot
> history.

For historic data. For the future it would be better to modify the
buildbot setup to submit data to the server.

> 3) Installation Statistics: Take help from Stats web app(Gsoc 2011 project)
>
> 4)Results of livecheck: I am not sure how to get that. i'll be needing help
> to how to get this.

Sample output from the package manager:

> port -v livecheck *
Bear seems to be up to date
CImg seems to have been updated (port version: 1.7.9, new version: 2.2.1)
CoreOSMakefiles seems to be up to date
Error: cannot check if Djinn was updated (regex didn't match)
Drizzle seems to have been updated (port version: 30, new version: 31)
FlowUI seems to be up to date
GASNet seems to have been updated (port version: 1.26.3, new version: 1.30.0)
GitX seems to be up to date
HaXml seems to have been updated (port version: 1.20.2, new version: 1.23.3)
Error: livecheck failed for LibVNCServer: extracted version '0.4' is
older than livecheck.version '0.9.9'
LucenePlusPlus seems to be up to date
Error: livecheck failed for MongoObjCDriver: extracted version 'test2'
is older than livecheck.version '1.0.6'
Error: cannot check if PlasmaShop was updated (regex didn't match)
ResIL seems to have been updated (port version: 1.7.9, new version: 1.8.3)
Error: cannot check if SilverCity was updated (regex didn't match)
SourceKitten seems to have been updated (port version: 0.17.6, new
version: 0.19.1)
Error: cannot check if VillainousStyle was updated (regex didn't match)
aap seems to be up to date

> 5) Link to track tickets : Can be done using scripting,html.
> 6)Pull request : Can be done.

Mojca

Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello all,

Thanks for rectifying the mail issues.

This is Vishnu.
I would Like to participate in GSOC 18 .And would like to work on :
#Collect build statistics
I needed some help in understanding the idea a bit more.
i wanted to understand the idea more deeply.
Can someone elaborate the idea more?

How many web pages do we need to make ?
Can i get some sample structure of webpage for 1 port . what all need to be
there.
Also where to get : collect per-port statistics & success matrix .Can i get
the exact sample link?




I really think this is the project i could work on because i have prior
experience with all the languages required for this project (HTML, Python,
javascript, in fact, did my last intern mainly involving JSON).
I also have good experience in web Scraping.

I am well aware about :
JSON
Html
Css
Python
JAVASCRIPT
Basic idea about perl

Have used macOS.
Access to macos during gsoc would be limited. But possible.
I have certain knowledge of macports too.


Thanks
Vishnu