Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Yes I was laos thinking about it for some time.

Please someone from infra team create another repository.
There I will maintain code in a proper way by Using Pull Requests.So that
code gets merged only if clean.


Regarding your general remarks.
I will look into them.
Please make the repo as soon as possible.


Things I will be doing:

- Limit testing.json to 1000 ports only.
- Implement Categories table and the modified port table.
- Implement some sort of test atleast very trivial but will
- Did not index the port table so far.But will now index the table.



Things discussed in the last meeting:

Implement these as well
- Implement ability to mark deleted ports in the port table itself
- For every port just display the last 5 - 10 commits on the ports
- Just mention the version everywhere. No portversion id.
- When build started and ended.

Future Goals:
- implment obsolete
- Fix portindex2json, atleast maintainers
- Differential feature to be in stretch goals
- Port History
- Port Version table

Also I think finalising the database ticket will be for long.As we are
continusally evolving it so... Probably the milestone to create Database
structure within the week may not be possible.

Mojca you said something about static files.
I think we should have all Javascript also in a separate individual file.
That's why in a seperate individual file.


Thanks

On 15 May 2018 at 02:53, Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> I just wanted to say that we *really really really* need to fix the
> repository since it's unacceptably large (some commits checked out are
> 240 MB, the size to transfer the contents is 133 MB).
>
> No matter what you think of it regarding breaking links etc., we
> definitely won't be hosting your linux bootloading images in our repo.
> This is one of the reasons where ... pull request make a huge
> difference over simple commits because such errors would be caught
> early on.
>
> Other general remarks:
> - please delete database_design.txt now that you have the markdown document
> - you should not commit __pycache__ folders
> - in dropdown.css you are mixing tabs and spaces; pick one and use it
> consistently, don't mix the two (and if you use spaces, use 2 or 4,
> not 9; identing is totally lacking)
> - by asking you to move code to static I did not mean that you should
> put data points for a graph to static files.
> - do not put 20 MB json files with sample data to the repository
>
> My suggestion would in fact be to start with an empty repository again
> and ask you to create your first pull request with cleaned up data.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Also can i know the status of your getting access to your VM ?

Thanks

On 13 May 2018 at 13:07, Vishnu <vishnum1...@gmail.com> wrote:

> Hey
>
> Mojca you said once that you have lot to comment on the current database.
> So it would be great if you could do that.
> As today is the last day of Community bonding period.
> From tommorrow coding starts.
>
> Thanks
>
> On 12 May 2018 at 23:14, Mojca Miklavec <mo...@macports.org> wrote:
>
>> Dear Vishnu,
>>
>> Sorry for being slightly out-of-sync (sending emails I wrote quite a
>> while ago and in the meantime no longer being up-to-date with other
>> responses).
>>
>> On 12 May 2018 at 17:26, Vishnu wrote:
>> > Hi
>> >
>> > "The PortIndex file itself is generated with the 'portindex' command,
>> > which only updates the data for Portfiles that have a modification time
>> > that is newer than the existing PortIndex file."
>> >
>> > So where can i see that code that checks the modification time?
>>
>> https://github.com/macports/macports-base/blob/master/src/po
>> rt/portindex.tcl
>>
>> > So suppose
>> > Our ports tree contains three ports A, B, C last modified time : 1:00
>> UTC
>> > present port index:
>> > [ A
>> >   B
>> >   C
>> > ] @ 1:00 UTC
>> >
>> > There was a edit in Port A @ 1:30 UTC
>> > so something will check all the port files modification time and if
>> > relatively newer it will update the port index.
>> >
>> > so new  port index:
>> > [ A
>> >   B
>> >   C
>> > ] @ 1:30 UTC
>> >
>> >
>> > "The portindex2postgres.sql script merely converts the PortIndex file
>> > from the custom format based on Tcl lists to SQL. The output will always
>> > contain SQL statements with the full data for every port."
>> >
>> > Then after the port index is updated this script runs and flushes the db
>> > then fills it with new portindex data?
>> > or just updates the db with the modified port data?
>>
>> I don't know the answer without looking at it first (you can check
>> what the code does yourself), but from the perspective of your app
>> development this is irrelevant.
>>
>> > So after every commit the entire 15mb portindex being processed and data
>> > being uploaded to the db.
>> > I think this is very inefficient.
>> > rather it would be best if after every commit the buildbot/ webhook
>> updates
>> > the database itself.
>>
>> I totally agree that it might make a lot more sense to submit just
>> differential data, but my suggestion would be to first implement the
>> functionality and optimise it at the end if there will be time. No
>> user will suffer from suboptimal data transfer (even if that database
>> only gets updated once per day, it will still be perfectly
>> acceptable), while nobody will benefit from spending a lot of time
>> figuring out how to properly implement a super efficient solution and
>> then running out of time to actually finish the app.
>>
>> Let's just work with what we have for now.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Mojca you said once that you have lot to comment on the current database.
So it would be great if you could do that.
As today is the last day of Community bonding period.
>From tommorrow coding starts.

Thanks

On 12 May 2018 at 23:14, Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> Sorry for being slightly out-of-sync (sending emails I wrote quite a
> while ago and in the meantime no longer being up-to-date with other
> responses).
>
> On 12 May 2018 at 17:26, Vishnu wrote:
> > Hi
> >
> > "The PortIndex file itself is generated with the 'portindex' command,
> > which only updates the data for Portfiles that have a modification time
> > that is newer than the existing PortIndex file."
> >
> > So where can i see that code that checks the modification time?
>
> https://github.com/macports/macports-base/blob/master/src/
> port/portindex.tcl
>
> > So suppose
> > Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
> > present port index:
> > [ A
> >   B
> >   C
> > ] @ 1:00 UTC
> >
> > There was a edit in Port A @ 1:30 UTC
> > so something will check all the port files modification time and if
> > relatively newer it will update the port index.
> >
> > so new  port index:
> > [ A
> >   B
> >   C
> > ] @ 1:30 UTC
> >
> >
> > "The portindex2postgres.sql script merely converts the PortIndex file
> > from the custom format based on Tcl lists to SQL. The output will always
> > contain SQL statements with the full data for every port."
> >
> > Then after the port index is updated this script runs and flushes the db
> > then fills it with new portindex data?
> > or just updates the db with the modified port data?
>
> I don't know the answer without looking at it first (you can check
> what the code does yourself), but from the perspective of your app
> development this is irrelevant.
>
> > So after every commit the entire 15mb portindex being processed and data
> > being uploaded to the db.
> > I think this is very inefficient.
> > rather it would be best if after every commit the buildbot/ webhook
> updates
> > the database itself.
>
> I totally agree that it might make a lot more sense to submit just
> differential data, but my suggestion would be to first implement the
> functionality and optimise it at the end if there will be time. No
> user will suffer from suboptimal data transfer (even if that database
> only gets updated once per day, it will still be perfectly
> acceptable), while nobody will benefit from spending a lot of time
> figuring out how to properly implement a super efficient solution and
> then running out of time to actually finish the app.
>
> Let's just work with what we have for now.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

"The PortIndex file itself is generated with the 'portindex' command,
which only updates the data for Portfiles that have a modification time
that is newer than the existing PortIndex file."

So where can i see that code that checks the modification time?
So suppose
Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
present port index:
[ A
  B
  C
] @ 1:00 UTC

There was a edit in Port A @ 1:30 UTC
so something will check all the port files modification time and if
relatively newer it will update the port index.

so new  port index:
[ A
  B
  C
] @ 1:30 UTC


"The portindex2postgres.sql script merely converts the PortIndex file
from the custom format based on Tcl lists to SQL. The output will always
contain SQL statements with the full data for every port."

Then after the port index is updated this script runs and flushes the db
then fills it with new portindex data?
or just updates the db with the modified port data?

Which one occurs?

Trying to understand the present mechanism.


So after every commit the entire 15mb portindex being processed and data
being uploaded to the db.
I think this is very inefficient.
rather it would be best if after every commit the buildbot/ webhook updates
the database itself.


Thanks

On 12 May 2018 at 17:47, Rainer Müller <rai...@macports.org> wrote:

> On 2018-05-12 10:34, Vishnu wrote:
> > I am not saying that my db and the exiting Db would be interdependent.
> > Rather i am saying just once in forever I can copy the content to my
> > database.
> > Then do the code to keep on updating it whenever something is added or
> > deleted or modified.
> >
> > Like for exaple maintainers.
> > You already have a table of mainatainers and their ports.So i can copy
> > the data to my database.
> > Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
> > updating my database independent of the existing database.
>
> Our database contains exactly what the portindex2postgres.tcl script
> produces (note the different naming). You can just generate the data
> locally by running it against PortIndex.
>
> You need to have the ports tree in a directory named "ports", then run
> the following command from the parent directory:
>
>   port-tclsh /path/to/macports-infra/jobs/portindex2postgres.tcl
>
> This will create a PortIndex.sql in the current directory. Ignore or
> remove the macports.conf and sources.conf files that are also created.
>
> > Again and again processing the 15mb json file would be bothersome.
> > We definitely need some way of getting a differential json.
> > To update only the changes happened because of commit to the port file.
> >
> > Or maybe we could make some changes in buildbot that.Whenever there is
> > some change in portfile it could update my port table as well with the
> > changes.
> > Not sure how to do this.But just an idea as of now.
>
> Yes, the idea would be that macports-webapp is updated after every commit.
>
> To achieve that, you could use a job in buildbot, but then the result
> needs to be transferred to the macports-webapp. Alternatively, you could
> deliver a WebHook from GitHub to do this within macports-webapp. It
> largely depends on what kind of data you want to gather and how.
> > And regarding working on a seperate branch.Rainer suggested not
> > to.Because daily pull requests have to merged by the mentor.
>
> I did not argue against working on a separate branch. I just wanted to
> point out that pull request reviews on GitHub might pose a problem as
> they cannot depend on each other. It is up to you and your mentors to
> work out the details.
>
> > One doubt I still have is that whether  portindex2postgres.tcl again and
> > again runs the entire MacPorts port tree on the database ?
> > Or just the port that has been changed ?
>
> The PortIndex file itself is generated with the 'portindex' command,
> which only updates the data for Portfiles that have a modification time
> that is newer than the existing PortIndex file.
>
> The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port.
>
> Rainer
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I am not saying that my db and the exiting Db would be interdependent.
Rather i am saying just once in forever I can copy the content to my
database.
Then do the code to keep on updating it whenever something is added or
deleted or modified.

Like for exaple maintainers.
You already have a table of mainatainers and their ports.So i can copy the
data to my database.
Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
updating my database independent of the existing database.

Again and again processing the 15mb json file would be bothersome.
We definitely need some way of getting a differential json.
To update only the changes happened because of commit to the port file.

Or maybe we could make some changes in buildbot that.Whenever there is some
change in portfile it could update my port table as well with the changes.
Not sure how to do this.But just an idea as of now.

"You need a separate glue table to connect maintainers with ports. But
that's already part of your database design. You should not store
maintainers in you port table, but perform correct joins (or maybe
create a view first?) instead."

You misunderstood me, Copying the data.I meant once in a lifetime.
Then keep on updating the table with whatever changes happens.
And i will definitely perform join or views.

I will be doing following things :
- Update Ticket descriptions
- Put the gsoc proposal in docs
- update the graph on the static website
- try to handle sensitive keys


Regarding heroku.
Don't worry about that. I got it figured out.

And regarding working on a seperate branch.Rainer suggested not to.Because
daily pull requests have to merged by the mentor.

Would directly commiting to master be a problem ?
As only i will be dealing with the repo.
Probably no one will right now clone the repository.

It is also very easy to directly commit to master than creating a PR.



One doubt I still have is that whether  portindex2postgres.tcl again and
again runs the entire MacPorts port tree on the database ?
Or just the port that has been changed ?


Thanks

On 11 May 2018 at 12:47, Mojca Miklavec <mo...@macports.org> wrote:

> On 11 May 2018 at 04:00, Vishnu wrote:
> > Hi
> >
> > Also i wanted to know
> > For this link
> > https://www.macports.org/ports.php?by=name=python27
> >
> > There is already existing database with loads of information.
> > Is it not updated?
> > is  it static ?
>
> It is not static, the database is updated via an external script from
> PortIndex and the page is rendered with php code, reading information
> from the database.
>
> > If it is updated automatically...Then half of the work for database would
> > already be over.
> >
> > Could i get access to the DB?
>
> Once your application reaches a certain quality & completeness, it
> should render the above page obsolete and the above page will most
> likely be removed.
>
> While we could theoretically give you access to the database (not
> without opening some security holes): first of all this hardly makes
> any sense because you should always be able to reproduce the exact
> state of the database if you run PortIndex2PGSQL.tcl yourself, but
> most importantly we don't want to introduce a complex network of
> interdependencies.
>
> Let's say that I need to change the new app a few months from now.
> Before I can deploy the change I need to test it locally on my
> machine. If you app will require password-protected or ip-limited
> access to some third-party database, this would greatly hinder my
> ability to patch the new app. The app should be as standalone as
> possible. The next very important aspect is that you might want to
> include more information than what's currently available in that
> database. For example, you might decide to include the information
> about which port supports which version of macOS. That information is
> not available in the existing database, so you would then first need
> to patch who-knows-what-other-code in random mixture of tcl, php,
> bash, ... And you would not be able to do table joins etc. It's
> absolutely bad idea to depend on third-party database. You should of
> course collect information from various sources, but you need to keep
> all the required data in a single database somewhere close to your
> app.
>
> Some time ago you had ideas about updating portindex just once per
> year? Well, here's an example why you should ideally do it after each
> commit rather than once per year. That said, we should ideally find a
> way to optimise the PortIndex2json part (probably not your task).
> PortIndex already updates just those ports that have in fact been
> changed, but conversion to json updates the complete file and needs to
> feed the web app with full 15 MB of data after each commit. T

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Also i wanted to know
For this link
https://www.macports.org/ports.php?by=name=python27

There is already existing database with loads of information.
Is it not updated?
is  it static ?

If it is updated automatically...Then half of the work for database would
already be over.

Could i get access to the DB ?
I got the rough idea of the db from here:
https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249

So is this updated regularly?How?

And there seems to be lot of data already existing. Which i could just copy.
Existing maintainer table,many other.

Thanks


On 11 May 2018 at 07:11, Vishnu <vishnum1...@gmail.com> wrote:

> Hi
>
> So should I start working on the same database?
>
> Community does not have any suggestions I guess.
> So should i go ahead with the existing structure?
>
> I had a doubt regarding version.
>
> How can i check the existing versions of any port.
> Say python27?
>
> Any list of maintainers i can get?
>
> Thanks
>
> On 9 May 2018 at 04:06, Vishnu <vishnum1...@gmail.com> wrote:
>
>> Hi
>>
>> And also i couldn't figure out any way to hide passwords/ Sensitive
>> information while creating app.
>>
>> Thanks
>>
>> On 9 May 2018 at 03:43, Vishnu <vishnum1...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I had one doubt.
>>> Should i switch the link in heroku account for integration with macports
>>> github ?
>>>
>>> Because i think then you need to give accesss to heroku of your account.
>>>
>>> I think it would be wise for me to do the commit update in my local
>>> repository itself..
>>>
>>> Once every 2 weeks or something ill push all the changes to macports
>>> repository.
>>> Do comment .What should be done?
>>>
>>> Thanks
>>>
>>> On 8 May 2018 at 11:40, Jackson Isaac <ijack...@macports.org> wrote:
>>>
>>>> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec <mo...@macports.org>
>>>> wrote:
>>>> >
>>>> > On 8 May 2018 at 01:12, Vishnu wrote:
>>>> > > Hi
>>>> > >
>>>> > > I will work on all the points mentioned today..
>>>> > >
>>>> > > But i have been trying many different ways.
>>>> > > To get 20k ports onto the database.. The site is crashing.. Timeout
>>>> error.
>>>> > > Or the site cant be reached.
>>>> > > Maximum i got till 9k ports successfully entered.
>>>> > >
>>>> > > Tried many ways..
>>>> > >
>>>> > > If you have any suggestions do let me know.
>>>> >
>>>>
>>>> We can even try firebase for the DB backend and host the frontend on
>>>> heroku.
>>>>
>>>> Firebase free version gives 1GB of storage and 10GB of bandwidth per
>>>> month.
>>>> I guess for testing out the app on complete portindex we could give it
>>>> a try.
>>>>
>>>> https://firebase.google.com/pricing/
>>>>
>>>> Then again we would have some kind of limitations over here in long run.
>>>>
>>>> >
>>>> > I was pretty sure that we would reach the database limit here.
>>>> >
>>>> >
>>>> > This site
>>>> > https://devcenter.heroku.com/articles/heroku-postgres-plans
>>>> > says that the limit is 10k rows which is not anywhere near enough and
>>>> > fully consistent with your observation. The plan that lets you have
>>>> > 10M rows is 9 USD per month which I gladly pay if that would solve the
>>>> > issues (I'm not sure if those are the only ones, we might not have
>>>> > sufficient memory etc.)
>>>> >
>>>>
>>>> 9USD is only for the DB though. We might need to pay for the compute
>>>> separately.
>>>>
>>>> Looks like free VMs come with only 512MB RAM.
>>>>
>>>> >
>>>> > (I also remembered that I might have access to create a clean virtual
>>>> > machine, but I need a couple of days to physically reach a place where
>>>> > I have the additional info. That would require setting up the whole
>>>> > machine manually of course, but we need to do that for the final
>>>> > deployment anyway.)
>>>> >
>>>> > > I think i have to break the data entry in separate chunks and do
>>>> it. But not
>>>> > > sure how to do that.
>>>> >
>>>> > In any case Heroku will apparently not let you import the full
>>>> > database anyway until we switch to a payed account. If you still have
>>>> > troubles with performance issues afterwards, this also won't help.
>>>> > What you could do is temporarily copy or move a few folders (say,
>>>> > math, science and python) and run portindex just on those. This will
>>>> > give you a smaller number of ports to work with, but still sufficient
>>>> > to figure out what other issues you'll need to deal with (port listing
>>>> > will have to be paginated etc.)
>>>> >
>>>> > It's a good exercise to know what the limitations are. When you play
>>>> > with new features and database design, it's always easier to play
>>>> > with, say 10-100 ports than with full 15 MB portindex file.
>>>> >
>>>>
>>>> This plan sounds good. +1
>>>>
>>>> --
>>>> Jackson Isaac
>>>>
>>>
>>>
>>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

So should I start working on the same database?

Community does not have any suggestions I guess.
So should i go ahead with the existing structure?

I had a doubt regarding version.

How can i check the existing versions of any port.
Say python27?

Any list of maintainers i can get?

Thanks

On 9 May 2018 at 04:06, Vishnu <vishnum1...@gmail.com> wrote:

> Hi
>
> And also i couldn't figure out any way to hide passwords/ Sensitive
> information while creating app.
>
> Thanks
>
> On 9 May 2018 at 03:43, Vishnu <vishnum1...@gmail.com> wrote:
>
>> Hi
>>
>> I had one doubt.
>> Should i switch the link in heroku account for integration with macports
>> github ?
>>
>> Because i think then you need to give accesss to heroku of your account.
>>
>> I think it would be wise for me to do the commit update in my local
>> repository itself..
>>
>> Once every 2 weeks or something ill push all the changes to macports
>> repository.
>> Do comment .What should be done?
>>
>> Thanks
>>
>> On 8 May 2018 at 11:40, Jackson Isaac <ijack...@macports.org> wrote:
>>
>>> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec <mo...@macports.org>
>>> wrote:
>>> >
>>> > On 8 May 2018 at 01:12, Vishnu wrote:
>>> > > Hi
>>> > >
>>> > > I will work on all the points mentioned today..
>>> > >
>>> > > But i have been trying many different ways.
>>> > > To get 20k ports onto the database.. The site is crashing.. Timeout
>>> error.
>>> > > Or the site cant be reached.
>>> > > Maximum i got till 9k ports successfully entered.
>>> > >
>>> > > Tried many ways..
>>> > >
>>> > > If you have any suggestions do let me know.
>>> >
>>>
>>> We can even try firebase for the DB backend and host the frontend on
>>> heroku.
>>>
>>> Firebase free version gives 1GB of storage and 10GB of bandwidth per
>>> month.
>>> I guess for testing out the app on complete portindex we could give it a
>>> try.
>>>
>>> https://firebase.google.com/pricing/
>>>
>>> Then again we would have some kind of limitations over here in long run.
>>>
>>> >
>>> > I was pretty sure that we would reach the database limit here.
>>> >
>>> >
>>> > This site
>>> > https://devcenter.heroku.com/articles/heroku-postgres-plans
>>> > says that the limit is 10k rows which is not anywhere near enough and
>>> > fully consistent with your observation. The plan that lets you have
>>> > 10M rows is 9 USD per month which I gladly pay if that would solve the
>>> > issues (I'm not sure if those are the only ones, we might not have
>>> > sufficient memory etc.)
>>> >
>>>
>>> 9USD is only for the DB though. We might need to pay for the compute
>>> separately.
>>>
>>> Looks like free VMs come with only 512MB RAM.
>>>
>>> >
>>> > (I also remembered that I might have access to create a clean virtual
>>> > machine, but I need a couple of days to physically reach a place where
>>> > I have the additional info. That would require setting up the whole
>>> > machine manually of course, but we need to do that for the final
>>> > deployment anyway.)
>>> >
>>> > > I think i have to break the data entry in separate chunks and do it.
>>> But not
>>> > > sure how to do that.
>>> >
>>> > In any case Heroku will apparently not let you import the full
>>> > database anyway until we switch to a payed account. If you still have
>>> > troubles with performance issues afterwards, this also won't help.
>>> > What you could do is temporarily copy or move a few folders (say,
>>> > math, science and python) and run portindex just on those. This will
>>> > give you a smaller number of ports to work with, but still sufficient
>>> > to figure out what other issues you'll need to deal with (port listing
>>> > will have to be paginated etc.)
>>> >
>>> > It's a good exercise to know what the limitations are. When you play
>>> > with new features and database design, it's always easier to play
>>> > with, say 10-100 ports than with full 15 MB portindex file.
>>> >
>>>
>>> This plan sounds good. +1
>>>
>>> --
>>> Jackson Isaac
>>>
>>
>>
>

Suggestions for Database Design

[Vishnu]

Hello all,

It would be really great if you could provide review/Suggestions for The
database design for the gsoc Project.

The Project aims at creating dynamic pages for port.
Displaying all information about a particular port.

The details are present in the readme.md
https://github.com/macports/macports-webap

Also suggestions are welcome here
https://github.com/macports/macports-webapp/blob/master/Database_Design.txt

Thanks
Vishnu

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

And also i couldn't figure out any way to hide passwords/ Sensitive
information while creating app.

Thanks

On 9 May 2018 at 03:43, Vishnu <vishnum1...@gmail.com> wrote:

> Hi
>
> I had one doubt.
> Should i switch the link in heroku account for integration with macports
> github ?
>
> Because i think then you need to give accesss to heroku of your account.
>
> I think it would be wise for me to do the commit update in my local
> repository itself..
>
> Once every 2 weeks or something ill push all the changes to macports
> repository.
> Do comment .What should be done?
>
> Thanks
>
> On 8 May 2018 at 11:40, Jackson Isaac <ijack...@macports.org> wrote:
>
>> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec <mo...@macports.org>
>> wrote:
>> >
>> > On 8 May 2018 at 01:12, Vishnu wrote:
>> > > Hi
>> > >
>> > > I will work on all the points mentioned today..
>> > >
>> > > But i have been trying many different ways.
>> > > To get 20k ports onto the database.. The site is crashing.. Timeout
>> error.
>> > > Or the site cant be reached.
>> > > Maximum i got till 9k ports successfully entered.
>> > >
>> > > Tried many ways..
>> > >
>> > > If you have any suggestions do let me know.
>> >
>>
>> We can even try firebase for the DB backend and host the frontend on
>> heroku.
>>
>> Firebase free version gives 1GB of storage and 10GB of bandwidth per
>> month.
>> I guess for testing out the app on complete portindex we could give it a
>> try.
>>
>> https://firebase.google.com/pricing/
>>
>> Then again we would have some kind of limitations over here in long run.
>>
>> >
>> > I was pretty sure that we would reach the database limit here.
>> >
>> >
>> > This site
>> > https://devcenter.heroku.com/articles/heroku-postgres-plans
>> > says that the limit is 10k rows which is not anywhere near enough and
>> > fully consistent with your observation. The plan that lets you have
>> > 10M rows is 9 USD per month which I gladly pay if that would solve the
>> > issues (I'm not sure if those are the only ones, we might not have
>> > sufficient memory etc.)
>> >
>>
>> 9USD is only for the DB though. We might need to pay for the compute
>> separately.
>>
>> Looks like free VMs come with only 512MB RAM.
>>
>> >
>> > (I also remembered that I might have access to create a clean virtual
>> > machine, but I need a couple of days to physically reach a place where
>> > I have the additional info. That would require setting up the whole
>> > machine manually of course, but we need to do that for the final
>> > deployment anyway.)
>> >
>> > > I think i have to break the data entry in separate chunks and do it.
>> But not
>> > > sure how to do that.
>> >
>> > In any case Heroku will apparently not let you import the full
>> > database anyway until we switch to a payed account. If you still have
>> > troubles with performance issues afterwards, this also won't help.
>> > What you could do is temporarily copy or move a few folders (say,
>> > math, science and python) and run portindex just on those. This will
>> > give you a smaller number of ports to work with, but still sufficient
>> > to figure out what other issues you'll need to deal with (port listing
>> > will have to be paginated etc.)
>> >
>> > It's a good exercise to know what the limitations are. When you play
>> > with new features and database design, it's always easier to play
>> > with, say 10-100 ports than with full 15 MB portindex file.
>> >
>>
>> This plan sounds good. +1
>>
>> --
>> Jackson Isaac
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I had one doubt.
Should i switch the link in heroku account for integration with macports
github ?

Because i think then you need to give accesss to heroku of your account.

I think it would be wise for me to do the commit update in my local
repository itself..

Once every 2 weeks or something ill push all the changes to macports
repository.
Do comment .What should be done?

Thanks

On 8 May 2018 at 11:40, Jackson Isaac <ijack...@macports.org> wrote:

> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec <mo...@macports.org>
> wrote:
> >
> > On 8 May 2018 at 01:12, Vishnu wrote:
> > > Hi
> > >
> > > I will work on all the points mentioned today..
> > >
> > > But i have been trying many different ways.
> > > To get 20k ports onto the database.. The site is crashing.. Timeout
> error.
> > > Or the site cant be reached.
> > > Maximum i got till 9k ports successfully entered.
> > >
> > > Tried many ways..
> > >
> > > If you have any suggestions do let me know.
> >
>
> We can even try firebase for the DB backend and host the frontend on
> heroku.
>
> Firebase free version gives 1GB of storage and 10GB of bandwidth per month.
> I guess for testing out the app on complete portindex we could give it a
> try.
>
> https://firebase.google.com/pricing/
>
> Then again we would have some kind of limitations over here in long run.
>
> >
> > I was pretty sure that we would reach the database limit here.
> >
> >
> > This site
> > https://devcenter.heroku.com/articles/heroku-postgres-plans
> > says that the limit is 10k rows which is not anywhere near enough and
> > fully consistent with your observation. The plan that lets you have
> > 10M rows is 9 USD per month which I gladly pay if that would solve the
> > issues (I'm not sure if those are the only ones, we might not have
> > sufficient memory etc.)
> >
>
> 9USD is only for the DB though. We might need to pay for the compute
> separately.
>
> Looks like free VMs come with only 512MB RAM.
>
> >
> > (I also remembered that I might have access to create a clean virtual
> > machine, but I need a couple of days to physically reach a place where
> > I have the additional info. That would require setting up the whole
> > machine manually of course, but we need to do that for the final
> > deployment anyway.)
> >
> > > I think i have to break the data entry in separate chunks and do it.
> But not
> > > sure how to do that.
> >
> > In any case Heroku will apparently not let you import the full
> > database anyway until we switch to a payed account. If you still have
> > troubles with performance issues afterwards, this also won't help.
> > What you could do is temporarily copy or move a few folders (say,
> > math, science and python) and run portindex just on those. This will
> > give you a smaller number of ports to work with, but still sufficient
> > to figure out what other issues you'll need to deal with (port listing
> > will have to be paginated etc.)
> >
> > It's a good exercise to know what the limitations are. When you play
> > with new features and database design, it's always easier to play
> > with, say 10-100 ports than with full 15 MB portindex file.
> >
>
> This plan sounds good. +1
>
> --
> Jackson Isaac
>

Re: New project member: vishnu

[Vishnu]

Hello

Thank you Rainer for welcoming me.

I look forward to progress i will make in GSOC 2018.

Thanks

On 8 May 2018 at 17:54, Rainer Müller <rai...@macports.org> wrote:

> Please join us in welcoming the following new MacPorts project member:
>
>  - Vishnu M (vishnu, @Vishnum98)
>
> We look forward to continued excellent contributions during Google
> Summer of Code 2018 and beyond.
>
> - Joshua, Ryan, and Rainer
>
>
> Do you want to join the MacPorts team? If you would like to be
> considered for team membership and commit access, please read this
> section of the Guide:
>
> http://guide.macports.org/chunked/project.membership.html
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Thanks for the repository link.

so my current email should be added as the collaborator for me to
contribute right?
or should i use the new macports id.
in that case do i have to make a new account or what?
not clear about that..


Also writing down some points we discussed for the community to know :

1) make the repository-  already done.
2) Upload a file with DB schema .
3) Set milestones in github.
4) Learn about unit tests
5) Really try to implement API feature.
7) Discuss about what commit needs to be shown on the port page.
8) Add columns for port and intended OS  , Port and build possible OS


jackson i would require the milestones page you talked about.


Thanks



On 7 May 2018 at 23:49, Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
> On Mon, May 7, 2018 at 12:22 AM, Bradley Giesbrecht <pixi...@macports.org>
> wrote:
>
>> > On Apr 30, 2018, at 10:34 AM, Joshua Root <j...@macports.org> wrote:
>> >
>> > On 2018-5-1 02:47 , Jackson Isaac wrote:
>> >> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu <vishnum1...@gmail.com>
>> wrote:
>> >>> Hey
>> >>>
>> >>> Repository names:
>> >>> MacPorts-features?
>> >>> Macportsinfo?
>> >>> Or just simple
>> >>> gsoc18?
>> >>>
>> >>>
>> >>
>> >> I was thinking about macports-portstatistics or macports-portstats.
>> >
>> > This is going to cover more than just stats; it's intended to fill the
>> > role that was envisioned for MPWA, so I think it would be appropriate to
>> > use that name.
>>
>> macports-webapp?
>>
>>
> New repository for GSoC 2018 - macports-webapp is set up now.
>
> It is accessible at [1]. Let us know if you have any queries. You would
> also
> require @macports id. I'll send a follow up mail on the thread.
>
> Links:
> [1] https://github.com/macports/macports-webapp
>
> @neverpanic, thanks for the quick help :)
>
> Regards,
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Yes.
I will remove case sensitive.
Right now the search works with only if any port beginswith those
characters.


Yes 7th may ITC my exams will be over at 5:00.

Time mentioned by you is fine.
I available any day.

Thanks

On Sun, May 6, 2018, 9:45 PM Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
> On Sun, May 6, 2018 at 3:52 PM, Vishnu <vishnum1...@gmail.com> wrote:
>
>> Hello
>>
>> Sorry for being inactive.
>> Had back to back exams .
>>
>> Got free today.
>>
>> I implemented a new search feature in my app .
>> Pushed the changes to heroku too.
>> Please go through :
>> sleepy-wave-33400.herokuapp.com/port/
>>
>> There you can search any port with its name.
>> Even just starting few characters would work.
>>
>>
> The search seems to be case-sensitive. For e.g., I entered 'app' but
> didn't get any matches whereas I had to enter 'App' to get some matches
> that too that begin with the entered text. It doesn't look for App in
> between the
> portname i.e., 'BGHUDAppKit'
>
> Also it would be great to populate the DB with the complete portindex. The
> installation
> statistics did you fetch from mpstats or manually entered some values ?
> Looks like they
> are from 2012.
>
> IIRC, your exam gets over tomorrow i.e., 7th May 2018 ? It would be great
> to setup
> an introduction meeting with mentors and student.
>
> I am okay with setting up a meeting any day after 1630 UTC (2200 IST). I
> would not be available the
> coming weekend i.e., from 10th May to 14th May.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Sorry for being inactive.
Had back to back exams .

Got free today.

I implemented a new search feature in my app .
Pushed the changes to heroku too.
Please go through :
sleepy-wave-33400.herokuapp.com/port/

There you can search any port with its name.
Even just starting few characters would work.

Thanks

On 30 April 2018 at 23:30, Mojca Miklavec <mo...@macports.org> wrote:

> On 30 April 2018 at 18:47, Jackson Isaac wrote:
> > On Mon, Apr 30, 2018 at 10:06 PM, Vishnu wrote:
> >> Hey
> >>
> >> Repository names:
> >> MacPorts-features?
> >> Macportsinfo?
> >> Or just simple
> >> gsoc18?
>
> No, gsoc18 is a bad name since the idea is to use the project
> afterwards as well and I don't like to rename it two years from now.
> (We can decide to rename it one month from now, but not once people
> start forking it etc.)
>
> > I was thinking about macports-portstatistics or macports-portstats.
>
> Or "macports-portinfo" as something inbetween? I'm somewhat reluctant
> about "stat[istics]" because it's about much more than statistics.
>
> >> And also in the long term. Parser would be run only once in forever or
> say a
> >> year.
>
> No way. Parser needs to be run after each commit. Initially it could
> be once per hour or so to simplify things and avoid hooks etc., but in
> the long term it needs to be run immediately after GitHub notifies you
> about the update.
>
> >> And before running the original db must be cleared.
> >>
> >> And then obvious question arise. What if a new port is added in
> macports.
> >
> > You are on a right path I would say. We would want it to update after
> > every port is added/updated
> > and built.
>
> Indeed, if a new port is added, you need to update the database. More
> important: also if a port is removed or if version changes etc.
>
> >> For that i would like to know the mechanics of new port addition.
>
> See one of recent pull requests:
>
> https://github.com/macports/macports-ports/pull/1694/files
>
> This commit will create a new Portfile with four subports
> (py-waitress, py27-waitress, py35-waitress, py36-waitress), update a
> bunch of existing ports and add a bunch of new subports to existing
> Portfiles (py35-paste, py36-paste, ...)
>
> We could at the same time declare a few ports obsolete or delete some.
>
> You get all that info from Portindex (it could be useful if we could
> get some kind of "differential Portindex"; I'm not absolutely sure how
> port deletions are handled).
>
> >> I think before the new port is added. It is obviously built before
> right?
>
> No. It is first added. Only once it is added it can be built on the
> buildbot (a commit to repository triggers a build on the buildbot).
>
> >> So
> >> maybe we can add some code there to update the table with a row when new
> >> port is added.
> >
> > When a port is submitted, it is built by Travis CI and also our
> > buildbots,
>
> Travis only builds pull requests, not actually commits.
>
> > which builds
> > the port for mac 10.6 to 10.13 versions and gives the build status. We
> > would be more
> > interested in the status provided by the buildbot.
>
> I would ignore Travis builds for now anyway. It could be interesting
> to add pointers to open pull requests for a particular port, but not
> right now. Evidently your code needs to first update the database to
> get the list of all new ports, information about successful or failed
> builds would come later from the buildbot.
>
> > There could be a trigger or a listener which runs whenever a build job
> > is initiated and then
> > updates the DB on completion. I believe there would be some mechanism
> > to know the status
> > of each job.
>
> You can access the API from the buildbot to ask for status. But if you
> don't have an entry for a particular port first, you might run into
> troubles if you'll want to add build results without being able to
> reference a port with that name.
>
> > Maybe someone more familiar with buildbot could share some insights ?
>
> In any case it would make a lot more sense to start with tens of
> thousands old builds first and only once that is done add the latest
> information as soon as it's available. There is a high chance that
> things will change if we switch to buildbot version one (which might
> happen in the meantime).
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Repository names:
MacPorts-features?
Macportsinfo?
Or just simple
gsoc18?



I will look into the feedback.
Also i am currently using create.

And also in the long term. Parser would be run only once in forever or say
a year.
And before running the original db must be cleared.

And then obvious question arise. What if a new port is added in macports.

For that i would like to know the mechanics of new port addition.

I think before the new port is added. It is obviously built before right?
So maybe we can add some code there to update the table with a row when new
port is added.

Just brainstorming.


Thanks

On Mon, Apr 30, 2018, 2:23 PM Mojca Miklavec <mo...@macports.org> wrote:

> On 30 April 2018 at 02:22, Vishnu wrote:
> > Hi
> >
> > After hours of fixing error in heroku and my ubuntu .
> > Finally got to run the so far app.
>
> Yay! That's great since it will be much easier for the whole community
> to test the app much more quickly.
>
> Now, it would be great to pick some name for the repository and
> populate it, to simplify further work.
>
> > Here all the ports are listed:
> > sleepy-wave-33400.herokuapp.com/port/
> >
> > When you'll open the page..probable the ports would not be there because
> i
> > cleared the DB.
> >
> > To add the ports info into DB..we need to open this url:
> > sleepy-wave-33400.herokuapp.com/port/parser
> >
> > Parsing is done using the portindex.json i have.
> > For simplicity i have only 6 portfile data in it.
> >
> > Once you open sleepy-wave-33400.herokuapp.com/port/
> > you can click on individual port to get further informant about that
> port.
> > infact u can access this information using :
>
> Cool. Some feedback:
> - the css should go to a separate static file
> - allowing port id might be a bad idea because there might be a port
> called "123"
> - calling parser multiple times populates the database with multiple
> copies of the same port, you should use something like
> "create_or_update"
> - we need to figure out the best way to call the "parser"
>
> I might have a few other comments, but the initial purpose was to
> check if we have an easy way to deploy the development version and
> this has been achieved.
>
> Please come up with proposal(s) for project name, so that we can make
> an explicit request to the infrastructure team to create a new repo.
>
> Random brainstorming ideas: webapp, "port guardian" (in whatever
> spelling), ...
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Hello

After hours of fixing error in heroku and my ubuntu .
Finally got to run the so far app .



Here all the ports are listed:
sleepy-wave-33400.herokuapp.com/port/

When you'll open the page..probable the ports would not be there because i
cleared the DB.

To add the ports info into DB..we need to open this url:
sleepy-wave-33400.herokuapp.com/port/parser

Parsing is done using the portindex.json i have.
For simplicity i have only 6 portfile data in it.

Once you open sleepy-wave-33400.herokuapp.com/port/
you can click on individual port to get further informant about that port.
infact u can access this information using :

sleepy-wave-33400.herokuapp.com/port/whatever_is_the_port_name
sleepy-wave-33400.herokuapp.com/port/whatever_is_the_port_id


Thanks


On 30 April 2018 at 02:14, Mojca Miklavec <mo...@macports.org> wrote:

> On 29 April 2018 at 19:34, Vishnu wrote:
> > Hey
> >
> > I wanted to work on finalising the database. What all final columns in
> each
> > table.
>
> This is a creative process that should in the first place be carefully
> crafted by you - of course with sufficient input and feedback from our
> side.
>
> > Take your final suggestions.
>
> I'm pretty sure that things will change during the course of the
> summer once you figure out that there might be additional
> requirements, that you might need additional fields to achieve what
> you wanted etc. Django usually lets you transition from one schema to
> a slightly different one in a relatively painless way.
>
> But there's one thing that we should probably discuss together with
> other members, and that is, to what extent to support:
> - historic information about ports (as versions, dependencies,
> maintainers etc. change, ports become obsolete, ...)
> - the fact that different OS versions might support different ports or
> different versions of ports
>
> Vishnu, I would like to suggest to put your Excel schematic to some
> easy-to-read-and-edit document in that github repository. This could
> be some Markdown table, or anything else. You can search a bit for
> programs that let you easily create ERD (Entity Relation Diagram)
> diagrams. This document will then serve at least three purposes:
> - brainstorming with us about the most reasonable schematic, improvements
> etc.
> - enormously help you when you start coding you app
> - as a documentation that will allow anyone else to understand and
> improve the program later on without reading the code
>
> > https://www.macports.org/ports.php?by=platform=linux
> > How does this work?
> > where is the database?
> > how are queries relayed?
> > where is the actual code for this?
>
> There is a PostgreSQL database on the server run by Clemens and Rainer
> in Germany (most likely the same server where you app would eventually
> run).
>
> I would be grateful if someone from the infrastructure team could
> correct me here in case I'm giving you wrong pointers, but I suspect
> that the database gets populated with this script:
> https://github.com/macports/macports-infrastructure/blob/
> master/jobs/PortIndex2PGSQL.tcl
> by converting data from PortIndex to SQL and filling the database with it.
>
> I suspect the rendering is done with
> https://github.com/macports/macports-www/blob/master/ports.php
>
> You may optionally take a glimpse at various repositories at
> https://github.com/macports/
> (ignore user repositories), quickly look at what is there and ask if
> there's something more that you would like to know.
>
> This can give you some ideas, but I assume you would populate the
> database in pretty much the same way as you did during your coding
> challenge, and according to the database design developed during
> proposal writing and during community bonding period.
>
> > Also for my project what database should i use.
> > i recall postgre sql.or something else?
>
> Yes, PostgreSQL is probably the best OpenSource relational database
> (or at least it was some years ago), my preference would be to use
> that one. SQLite is too limited. MySQL (derivatives?) would be
> acceptable, but PG sounds better. Commercial databases are out of
> question.
>
> > So should i start creating tickets?
> > like ..at least putting putting up my basic milestones.
>
> Yes, I would like to see the tickets with milestones once we have the
> repository. Things like "transfer project proposal to documentation"
> with a milestone bonding period could also be stuff that ends up
> there.
>
> > Also you said :
> > (c.3) Plan the API to get the data from the database in JSON format
> > (so that someone else could write an independent app with the same
> > display funct

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Do let me know when the repository is gonna be made. I can start using it.

I wanted to work on finalising the database. What all final columns in each
table.
Take your final suggestions.

https://www.macports.org/ports.php?by=platform=linux
How does this work?
where is the database?
how are queries relayed?
where is the actual code for this?

Also for my project what database should i use.
i recall postgre sql.or something else?

So should i start creating tickets?
like ..at least putting putting up my basic milestones.

Also you said :
(c.3) Plan the API to get the data from the database in JSON format
(so that someone else could write an independent app with the same
display functionality).
More in a separate email, I guess.

I'll do this once i finish my database.


I tried to understand a bit about heroku.It'll work when needed.Not an
issue i guess.


Thanks




On 29 April 2018 at 20:37, Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> On 29 April 2018 at 16:45, Vishnu wrote:
> > Hi
> >
> > Jackson i have already replied to the mail regarding username.
> >
> > Mojca, I was able to edit the file and entered values manually in the
> code
> > of mpstats.
> >
> > vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
> show
> > Would submit the following data to
> > http://stats.macports.neverpanic.de/submissions:
> > {
> >   "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
> >   "os": {
> > "macports_version": "2.4.99",
> > "osx_version": "10.15",
> > "os_arch": "i386",
> > "os_platform": "linux",
> > "build_arch": "ppc",
> > "gcc_version": "5.4.0",
> > "xcode_version": "none"
> >   },
> >   "active_ports": [
> > {"name": "ncurses", "version": "6.1_0"},
> > {"name": "libedit", "version": "20170329-3.1_2"},
> > {"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
> >   ],
> >   "inactive_ports": []
> > }
> > vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
> > submit
> > Submitting data to http://stats.macports.neverpanic.de/submissions ...
> > Success.
> >
> > So even upon submitting .There is no update on the site.
> > idk why.
>
> I'm almost equally not familiar with the code as you are (I did make
> some minor changes a while ago). While we could start debugging it now
> (with only Clemens being able to access the logs and deploying any
> changes), I believe this would mostly be a waste of time since a good
> implementation of your project would eventually make this site
> obsolete anyway. Of course it makes sense to get familiar with rails
> application serving the statistics page, but spending a lot of time
> fixing it is simply not worth the trouble in my opinion.
>
> https://github.com/macports/macports-contrib/tree/gsoc11-statistics/
>
> Continue the planning phase (or any other task) would likely be much
> more useful at this point.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Jackson i have already replied to the mail regarding username.

Mojca, I was able to edit the file and entered values manually in the code
of mpstats.

vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
Would submit the following data to
http://stats.macports.neverpanic.de/submissions:
{
  "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
  "os": {
"macports_version": "2.4.99",
"osx_version": "10.15",
"os_arch": "i386",
"os_platform": "linux",
"build_arch": "ppc",
"gcc_version": "5.4.0",
"xcode_version": "none"
  },
  "active_ports": [
{"name": "ncurses", "version": "6.1_0"},
{"name": "libedit", "version": "20170329-3.1_2"},
{"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
  ],
  "inactive_ports": []
}
vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
submit
Submitting data to http://stats.macports.neverpanic.de/submissions ...
Success.


So even upon submitting .There is no update on the site.
idk why.

Thanks
Vishnu

On 29 April 2018 at 15:19, Mojca Miklavec <mo...@macports.org> wrote:

> On 29 April 2018 at 10:17, Jackson Isaac wrote:
> > On Sun, Apr 29, 2018 at 1:21 PM, Mojca Miklavec wrote:
> >>
> >>
> >> I would say that the next most reasonable step (apart from any other
> >> thing from the list I sent in the beginning) would be to create a
> >> repository and put all the documentation / plan / milestones / sample
> >> page etc. to that repository.
> >> This means:
> >> - deciding for the name
> >
> > We can go with something like 'macports-buildstatistics' for repo name.
>
> But keep in mind that this project will now contain a lot more than
> just build statistics. It's:
> - portindex
> - build summary
> - usage statistics
> - whether the port is outdated
> - etc.
>
> So "buildstatistics" sounds too narrow.
>
> > If it would be a branch under /macports-base then
> 'gsoc18-buildstatistics'.
>
> Having it under base would be a bad idea since it will be completely
> independent.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Thanks a lot.
Now it works.
I dont know why it was not working earlier.

And jackson . I ran both the commands.
root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
Would submit the following data to
http://stats.macports.neverpanic.de/submissions:
{
  "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
  "os": {
"macports_version": "2.4.99",
"osx_version": "",
"os_arch": "i386",
"os_platform": "linux",
"build_arch": "",
"gcc_version": "5.4.0",
"xcode_version": "none"
  },
  "active_ports": [
{"name": "ncurses", "version": "6.1_0"},
{"name": "libedit", "version": "20170329-3.1_2"},
{"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
  ],
  "inactive_ports": []
}
root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats submit
Submitting data to http://stats.macports.neverpanic.de/submissions ...
Success.




On 29 April 2018 at 07:29, Jeremy Lavergne <jer...@lavergne.me> wrote:

> There's one last thing to try regarding paths; that is to try invoking
> the file this way:
> /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
>
>
> If that doesn't work, it may lead to wondering if all of MacPorts is
> installed properly.
>
> For example, I see you're using the root user: did we ensure that all
> the file permissions were correct for when you're not the root user? Or
> have you been using the root user exclusively?
>
>
> On 04/28/2018 09:28 PM, Vishnu wrote:
> > I dont think its about PATH.
> > root@vishnupc:/# /opt/local/libexec/mpstats
> > can't find package macports
> > while executing
> > "package require macports"
> > (file "/opt/local/libexec/mpstats" line 42)
> > root@vishnupc:/# echo $PATH
> > /usr/local/sbin:/opt/local/bin:/usr/local/bin:/usr/sbin:/
> usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> >
> > I tried all combinations of path.
> > Didn't work.
> >
> > what does it mean exactly by macports package?
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I dont think its about PATH.
root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)
root@vishnupc:/# echo $PATH
/usr/local/sbin:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

I tried all combinations of path.
Didn't work.

what does it mean exactly by macports package?


On 29 April 2018 at 06:47, Jeremy Lavergne <jer...@lavergne.me> wrote:

> But also notice how you already have /opt/local/bin in your PATH...
>
> On 04/28/2018 09:16 PM, Jeremy Lavergne wrote:
> > A missing / on /opt
> >
> > On 04/28/2018 08:49 PM, Vishnu wrote:
> >> I added the path.But still same error.
> >>
> >> root@vishnupc:/# /opt/local/libexec/mpstats
> >> can't find package macports
> >> while executing
> >> "package require macports"
> >> (file "/opt/local/libexec/mpstats" line 42)
> >> root@vishnupc:/# echo $PATH
> >> /usr/local/sbin:opt/local/bin/port-tclsh:/opt/local/bin:/
> usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> >>
> >>
> >> On 29 April 2018 at 05:56, Ken Cunningham
> >> <ken.cunningham.web...@gmail.com
> >> <mailto:ken.cunningham.web...@gmail.com>> wrote:
> >>
> >>
> >>
> >>> On Apr 28, 2018, at 5:06 PM, Vishnu <vishnum1...@gmail.com
> >>> <mailto:vishnum1...@gmail.com>> wrote:
> >>>
> >>> am i supposed to install port tclsh ?
> >>>
> >>> There's no port named tclsh.
> >>>
> >>
> >> you should have it
> >>
> >> /opt/local/bin/port-tclsh
> >>
> >> but your path is probably calling
> >>
> >> /usr/bin/tclsh
> >>
> >>
> >> Ken
> >>
> >>
> >
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I added the path.But still same error.

root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)
root@vishnupc:/# echo $PATH
/usr/local/sbin:opt/local/bin/port-tclsh:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games


On 29 April 2018 at 05:56, Ken Cunningham <ken.cunningham.web...@gmail.com>
wrote:

>
>
> On Apr 28, 2018, at 5:06 PM, Vishnu <vishnum1...@gmail.com> wrote:
>
> am i supposed to install port tclsh ?
>
> There's no port named tclsh.
>
>
> you should have it
>
> /opt/local/bin/port-tclsh
>
> but your path is probably calling
>
> /usr/bin/tclsh
>
>
> Ken
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

am i supposed to install port tclsh ?

There's no port named tclsh.




On 29 April 2018 at 05:24, Joshua Root <j...@macports.org> wrote:

> On 2018-4-29 09:36 , Vishnu wrote:
> > Hi
> >
> > I had created the user.
> >
> > But when trying /opt/local/libexec/mpstats
> >
> > root@vishnupc:/# /opt/local/libexec/mpstats
> > can't find package macports
> > while executing
> > "package require macports"
> > (file "/opt/local/libexec/mpstats" line 42)
> >
> > What's wrong.
>
> Your tclsh can't find the 'macports' package. Probably the location of
> port-tclsh is missing from your PATH and it's using /usr/bin/tclsh.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I had created the user.

But when trying /opt/local/libexec/mpstats

root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)

What's wrong.


On 26 April 2018 at 01:24, Jackson Isaac <ijack...@macports.org> wrote:

> On Thu, Apr 26, 2018 at 1:02 AM, Vishnu <vishnum1...@gmail.com> wrote:
> > these are the errors.
> >
> > On 26 April 2018 at 00:59, Vishnu <vishnum1...@gmail.com> wrote:
> >>
> >> Yes I did add  'revupgrade_autorun no' line to
> >> /opt/local/etc/macports/macports.conf
> >> with sudo..
> >> I cross checked it..it has been updated.
> >>
>
> It's been quite some time I have touched Ubuntu. From Joshua's mail
> and your logs I understand that macports user was not created.
>
> Could you try running
>
> $ sudo addgroup macports
>
> $ sudo adduser macports macports
>
> I am not exactly sure though if the group name is correct or not.
>
> Let us know if you still get the same errors after following these steps.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Finally was able to install it.
And successfully run mpstats.
root@vishnupc:~# /opt/local/bin/port content  mpstats
Port mpstats contains:
  /Library/LaunchDaemons/org.macports.mpstats.plist

/opt/local/etc/LaunchDaemons/org.macports.mpstats/org.macports.mpstats.plist.default
  /opt/local/etc/macports/stats.conf
  /opt/local/libexec/mpstats

Mojca you said there will be a change in os statistics (
http://stats.macports.neverpanic.de/os_statistics#os_platform) .
I couldn't see any change.

Thanks

On 26 April 2018 at 01:24, Jackson Isaac <ijack...@macports.org> wrote:

> On Thu, Apr 26, 2018 at 1:02 AM, Vishnu <vishnum1...@gmail.com> wrote:
> > these are the errors.
> >
> > On 26 April 2018 at 00:59, Vishnu <vishnum1...@gmail.com> wrote:
> >>
> >> Yes I did add  'revupgrade_autorun no' line to
> >> /opt/local/etc/macports/macports.conf
> >> with sudo..
> >> I cross checked it..it has been updated.
> >>
>
> It's been quite some time I have touched Ubuntu. From Joshua's mail
> and your logs I understand that macports user was not created.
>
> Could you try running
>
> $ sudo addgroup macports
>
> $ sudo adduser macports macports
>
> I am not exactly sure though if the group name is correct or not.
>
> Let us know if you still get the same errors after following these steps.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Yes I did add  'revupgrade_autorun no' line to
/opt/local/etc/macports/macports.conf
with sudo..
I cross checked it..it has been updated.

On 26 April 2018 at 00:54, Jackson Isaac <ijack...@macports.org> wrote:

> On Thu, Apr 26, 2018 at 12:49 AM, Vishnu <vishnum1...@gmail.com> wrote:
> > It is exactly the same error as in previous one.
> >
> >> --->  Cleaning mpstats
> >> --->  Updating database of binaries
> >> Warning: Error determining file type of `/opt/local/bin/captoinfo':
> >> invalid command name "fileIsBinary"
> >> Warning: A file belonging to the `ncurses' port is missing or
> >> unreadable. Consider reinstalling it.
> >> Warning: Error determining file type of `/opt/local/bin/clear': invalid
> >> command name "fileIsBinary"
> >> Warning: A file belonging to the `ncurses' port is missing or
> >> unreadable. Consider reinstalling it.
> >> Warning: Error determining file type of `/opt/local/bin/infocmp':
> >> invalid command name "fileIsBinary"
> >
>
> Did you try adding 'revupgrade_autorun no' line to
> /opt/local/etc/macports/macports.conf ?
> You might need to open the file with sudo.
>
> > vishnu@macports would be fine i guess.
> >
> > i'll join as soon as i get the id.
> >
>
> You can join IRC without @macports id.
>
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

It is exactly the same error as in previous one.

> --->  Cleaning mpstats
> --->  Updating database of binaries
> Warning: Error determining file type of `/opt/local/bin/captoinfo':
> invalid command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/clear': invalid
> command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/infocmp':
> invalid command name "fileIsBinary"

vishnu@macports would be fine i guess.

i'll join as soon as i get the id.

Thanks

On 26 April 2018 at 00:37, Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
> On Thu, Apr 26, 2018 at 12:33 AM, Vishnu <vishnum1...@gmail.com> wrote:
> > anyone?
> >
> > On 25 April 2018 at 21:49, Vishnu <vishnum1...@gmail.com> wrote:
> >>
> >> Hi
> >>
> >> Still same error..Dont know what to do.
> >>
>
> Could you please paste the error that you are getting somewhere e.g.,
> dpaste.de ?
>
> It would be really helpful if you join IRC channel #macports on freenode.
> I use IRCCloud to stay connected -
> https://www.irccloud.com/irc/freenode/channel/macports
> The link should help you get started.
>
> You would also need a @macports id. Do you have any preference for an
> username ?
>
> >>
> >> On 25 April 2018 at 21:24, Joshua Root <j...@macports.org> wrote:
> >>>
> >>> On 2018-4-26 01:39 , Vishnu wrote:
> >>> > Hi
> >>> >
> >>> > I think its best if you can create a repository under Macports.
> >>> > I'll be coding there.
> >>> >
> >>> > Also while trying to install mpstats .
> >>> > I am getting these error
> >>> >
> >>> > Warning: configured user/group macports does not exist, will build as
> >>> > root
> >>>
> >>> Our build system only knows how to create users on macOS. You'll have
> to
> >>> create the macports user manually.
> >>>
> >>> > --->  Cleaning mpstats
> >>> > --->  Updating database of binaries
> >>> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
> >>> > invalid command name "fileIsBinary"
> >>> > Warning: A file belonging to the `ncurses' port is missing or
> >>> > unreadable. Consider reinstalling it.
> >>> > Warning: Error determining file type of `/opt/local/bin/clear':
> invalid
> >>> > command name "fileIsBinary"
> >>> > Warning: A file belonging to the `ncurses' port is missing or
> >>> > unreadable. Consider reinstalling it.
> >>> > Warning: Error determining file type of `/opt/local/bin/infocmp':
> >>> > invalid command name "fileIsBinary"
> >>> >
> >>> > Didnt find a solution on the net either.
> >>>
> >>> That's rev-upgrade failing. It also only works on macOS. It's not
> >>> essential though, so you can set 'revupgrade_autorun no' in your
> >>> macports.conf.
> >>>
> >>> - Josh
> >>
> >>
> >
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

anyone?

On 25 April 2018 at 21:49, Vishnu <vishnum1...@gmail.com> wrote:

> Hi
>
> Still same error..Dont know what to do.
>
>
> On 25 April 2018 at 21:24, Joshua Root <j...@macports.org> wrote:
>
>> On 2018-4-26 01:39 , Vishnu wrote:
>> > Hi
>> >
>> > I think its best if you can create a repository under Macports.
>> > I'll be coding there.
>> >
>> > Also while trying to install mpstats .
>> > I am getting these error
>> >
>> > Warning: configured user/group macports does not exist, will build as
>> root
>>
>> Our build system only knows how to create users on macOS. You'll have to
>> create the macports user manually.
>>
>> > --->  Cleaning mpstats
>> > --->  Updating database of binaries
>> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
>> > invalid command name "fileIsBinary"
>> > Warning: A file belonging to the `ncurses' port is missing or
>> > unreadable. Consider reinstalling it.
>> > Warning: Error determining file type of `/opt/local/bin/clear': invalid
>> > command name "fileIsBinary"
>> > Warning: A file belonging to the `ncurses' port is missing or
>> > unreadable. Consider reinstalling it.
>> > Warning: Error determining file type of `/opt/local/bin/infocmp':
>> > invalid command name "fileIsBinary"
>> >
>> > Didnt find a solution on the net either.
>>
>> That's rev-upgrade failing. It also only works on macOS. It's not
>> essential though, so you can set 'revupgrade_autorun no' in your
>> macports.conf.
>>
>> - Josh
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Still same error..Dont know what to do.


On 25 April 2018 at 21:24, Joshua Root <j...@macports.org> wrote:

> On 2018-4-26 01:39 , Vishnu wrote:
> > Hi
> >
> > I think its best if you can create a repository under Macports.
> > I'll be coding there.
> >
> > Also while trying to install mpstats .
> > I am getting these error
> >
> > Warning: configured user/group macports does not exist, will build as
> root
>
> Our build system only knows how to create users on macOS. You'll have to
> create the macports user manually.
>
> > --->  Cleaning mpstats
> > --->  Updating database of binaries
> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
> > invalid command name "fileIsBinary"
> > Warning: A file belonging to the `ncurses' port is missing or
> > unreadable. Consider reinstalling it.
> > Warning: Error determining file type of `/opt/local/bin/clear': invalid
> > command name "fileIsBinary"
> > Warning: A file belonging to the `ncurses' port is missing or
> > unreadable. Consider reinstalling it.
> > Warning: Error determining file type of `/opt/local/bin/infocmp':
> > invalid command name "fileIsBinary"
> >
> > Didnt find a solution on the net either.
>
> That's rev-upgrade failing. It also only works on macOS. It's not
> essential though, so you can set 'revupgrade_autorun no' in your
> macports.conf.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I think its best if you can create a repository under Macports.
I'll be coding there.

Also while trying to install mpstats .
I am getting these error

Warning: configured user/group macports does not exist, will build as root
--->  Cleaning mpstats
--->  Updating database of binaries
Warning: Error determining file type of `/opt/local/bin/captoinfo': invalid
command name "fileIsBinary"
Warning: A file belonging to the `ncurses' port is missing or unreadable.
Consider reinstalling it.
Warning: Error determining file type of `/opt/local/bin/clear': invalid
command name "fileIsBinary"
Warning: A file belonging to the `ncurses' port is missing or unreadable.
Consider reinstalling it.
Warning: Error determining file type of `/opt/local/bin/infocmp': invalid
command name "fileIsBinary"

Didnt find a solution on the net either.

On 25 April 2018 at 19:09, Mojca Miklavec <mo...@macports.org> wrote:

> On 25 April 2018 at 14:45, Vishnu wrote:
> > Hello
> >
> > Succesfully ran the portindex2json.
> > and saved it ino a json.
> >
> > So what next?
>
> Wonderful, that's very good news, that will make it much easier to
> proceed with the rest of the tasks.
>
> Now:
> - play a bit with "port search python", "port info python27", ...
> - try to install some super simple packages[***]
> - try to run "port -v livecheck category:math" to see if you can get
> livecheck running correctly (you'll need that info)
> - try to install "sudo port install mpstats", then "port contents
> mpstats"; if successful, you should see a new entry at
> http://stats.macports.neverpanic.de/os_statistics#os_platform
>   I forgot the exact command, but there's a way to show what will be
> submitted, I guess you can find that out by inspecting the source or
> by running help or something
> (I'm not behind my mac at the moment)
>
>
> Just try to get familiar with the "port" command and then proceed to
> whatever other point you want, like
>
>
> - trying to get buildbot master and slave installed (I guess you can
> install them from the native package manager; alternatively via
> virtualenv which gives you slightly more flexibility in picking up the
> exact version that you want); the next step would be to try to
> replicate our setup, see
> https://github.com/macports/macports-infrastructure/tree/
> master/buildbot
>
> - converting the application (without personal details) into
> documentation/plan in a git(hub) repository (something like a markdown
> document)
>
> - (lower priority) proceeding with the simple html page that you
> started working on as a coding challenge, as part of improving &
> finalizing the plan for the summer
>
> - any other item from suggestions sent earlier
>
> Mojca
>
> [***] It's hard to tell what "simple packages" are, but if "platforms"
> contains "linux", there's at least a somewhat bigger chance that it
> will compile correctly. I guess you can expect a number of failures
> because the number of macports users on linux is super limited. Also,
> the build time might be somewhat high since everything needs to be
> compiled from source. A package like "mpstats" should be simple enough
> though, I assume you can find some others. Avoid anything that depends
> on Qt.
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Succesfully ran the portindex2json.
and saved it ino a json.

So what next?

On 25 April 2018 at 17:22, Mojca Miklavec <mo...@macports.org> wrote:

> On 25 April 2018 at 13:45, Vishnu wrote:
> > Hi
> >
> > I ran portindex.
> > It gives me 2 files Portindex and Portindex.quick
>
> I suggest to keep the discussion on the list, so that others can jump
> in and help when needed.
>
> But in short: https://github.com/macports/macports-contrib/blob/master/
> portindex2json/portindex2json.tcl
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi all

was succesfully able to install macports using  |$| *|cd
/opt/mports/macports-base|*
> |$| *|./configure --enable-readline|*
> |$| *|make|*
> |$| *|sudo make install|*
> |$| *|make distclean|*

but right now problem is

root@vishnupc:~# /opt/local/bin/port -v selfupdate
--->  Updating MacPorts base sources using rsync
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by
peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(276)
[Receiver=3.1.1]
Command failed: /usr/bin/rsync -rtzvl --delete-after rsync://
rsync.macports.org/macports/release/tarballs/base.tar
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs
Exit code: 1

what to do now?

On 24 April 2018 at 22:14, Joshua Root <j...@macports.org> wrote:

> On 2018-4-25 02:36 , Vishnu wrote:
> > I tried using these
> >
> >  1.
> >
> > *|mkdir -p /opt/mports|*
> > |$| *|cd /opt/mports|*
> > |$| *|git clone https://github.com/macports/macports-base.git|*
> > |$| *|git checkout v2.4.3 # skip this if you want to use the
> development
> > version|*
> >
> >  2.
> >
> > Build and Install MacPorts
> >
> > MacPorts uses autoconf and makefiles for installation. These
> > commands will build and install MacPorts to |/opt/local|. You can
> > add |--prefix| to |./configure| to relocate MacPorts to another
> > directory if needed.
> >
> > |$| *|cd /opt/mports/macports-base|*
> > |$| *|./configure --enable-readline|*
> > |$| *|make|*
> > |$| *|sudo make install|*
> > |$| *|make distclean|*
> >
> > while trying to configure i am getting :configure: error: mtree not
> > found.
> >
> > How to proceed?
> >
> > I think i need to install additional files.
> > Can someone help?
>
> See the list of prerequisites at
> <https://www.macports.org/install.php#other>.
>
> Look up which package provides mtree on your distro and install it. The
> executable is sometimes called 'fmtree' on Linux, so you may need to set
> MTREE=fmtree (or whatever it's actually called) when running configure.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I tried using these

   1.

   *mkdir -p /opt/mports*$ *cd /opt/mports*$ *git clone
https://github.com/macports/macports-base.git
<https://github.com/macports/macports-base.git>*$ *git checkout v2.4.3
 # skip this if you want to use the development version*

   2.

   Build and Install MacPorts

   MacPorts uses autoconf and makefiles for installation. These commands
   will build and install MacPorts to /opt/local. You can add --prefix to
   ./configure to relocate MacPorts to another directory if needed.

   $ *cd /opt/mports/macports-base*$ *./configure --enable-readline*$
*make*$ *sudo make install*$ *make distclean*


while trying to configure i am getting :configure: error: mtree not
found.

How to proceed?

I think i need to install additional files.
Can someone help?

On 24 April 2018 at 21:58, Vishnu <vishnum1...@gmail.com> wrote:

> Hey
>
> I have been trying to install Macports on ubuntu.
> And am unable to do so.
>
> Downloaded the tar.
> Somehelp would be appreciated.
>
>
> On 24 April 2018 at 08:55, Jackson Isaac <ijack...@macports.org> wrote:
>
>> Hi Vishnu,
>>
>> Welcome to The MacPorts Project!
>>
>> On Tue, Apr 24, 2018 at 4:15 AM, Vishnu <vishnum1...@gmail.com> wrote:
>> > Hey,
>> >
>> > So first thing.
>> > (a.1) I am planning to install ubuntu on my pc in a day.
>> > I guess We can install Macports on that.
>>
>> That's great. Let us know if you face any issues and if we could be of
>> any help.
>>
>> I was also thinking about cloud platform (specifically Google Cloud)
>> which gives
>> you around 300$ of credits for one year where you could spin up a VM and
>> test
>> the installation.
>>
>> Note: It might need a Credit Card for authentication.
>>
>> > (a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot
>> too.
>> >
>> > (b.1) I went and completed week 1.But i am unable to access week 3 ,
>> week
>> > 4.Because the course starts on April 30 i guess.
>> > I went through the topics.
>> > Week 3 : Subqueries and Join
>> > Week 4 : Modifying and Analyzing Data with SQL
>> >
>>
>> I guess you can join the upcoming session and complete all the
>> assignments and weeks
>> before hand too (I was able to do so few weeks back).
>>
>> > So now would learn more about these topics through youtube, many other
>> > resources.
>> >
>> > (b.2) Will look into them.
>> > (b.3) Django hopefully wouldn't be a problem.
>> >
>> > (c.0)  I would suggest that we create a new repository for your project
>> > under
>> > MacPorts, you fork it and use your own fork for endlessly playing with
>> > it, testing etc
>> > This sounds good.
>> > I think i would be more comfortable with Github.
>> >
>> > (c.2) Can be done.
>> > (c.3) You are talking about Build History data ..right?
>> > There is already a JSON API for it as you said..so cant we use it? (
>> > https://build.macports.org/json/help )
>> >
>> > (d.1 , d.2) Yes will do the research .
>> >
>> >
>> > (e.1) I think when i will be changing Buildbot to update the database
>> after
>> > every build ..Then i would need Tcl.
>> > Will learn them as i get time.
>> >
>>
>> Speaking from experience, I would suggest go through the basics of Tcl
>> and write some small sample programs to get the gist of Tcl.
>> This would help you recall whatever you tried now at a later point
>> of time when you actually touch macports-base.
>>
>> https://www.tcl.tk/man/tcl8.5/tutorial/Tcl0.html helped me a lot.
>> I always keep this window open while developing base.
>>
>> > Will see Project Description.
>> >
>> > Also i have my Semester End exams from 27 April - 7 may.
>> > So maybe i would not be able to reply on time.
>> >
>>
>> No issues. Do well for the exams :) Keep us updated.
>>
>> Looking forward to a great summer this year :)
>>
>> --
>> Jackson Isaac
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

I have been trying to install Macports on ubuntu.
And am unable to do so.

Downloaded the tar.
Somehelp would be appreciated.


On 24 April 2018 at 08:55, Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
> Welcome to The MacPorts Project!
>
> On Tue, Apr 24, 2018 at 4:15 AM, Vishnu <vishnum1...@gmail.com> wrote:
> > Hey,
> >
> > So first thing.
> > (a.1) I am planning to install ubuntu on my pc in a day.
> > I guess We can install Macports on that.
>
> That's great. Let us know if you face any issues and if we could be of any
> help.
>
> I was also thinking about cloud platform (specifically Google Cloud) which
> gives
> you around 300$ of credits for one year where you could spin up a VM and
> test
> the installation.
>
> Note: It might need a Credit Card for authentication.
>
> > (a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot
> too.
> >
> > (b.1) I went and completed week 1.But i am unable to access week 3 , week
> > 4.Because the course starts on April 30 i guess.
> > I went through the topics.
> > Week 3 : Subqueries and Join
> > Week 4 : Modifying and Analyzing Data with SQL
> >
>
> I guess you can join the upcoming session and complete all the
> assignments and weeks
> before hand too (I was able to do so few weeks back).
>
> > So now would learn more about these topics through youtube, many other
> > resources.
> >
> > (b.2) Will look into them.
> > (b.3) Django hopefully wouldn't be a problem.
> >
> > (c.0)  I would suggest that we create a new repository for your project
> > under
> > MacPorts, you fork it and use your own fork for endlessly playing with
> > it, testing etc
> > This sounds good.
> > I think i would be more comfortable with Github.
> >
> > (c.2) Can be done.
> > (c.3) You are talking about Build History data ..right?
> > There is already a JSON API for it as you said..so cant we use it? (
> > https://build.macports.org/json/help )
> >
> > (d.1 , d.2) Yes will do the research .
> >
> >
> > (e.1) I think when i will be changing Buildbot to update the database
> after
> > every build ..Then i would need Tcl.
> > Will learn them as i get time.
> >
>
> Speaking from experience, I would suggest go through the basics of Tcl
> and write some small sample programs to get the gist of Tcl.
> This would help you recall whatever you tried now at a later point
> of time when you actually touch macports-base.
>
> https://www.tcl.tk/man/tcl8.5/tutorial/Tcl0.html helped me a lot.
> I always keep this window open while developing base.
>
> > Will see Project Description.
> >
> > Also i have my Semester End exams from 27 April - 7 may.
> > So maybe i would not be able to reply on time.
> >
>
> No issues. Do well for the exams :) Keep us updated.
>
> Looking forward to a great summer this year :)
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey,

So first thing.
(a.1) I am planning to install ubuntu on my pc in a day.
I guess We can install Macports on that.
(a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot too.

(b.1) I went and completed week 1.But i am unable to access week 3 , week
4.Because the course starts on April 30 i guess.
I went through the topics.
Week 3 : Subqueries and Join
Week 4 : Modifying and Analyzing Data with SQL

So now would learn more about these topics through youtube, many other
resources.

(b.2) Will look into them.
(b.3) Django hopefully wouldn't be a problem.

(c.0)  I would suggest that we create a new repository for your project
under
MacPorts, you fork it and use your own fork for endlessly playing with
it, testing etc
This sounds good.
I think i would be more comfortable with Github.

(c.2) Can be done.
(c.3) You are talking about Build History data ..right?
There is already a JSON API for it as you said..so cant we use it? (
*https://build.macports.org/json/help
<https://build.macports.org/json/help>* )

(d.1 , d.2) Yes will do the research .


(e.1) I think when i will be changing Buildbot to update the database after
every build ..Then i would need Tcl.
Will learn them as i get time.

Will see Project Description.

Also i have my Semester End exams from 27 April - 7 may.
So maybe i would not be able to reply on time.

Thanks
Vishnu







On 24 April 2018 at 03:37, Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> I have a bunch of suggestions written below. Community bonding should
> usually be used to getting familiar with our codebase, our tools, our
> communication channels, getting to know people, ...
>
> Since you are creating a standalone product, this community bonding
> could be slightly different (not so much need to learn our base code
> by heart), but it should warm you up and get you up to full speed by
> the time the coding period starts.
>
> The most important parts are:
> - getting MacPorts installed somewhere where you have regular access
> - creating a repository and finish/improve the planning, I would like
> to see the database design finalized, finish the sample HTML document,
> ...
> - creating tickets with milestones
>
> I hope others will be able to comment on my suggestions (they are not
> universal truths :)
>
>
>
> INSTALLING MACPORTS, BUILDBOT
>
> (a.1) We explained that owning a Mac was not a requirement for working
> on this project, but it would be orders of magnitude easier to work on
> some tasks if you had a regular access to MacPorts. This could either
> be:
>   - SSH access to a remote Mac or VM
>   - A Linux machine (even if Raspberry PI) or a VM with Linux
>
> You should start working on this as soon as possible because it would
> be a pity to waste precious time during the coding period. MacPorts
> can theoretically be installed on a UNIX machine, but there might be
> problems of one kind or the other. We can help you circumvent the
> problems, but you need to star working on this ASAP.
>
> Please let us know what your plan is.
>
> (a.2) With MacPorts you should be able to install mpstats and use it
> to submit statistics (to either the official server or your new server
> once ready).
>
> (a.3) It would probably be quite helpful if you manage to install
> buildbot and replicate our setup (at least to some extent). But (a.1)
> is a strong requirement for that, else it makes no sense.
>
>
> SQL & OTHER TOOLS
>
> (b.1) This is an absolute must:
> https://www.coursera.org/learn/sql-for-data-science#
> along with some further understanding of primary keys, indices,
> foreign keys, ...
>
> Let us know whether you managed to subscribe and how you are
> progressing, whether you have any further questions etc.
>
> (b.2) I found two (fully optional) courses on web design:
> - a very lightweight one with very clear explanations and adorable
> teacher, good for listening to while going for a walk :)
>   https://www.coursera.org/learn/responsivedesign/
> - a more comprehensive one about bootstrap 4:
>   https://www.coursera.org/learn/bootstrap-4/
> You might know these things already and even if not, you probably know
> enough to come up with some basic styles for what we need; we don't
> need perfect design anyway.
>
> (b.3) Django (which you should know from inside out :)
>
>
> PLANNING
>
> (c.0) Open for discussion/suggestions ...
> I would suggest that we create a new repository for your project under
> MacPorts, you fork it and use your own fork for endlessly playing with
> it, testing etc. (No need to fork though, you can just start
> somewhere.) Once you think that part of your code is ready for review,
> you submit a pull request, we review it and merge it. That way we
> w

Re: GSoC 2018: Congratulations to the selected student and mentors

[Vishnu]

Hello

I am overwhelmed by the results.
I would definitely keep in mind all the points mentioned by you.

Thanks

On Tue, Apr 24, 2018, 2:31 AM Umesh Singla <umeshksin...@macports.org>
wrote:

> Hello everyone,
>
> Google just announced 1,264 projects for this year's summer of code. I'm
> happy to share with you all that as requested MacPorts got one slot for the
> following project:
>
> 1. Vishnu M
> Project: Creating a dynamic website for Ports Index [1]
> Mentors: Mojca Miklavec and Jackson Isaac
>
> This goes without saying but I'm just reiterating:
>
> 1. I would like the student and the mentors to publicly interact with the
> community on the mailing list (preferably under a single thread) and IRC so
> that the rest of the community is able to follow the project and help you
> in case mentors go temporarily offline.
>
> 2. I would like the student and mentors to come up with a communication
> plan (how often do you want to meet, platform - mail/IRC/video) as soon as
> possible. This is really necessary so everyone has a clear idea from the
> beginning what to expect out of the project.
>
> This time until only the mid of May is meant for community bonding period,
> so learning any tools required or improving the project plan needs to be
> taken up before the official coding period begins.
>
> I hope others will add more to this email. I'll myself add more to this
> once my exams get over.
>
> Regards,
> Umesh
>
> [1]: https://summerofcode.withgoogle.com/projects/#6208860931489792
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Yes i wanted to get some feedback on the updated demo project.
And what should i do now?
Thanks

On Fri, Apr 6, 2018, 10:27 PM Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
>
> On Fri 6 Apr, 2018, 22:19 Vishnu, <vishnum1...@gmail.com> wrote:
>
>> Could i get a reply please.
>>
>
> Was there any question or query that I might have missed ?
>
>
> On Thu, Apr 5, 2018, 11:23 PM Vishnu <vishnum1...@gmail.com> wrote:
>>
>>> Hello
>>>
>>> Here is the git repository i made and am working on for demo project.
>>> https://github.com/VishnuM-sopho/demo-macports
>>>
>>> Thanks
>>>
>>> On 5 April 2018 at 21:31, Vishnu <vishnum1...@gmail.com> wrote:
>>>
>>>> Hello Jackson
>>>>
>>>>
>>>> I am sorry .
>>>> I was in a hurry to finish the demo as early as possible. So, I didnt yet
>>>> make a git rep for this folder so you cant see any history.
>>>> Right now we can only compare the files i sent earlier thrugh mails
>>>> earlier to see changes.
>>>>
>>>> Thanks
>>>>
>>>> On 5 April 2018 at 19:53, Jackson Isaac <ijack...@macports.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu 5 Apr, 2018, 19:38 Vishnu, <vishnum1...@gmail.com> wrote:
>>>>>
>>>>>> Dear mentors,
>>>>>>
>>>>>> I further improved my demo project based on feedback.
>>>>>>
>>>>>> I am now able to parse the ports(stored in json locally) using the
>>>>>> website itself(by accessing  {url}/port/parser).
>>>>>> This will read the json file and fill the port table using django
>>>>>> models.
>>>>>>
>>>>>> also upon adding the ports to the table we get to see how many ports
>>>>>> were added .
>>>>>>
>>>>>> Please go through the images attached in the link below in sequential
>>>>>> order.
>>>>>> And do provide some feedback.What else should i do?
>>>>>>
>>>>>>
>>>>>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>>>>>
>>>>>
>>>>> Could you please push the code on git so that we can track changes ?
>>>>>
>>>>
>>>>
>>>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Here is the git repository i made and am working on for demo project.
https://github.com/VishnuM-sopho/demo-macports

Thanks

On 5 April 2018 at 21:31, Vishnu <vishnum1...@gmail.com> wrote:

> Hello Jackson
>
>
> I am sorry .
> I was in a hurry to finish the demo as early as possible. So, I didnt yet
> make a git rep for this folder so you cant see any history.
> Right now we can only compare the files i sent earlier thrugh mails
> earlier to see changes.
>
> Thanks
>
> On 5 April 2018 at 19:53, Jackson Isaac <ijack...@macports.org> wrote:
>
>> Hi,
>>
>> On Thu 5 Apr, 2018, 19:38 Vishnu, <vishnum1...@gmail.com> wrote:
>>
>>> Dear mentors,
>>>
>>> I further improved my demo project based on feedback.
>>>
>>> I am now able to parse the ports(stored in json locally) using the
>>> website itself(by accessing  {url}/port/parser).
>>> This will read the json file and fill the port table using django models.
>>>
>>> also upon adding the ports to the table we get to see how many ports
>>> were added .
>>>
>>> Please go through the images attached in the link below in sequential
>>> order.
>>> And do provide some feedback.What else should i do?
>>>
>>>
>>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>>
>>
>> Could you please push the code on git so that we can track changes ?
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello Jackson


I am sorry .
I was in a hurry to finish the demo as early as possible. So, I didnt yet
make a git rep for this folder so you cant see any history.
Right now we can only compare the files i sent earlier thrugh mails earlier
to see changes.

Thanks

On 5 April 2018 at 19:53, Jackson Isaac <ijack...@macports.org> wrote:

> Hi,
>
> On Thu 5 Apr, 2018, 19:38 Vishnu, <vishnum1...@gmail.com> wrote:
>
>> Dear mentors,
>>
>> I further improved my demo project based on feedback.
>>
>> I am now able to parse the ports(stored in json locally) using the
>> website itself(by accessing  {url}/port/parser).
>> This will read the json file and fill the port table using django models.
>>
>> also upon adding the ports to the table we get to see how many ports were
>> added .
>>
>> Please go through the images attached in the link below in sequential
>> order.
>> And do provide some feedback.What else should i do?
>>
>>
>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>
>
> Could you please push the code on git so that we can track changes ?
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello Jackson


I am sorry .
I was in a hurry to finish the demo as early as possible. So, I didnt yet
make a git rep for this folder so you cant see any history.
Right now we can only compare the files i sent earlier through mails
earlier to see changes.

Will be making one now.So that the changes can be tracked.

Thanks



On 5 April 2018 at 19:53, Jackson Isaac <ijack...@macports.org> wrote:

> Hi,
>
> On Thu 5 Apr, 2018, 19:38 Vishnu, <vishnum1...@gmail.com> wrote:
>
>> Dear mentors,
>>
>> I further improved my demo project based on feedback.
>>
>> I am now able to parse the ports(stored in json locally) using the
>> website itself(by accessing  {url}/port/parser).
>> This will read the json file and fill the port table using django models.
>>
>> also upon adding the ports to the table we get to see how many ports were
>> added .
>>
>> Please go through the images attached in the link below in sequential
>> order.
>> And do provide some feedback.What else should i do?
>>
>>
>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>
>
> Could you please push the code on git so that we can track changes ?
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Dear mentors,

I further improved my demo project based on feedback.

I am now able to parse the ports(stored in json locally) using the website
itself(by accessing  {url}/port/parser).
This will read the json file and fill the port table using django models.

also upon adding the ports to the table we get to see how many ports were
added .

Please go through the images attached in the link below in sequential order.
And do provide some feedback.What else should i do?


https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS

Thanks
Vishnu

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

I wanted to get feedback on the demo project i submitted

https://drive.google.com/open?id=1LbMnyHjaRCbg7IrAUMM042eD7O2Br-jU


Thanks


On Wed, Apr 4, 2018, 11:08 AM Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> On 3 April 2018 at 23:54, Vishnu wrote:
> >
> > Hello
> >
> > I have finished the mini project given to me.
> > I am now successfully able to scrape data from json into the database.
> > And have integrated the database and website to display all the ports
> and their basic information as it was required
> > All the images and files are attached.
> > Please go through the link.
> > https://drive.google.com/open?id=1LbMnyHjaRCbg7IrAUMM042eD7O2Br-jU
>
> Thank you very much.
>
> As far as I am concerned this serves ok as a demo.
> (The database would ideally be filled with Django as well, but that's
> ok for now.)
> I don't know if other mentors have any further questions.
>
> > On 4 April 2018 at 03:19, Vishnu wrote:
> >>
> >> Hello
> >>
> >> I have finished the mini project given to me.
> >> I am successfully able to scrape data from json into the database.
> >> And have integrated the database and website to display all the ports
> and their basic information as it was required
> >> All the images and files are attached.
> >>
> >> My last 3 mails got rejected due to large size.
> >> Hence uploaded all the images as attachment..
>
> You can see which emails reached the mailing list by looking at
>
> https://lists.macports.org/pipermail/macports-dev/2018-April/thread.html
>
> I'm not absolutely sure, but I think this email did not reach others
> either because the images were still present inline (in quoted
> messages) when you hit the "reply to" button and thus the email size
> was still prohibitively large.
>
> In the future try not to send anything but trivial/small text
> attachments (or super small images like a logotype), any screenshots
> should better go to some other place where only those interested in
> the topic will go to check them out, while hundreds of other
> subscribers to the mailing list would not get their mailboxes filled.
>
> Mojca
>
>
>
> >> On 4 April 2018 at 03:06, Vishnu <vishnum1...@gmail.com> wrote:
> >>>
> >>> In the last mail it was working but by taking portid from url.
> >>>
> >>> Now i created my custom view which will take the text after "port/"
> and search it in port table and return all its properties.
> >>>
> >>> Please look at the url.This is what our final result would want.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On 4 April 2018 at 02:06, Vishnu <vishnum1...@gmail.com> wrote:
> >>>>
> >>>> Hello
> >>>>
> >>>> I have succesfully integrated django & sqlite3 .
> >>>> Also am able to display the content on web.
> >>>> Following Screenshots will explain :
> >>>>
> >>>>
> >>>>
> >>>> Databse :
> >>>>
> >>>>
> >>>>
> >>>> Is my progress fine?
> >>>>
> >>>> I think this would be enough as my proof of skill.
> >>>>
> >>>> Thanks
> >>>>
> >>>>
> >>>>
> >>>> On 4 April 2018 at 01:12, Vishnu <vishnum1...@gmail.com> wrote:
> >>>>>
> >>>>> Hello
> >>>>>
> >>>>>
> >>>>> Yes that's what I have been doing.
> >>>>> Its almost ready.. Now integrating it with my front-end website.so
> that table is updated automatically.
> >>>>> Will be mailing you very soon. All the screenshots.
> >>>>>
> >>>>> On Wed, Apr 4, 2018, 1:05 AM Mojca Miklavec <mo...@macports.org>
> wrote:
> >>>>>>
> >>>>>> Dear Vishnu,
> >>>>>>
> >>>>>> On 3 April 2018 at 20:10, Vishnu wrote:
> >>>>>> >
> >>>>>> > Hello
> >>>>>> >
> >>>>>> > Yes portindex.json is very useful.
> >>>>>> >
> >>>>>> > I tried my hands with python and sql.
> >>>>>> > And am successfully able to parse the json and store the data
> into the table .
> >>>>>>
> >>>>>> Thank you very much. The initial steps look promising.
> >>>>>>
> >>>>>> Now, try to integrate this into a simple django app with the
> database
> >>>>>> abstraction layer. A simple tutorial is here to demonstrate how to
> >>>>>> define simple python classes that automatically interoperate with
> >>>>>> database (so no need to write direct sql code at this step):
> >>>>>> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
> >>>>>>
> >>>>>> Using SQLite is fine (for the production we would probably want to
> go
> >>>>>> for PostgreSQL, but now it doesn't make any difference).
> >>>>>>
> >>>>>> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello


Yes that's what I have been doing.
Its almost ready.. Now integrating it with my front-end website.so that
table is updated automatically.
Will be mailing you very soon. All the screenshots.

On Wed, Apr 4, 2018, 1:05 AM Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> On 3 April 2018 at 20:10, Vishnu wrote:
> >
> > Hello
> >
> > Yes portindex.json is very useful.
> >
> > I tried my hands with python and sql.
> > And am successfully able to parse the json and store the data into the
> table .
>
> Thank you very much. The initial steps look promising.
>
> Now, try to integrate this into a simple django app with the database
> abstraction layer. A simple tutorial is here to demonstrate how to
> define simple python classes that automatically interoperate with
> database (so no need to write direct sql code at this step):
> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
>
> Using SQLite is fine (for the production we would probably want to go
> for PostgreSQL, but now it doesn't make any difference).
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello,

First thing. I am. Not clear about what document you are talking about
perfecting.

And i will now focus attention on django as mentioned in my earlier mails.

And thanks a lot For that in depth reply. Will work on all things related
to Html after I complete working with django.

Thanks

On Tue, Apr 3, 2018, 9:04 PM Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> Thank you very much for sharing the document. The purpose of this HTML
> was two-fold:
> - demonstrating your skills
> - first step of the planning phase for the actual implementation
>
> Below I'm providing some feedback, but I would suggest to concentrate
> on a simple django app at this moment and then return back to this
> html once you are "done" with Django to address (some of) the
> comments. In short: if selected, I'll insist to make this document
> "perfect" before proceeding (and to address all the feedback + more I
> didn't yet bother writing), but there's no point in asking you to
> spend a week making this document ten times longer and fixing tiny
> unimportant details that don't really demonstrate the skillset :)
>
> On 2 April 2018 at 23:32, Mojca Miklavec wrote:
> > V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:
> >>
> >> In the database.
> >> Because then it would be very easy to count the number of os for that
> >> port.
> >
> > I'll explain tomorrow why this is suboptimal. (But there's no need to
> > further optimise the database design right now.)
>
> There are probably better resources that explain this, but here's the
> first hit from Google:
>
> https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
> https://en.wikipedia.org/wiki/Database_normalization
>
> In extreme case, imagine that we decide to send a questionnaire to our
> participants of statistics collection, asking them some 100 optional
> questions, including anything from gender, age, country of origin,
> country of current residence, education, favourite animal, ... Then we
> decide that we would want to compare the age distribution of users of
> package A vs. age distribution of users of package B.
>
> Your idea that allows "very easy number counting" would mean that:
>
> At the moment you only have (submission id, port, port version,
> variants) in the table. You would need to extend the table to contain
> (submission id, submission time, user id, port, port version,
> variants, os version, stdlib, xcode version, age, gender, country,
> education, favourite animal, ...)
> And if the user has 1000 ports installed, you would need to store
> 100x1000 cells (repeat that same information one thousand times and
> then again in any subsequent submission from the same user) instead of
> having a single copy in a separate "questionnaire" table. Multiply
> that with 10.000 users submitting statistics and you end up with tens
> of gigabytes of data each month, just to store results of that
> one-time questionnaire.
>
> On top of that, once the user submits a questionnaire, if you keep
> those answers in a separate table and use proper SQL queries, you
> could easily get the answer to question "what was the prevailing
> gender of users of package A" even for submissions that were made many
> months ago. If you store everything into a single monstrous table, you
> would either need to modify plenty of old submissions or you would not
> be able to get that information for old submissions at all.
>
> Additionally, it could happen that while you are updating old
> submissions, the database crashes. You could end up with half of the
> entries updated and the other half left at their old value, in
> inconsistent state. There are plenty of problems if you don't make
> sure that you keep your database design in a good shape from the very
> beginning.
>
>
> That's a super common use case in databases that has already been
> solved. One should use table joins and views. Random link (I'm sure
> there are better ones):
> https://db.grussell.org/sql3.html
>
> I don't know how Django handles joins and views (some hints I skimmed
> through are here https://stackoverflow.com/a/1281051/585897), but one
> should certainly make sure that the database design is done well.
> Learning more about that topic is part of the process.
>
>
> On 2 April 2018 at 23:50, Vishnu wrote:
> >
> > Please go through this https://jsfiddle.net/vishnum98/3r4vL4L3/21/
> >
> > I did some changes.
>
> Thank you very much. The chart looks ok. For the remaining (missing)
> charts just add a section (and optionally an empty box) and describe
> what kind of chart goes there (no need for a long paragraph, just make

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Sure thanks a lot for the prompt reply.
Will start working on the idea and produce results very quickly.

On Tue, Apr 3, 2018, 8:32 PM Jackson Isaac <ijack...@macports.org> wrote:

> Hi Vishnu,
>
> On Tue 3 Apr, 2018, 20:26 Vishnu, <vishnum1...@gmail.com> wrote:
>
>> Regarding mini django project. What would you suggest i do?
>>
>> I was thinking on code to populate the port table from portindex.json
>>
>
> Please go ahead and implement your idea :)
>
>
>> Would that act as a sample for my Coding skill?
>> Or should i do something else?
>>
>
> For me, it should be enough to get a gist of your skills and thought
> process. We can probably add some mini tasks on the fly, if required.
>
> Feel free to reach out to us in case you have any queries.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Regarding mini django project. What would you suggest i do?

I was thinking on code to populate the port table from portindex.json

Would that act as a sample for my Coding skill?
Or should i do something else?

On Tue, Apr 3, 2018, 8:03 PM Mojca Miklavec <mo...@macports.org> wrote:

> On 3 April 2018 at 15:55, Vishnu <vishnum1...@gmail.com> wrote:
> > Hello
> >
> >
> > Is the graph fine?
>
> Yes, the one graph is ok. I'm writing the feedback right now (I'm just
> back from work) with further suggestions.
>
> But you should probably start working on the django part and I'll send
> you feedback about the html site in the meantime.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Please go through this https://jsfiddle.net/vishnum98/3r4vL4L3/21/

I did some changes.

Thanks

On 3 April 2018 at 03:02, Mojca Miklavec <mo...@macports.org> wrote:

>
> V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:
>
>> In the database.
>> Because then it would be very easy to count the number of os for that
>> port.
>>
>
> I'll explain tomorrow why this is suboptimal. (But there's no need to
> further optimise the database design right now.)
>
> Just plan the final result now rather than how the database queries should
> look like.
>
> Mojca
>
>>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

In the database.
Because then it would be very easy to count the number of os for that port.

Ok.
would be updating the html.


On 2 April 2018 at 22:54, Mojca Miklavec <mo...@macports.org> wrote:

> On 2 April 2018 at 18:50, Vishnu wrote:
> > Hi
> >
> > Here is the basic structure i prepared.
> >
> > https://jsfiddle.net/3r4vL4L3/5/
> >
> > And i wanted to know what all am i supposed to show in installation
> > statistics mandatorily?
> >
> > Top port Versions of a month?
> > Top os of that port?
> > Number of installations versus time ? (of that particular port)
> >
> > what else?
>
> Just get an inspiration from
> http://stats.macports.neverpanic.de/installed_ports
> https://trac.macports.org/wiki/StatisticsIdeas
> or from one particular port (which is what your port page should most
> likely show):
> http://stats.macports.neverpanic.de/categories/8/ports/587
>
> With those few exceptions:
> - The "All versions" cumulated since the beginning of statistics
> collection is pretty useless.
> - The "All variants" is somewhat controversial (I'm not even sure what
> to do with it), but if, then it should be shown as a line chart rather
> than pie chart
> - I would add OS version and build architecture like you suggested.
>
> I would vote for line charts with time on the x axis for nearly everything.
>
> (Of course we'll need another page for statistics, like OS version
> distribution etc that doesn't depend on individual ports, but let's
> not bother about that now.)
>
> > Also in the port history table. Should we add another coloumn named os .
> > which will store the current os version??
>
> Database table or the table on the website?
>
> If you were asking about the table on the website with build summary,
> then I would just replace builder name in that table with OS version +
> arch (10.5/ppc, 10.6/i386, 10.6/x86_64, ...). I would slightly change
> the wording in "Reason", but that's nitpicking. (I could add some
> other minor changes on the list, but they don't feel so important
> right now.)
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Here is the basic structure i prepared.

https://jsfiddle.net/3r4vL4L3/5/

And i wanted to know what all am i supposed to show in installation
statistics mandatorily?

Top port Versions of a month?
Top os of that port?
Number of installations versus time ? (of that particular port)

what else?

Also in the port history table. Should we add another coloumn named os
. which will store the current os version??

Databse Spreadsheet :
*https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing>*

Thanks

On 2 April 2018 at 19:44, Vishnu <vishnum1...@gmail.com> wrote:

> Hello,
>
> I would be getting back on the project from now.
> Sorry i could not deliver soon because i was having my exams going on.
> Now i would deliver my best.
>
> Thanks
>
> On 2 April 2018 at 00:36, Mojca Miklavec <mo...@macports.org> wrote:
>
>> On 1 April 2018 at 20:33, Vishnu <vishnum1...@gmail.com> wrote:
>> > Hello mojca
>> >
>> > I am planning on submitting the simple static form of the webpage(for
>> port).
>> >
>> > Had some doubts for the same. Is css required for that? As of now while
>> > showing you the structure?
>> >
>> > Right now its in plain html.
>> > That's  why.
>>
>> The very first step is to have a sound understanding and clear plan
>> for the content, there's no need to prepare css for that.
>>
>> Then we expect some proof of coding skills to build a simple django app.
>>
>> If you have time left, it would be nice to see a nicely looking html,
>> but you should concentrate on content for now and only add design at
>> the end.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello,

I would be getting back on the project from now.
Sorry i could not deliver soon because i was having my exams going on.
Now i would deliver my best.

Thanks

On 2 April 2018 at 00:36, Mojca Miklavec <mo...@macports.org> wrote:

> On 1 April 2018 at 20:33, Vishnu <vishnum1...@gmail.com> wrote:
> > Hello mojca
> >
> > I am planning on submitting the simple static form of the webpage(for
> port).
> >
> > Had some doubts for the same. Is css required for that? As of now while
> > showing you the structure?
> >
> > Right now its in plain html.
> > That's  why.
>
> The very first step is to have a sound understanding and clear plan
> for the content, there's no need to prepare css for that.
>
> Then we expect some proof of coding skills to build a simple django app.
>
> If you have time left, it would be nice to see a nicely looking html,
> but you should concentrate on content for now and only add design at
> the end.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

The database i am planning to make.
https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing

Please go through it.

On 26 March 2018 at 18:37, Mojca Miklavec <mo...@macports.org> wrote:

> On 26 March 2018 at 14:47, Vishnu wrote:
> > Ok .
> > I will be writing Milestones seperately very soon.
> > What is your irc id? and on what channel are you available.i use
> freenode.
>
> You can join #macports
>
> > Also rather than irc could we use comments in docs.google to interact.And
> > simultaneously you can see the changes.That would quicken the process.
>
> The problem is that I don't get any feedback when you add a comment.
> So unless I stare in the document all the time ...
>
> > I have a good prior expeirence with sql.I had planned to use that
> instead of
> > log file.This idea had striked my mind to create table.As it would really
> > simplfy the statistics section.
> > But creating tables for so many ports would be good or not .is something
> i
> > cant decide.
>
> Databases are designed to handle terabytes of data in an efficient
> way. It's creating files for so many ports that should be questioned,
> not the other way around.
>
> > I can learn django in no time.As i mentioned i am a quick learner.
> > So i guess it would be wise to make everything in django.
> >
> > I will try to create the Static Website .If possible will submit that
> before
> > the deadline.
>
> That would be great. But again: it's not a big problem if you create a
> website a bit later. But if you fail to submit all the paperwork
> (proposal + enrolment proof) properly by tomorrow, you would not be
> eligible to participate at all.
>
> > I wanted to know if we can chat somewhere.
>
> Just join the #macports channel.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Ok .
I will be writing Milestones seperately very soon.
What is your irc id? and on what channel are you available.i use freenode.
Also rather than irc could we use comments in docs.google to interact.And
simultaneously you can see the changes.That would quicken the process.


I have a good prior expeirence with sql.I had planned to use that instead
of log file.This idea had striked my mind to create table.As it would
really simplfy the statistics section.
But creating tables for so many ports would be good or not .is something i
cant decide.

I can learn django in no time.As i mentioned i am a quick learner.
So i guess it would be wise to make everything in django.

I will try to create the Static Website .If possible will submit that
before the deadline.


I wanted to know if we can chat somewhere.

Thanks
Vishnu

On 26 March 2018 at 17:23, Mojca Miklavec <mo...@macports.org> wrote:

> Dear Vishnu,
>
> First some general remarks, then the answers to your questions below.
>
> - One of the most important things that we should discuss and improve
> is the timeline. This is very little work, but it requires some
> coordination to make sure that they are defined in the most sensible
> way. One option is to discuss over IRC, but email is also fine. I'll
> write about that separately.
>
> - I suggest to write the milestones in a separate section. For example
> something along those lines:
>   May X: The first version of database design is ready, finish the
> first import of portindex.
>   May Y: The prototype website gets deployed at a temporary location
> xyz.com. It is possible to show the most basic properties of any port
> "foo" on a plain html page via xyz.com/index/port/foo and list all
> ports via xyz.com/index/list
>   ...
>   June Z: The website is accepting installation statistics submissions
> from users.
>
> - At the moment your schedule says "June 11-15: Phase 1 Evaluation"
> and no plans to code during those days and no explicit criteria that
> would define whether the project passed or not. It makes sense to put
> evaluation on the schedule, so that you see where it is, but note that
> this doesn't take any of your time, that's when mentors are busy and
> that's when you need to make sure to show results that were agreed on.
> (It can happen that there are valid reasons why the milestones would
> not be met, for example if the plan gets changed during the coding
> since a better solution or idea is found etc., so the milestones are
> not set in stone, but in case of disagreement between a student and
> mentor it helps a lot if it's straightforward to check for outsiders
> or admin as well.)
> I didn't check if those days are weekend, but don't make it sound like
> you would be spending all your time on evaluation. You only need to
> fill in a simple webform some time during the evaluation period,
> that's all.
>
> - Please do take a look at how relational databases work and get
> familiar with some basic Object-oriented programming in case you don't
> know those concepts already. This is of crucial importance for success
> of the project. In case you would work in Django (this is not the only
> option, there are other possibilities as well), you should be familiar
> with
> https://docs.djangoproject.com/en/2.0/intro/tutorial02/
> (this is part of a slightly longer tutorial). We need a single uniform
> website rather than something scraped from random chunks of scripts
> which do individual tasks without knowing about each other.
>
> - After the proposal gets submitted (because that's the highest
> priority task at the moment), we would still want to verify your
> skills with a simple coding challenge before we request slots (so
> ideally the challenge should be complete cca. one week after the
> deadline for proposal submission). Other organisations would ask for a
> pull request, but in this case it probably makes more sense to
> demonstrate a simple prototype. You may pick a slightly different
> task, but it should demonstrate comparable skillset or ability to
> learn. My suggestion would be to do these two tasks:
> (a) Create a simple (static) website for one single port of your
> choice and include nearly everything that will be part of the final
> product. You can copy some data from a Portfile to help you, but you
> can just as well make up the information, statistics etc. I would
> actually consider this to be part of the proposal. The page should
> demonstrate how the final product will look like (only with fake data)
> and will also help you while developing the actual functionality of
> the website: you would know where is it that you are heading. This
> should not take you more than a few hours to do, you basically need to
> visualiz

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Also i wanted to know.Like if a new port is created .Buildbot is
compulsorily run right?
and if it is deleted.There is no need for buildbot to run.So buildbot wont
send updates to the log that port has been deleted.
In that case i want to know how / where is the port deleted from.What code ?
Because i would have to do changes there.That as soon as port is deleted
log of that port is updated.

" Buildbot master and this website will live on a different server. How
exactly do you plan to update the log file? "
In that case. I would need to make a mediator(present on same server as
that of website) of some sort. Wherein requests would be sent to that
mediator. And that mediator would inturn write in the log files.

or i think that would be hectic.
It would be best to have the logs on the Buildbot master side.Because every
time buildbot is used it can easily update the log at that time only.No
need for continuously sending stuff to mediator as i previously planned
above.
As at max the logs(history) required by the web app is 20 logs.It can make
a request to the backend to get them.

I think this would be better.

Also is backend and your website lives on different servers ?
How does the interaction happen ?

On 26 March 2018 at 15:09, Vishnu <vishnum1...@gmail.com> wrote:

> " I just wanted to say that in the long term it's more important to make
> sure that new builds are stored soon after they finish, but it's
> probably much easier to start with historical data (and it's also
> super useful in the short term), so that you have immediately
> something to work with. "
>
> yes i definitely agree with you.It would be best to first work upon
> history.That will allow me to understand the code more.
>
> Can you help me with where does that json api work from?where is the code
> that sends request to the backend.To get that data.
> Once i get to know that it will simplify my job to send request to backend
> for port
>
> Also is there any existing method to show all ports of a maintainer?
> or to show all ports of a category?say python?
>
>
> It would be best if i start working gmy gsoc with 3 most important things .
> 1) Basic Port information through portindex2json
> 2) Port Build history. It would use JSON API
> 3) Buildbot code updation to send updates to the logs.
>
>
> And as you said logs would be ineffective to store .For history of builds
> of the port.I think logfile would be the bestway to store history.
> And regarding the syntax . I was thinking of storing the build history in
> json format in one line in log.txt of a particular port.
>
> Also mojca can i know your working hours.or atleast the timezone where you
> live.
> I will try to do most of the work on proposal during that time.Because
> only one day is left and a lot to change.
>
> Thanks
> Vishnu
>
>
>
>
>
> On 26 March 2018 at 05:46, Mojca Miklavec <mo...@macports.org> wrote:
>
>> Dear Vishnu,
>>
>> On Mon, Mar 26, 2018 at 2:58 AM, Vishnu wrote:
>> >
>> > Hi,
>> >
>> > I wanted some help in build bot summary.
>> > I am not sure i completely understand what's happening here.
>> > Could you please explain me about buildbot summary more?
>> >
>> > what i understand is :
>> > There can be n number of builds for each port.
>> > So how am i supposed to get the information of what all builds have
>> > already been done for that particular port.Where can i find that history
>> > from? from build.macports.org ?
>>
>> Yes, that information can be obtained from build.macports.org.
>>
>> > First and foremost I think first i have to implement the Buildbot idea
>> > .That would create logs.
>> > But for that idea..how to get all the exiting build history of
>> ports?Build
>> > history such as timestamp,os,Succesful or not?
>> > Where to get exclusive information about a port.
>>
>> You cannot get that at the moment.
>> That's what the whole idea is about: to implement the functionality
>> that will let developers query a single port.
>>
>> Take a look at
>> https://build.macports.org/builders/ports-10.13_x86_64-builder
>>
>> You'll see a list of builds. The one currently being built is
>> https://build.macports.org/builders/ports-10.13_x86_64-build
>> er/builds/21976
>> for the port "sumo".
>>
>> The task is then to go back to the history by decreasing that number
>> (21976) one by one and fetching information from each individual build
>> until you reach number 1 or the first build where the information is
>> no longer available. I believe the build logs get deleted after cca.
>&g

Re: Gsoc

[Vishnu]

Thanks.. I will work upon the feedback.


On Mon, Mar 26, 2018, 2:10 AM Umesh Singla <umeshksin...@macports.org>
wrote:

> Hi Vishnu,
>
> I think the designated mentor to your project Mojca already gave you a
> feedback on the previous mailing thread and I don't see any improvement in
> the proposal. Here is the link to email if you missed it -
> https://lists.macports.org/pipermail/macports-dev/2018-March/037885.html.
>
> Thanks,
> Umesh
>
> On Mon, Mar 26, 2018 at 2:01 AM, Vishnu <vishnum1...@gmail.com> wrote:
>
>> Can i please get a feedback for my proposal.
>>
>> Thanks
>>
>
>

Gsoc

[Vishnu]

Can i please get a feedback for my proposal.

Thanks

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Can someone from MacPorts help me with my proposal?
There's not much time left.
I really need some suggestion/improvements in the proposal.

On 25 March 2018 at 14:59, Vishnu <vishnum1...@gmail.com> wrote:

> Hi Mojca,
>
> I have submitted a draft proposal through GSOC portal.
> Please go through it and do review and suggest changes.
>
> Thanks
> Vishnu
>
> On 25 March 2018 at 12:59, Mojca Miklavec <mo...@macports.org> wrote:
>
>> On 25 March 2018 at 00:42, Vishnu wrote:
>> > If i take up the project of improving the buildbot or the website.
>> > would it be considered a different project? Or we both still have to
>> > compete?
>>
>> Both :)
>>
>> > The other student could work on Option B and i could work on Option A
>> > Because those also sounded interesting.
>>
>> First of all, any projects will be first and foremost evaluated to see
>> whether the quality of proposal & and the "proof of competence" of the
>> student is above the threshold. As Google said, "only OK" proposals
>> should usually not be accepted, they need to be well above average. In
>> addition to that, even if we get two excellent proposals we may only
>> get one slot and then students will "compete" for that single slot no
>> matter whether the projects are different or not.
>>
>> But if more than one proposal for the same project meets the quality
>> standard and if we had enough slots to cover both, it's a complete
>> waste of resources if two people start coding exactly the same thing
>> and then we need to throw one of the two works away after the end of
>> the summer.
>>
>> If one works on the buildbot side and the other one on the website,
>> that's OK because the outcome would be a different code serving a
>> different purpose (even if we have an excellent website with build
>> statistics, we still hesitate to switch to buildbot 1.x just because
>> nobody in the team is currently willing to invest time into learning
>> some new javascript frameworks and figure out how to implement the new
>> views that we would need; and the code could also be useful as a demo
>> to other groups who hesitate to switch to 1.1).
>>
>> If you want to know more about the buildbot idea, you should ask Piere
>> (CC-ed in my initial reply) and CC this list.
>>
>> To answer the rest of you question:
>>
>> I don't mind if one document describes two potential project, but
>> describing one project well is already time consuming (and keep in
>> mind that you are running out of time). If you end up writing two bad
>> proposals instead of one excellent, you are doing yourself a
>> disservice. (Writing one excellent project and drafting an alternative
>> should be ok.)
>>
>> In any case don't wait for the last day to submit your draft because
>> you won't be able to get any feedback and even if you do get feedback,
>> you won't have time to improve it later.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi Mojca,

I have submitted a draft proposal through GSOC portal.
Please go through it and do review and suggest changes.

Thanks
Vishnu

On 25 March 2018 at 12:59, Mojca Miklavec <mo...@macports.org> wrote:

> On 25 March 2018 at 00:42, Vishnu wrote:
> > If i take up the project of improving the buildbot or the website.
> > would it be considered a different project? Or we both still have to
> > compete?
>
> Both :)
>
> > The other student could work on Option B and i could work on Option A
> > Because those also sounded interesting.
>
> First of all, any projects will be first and foremost evaluated to see
> whether the quality of proposal & and the "proof of competence" of the
> student is above the threshold. As Google said, "only OK" proposals
> should usually not be accepted, they need to be well above average. In
> addition to that, even if we get two excellent proposals we may only
> get one slot and then students will "compete" for that single slot no
> matter whether the projects are different or not.
>
> But if more than one proposal for the same project meets the quality
> standard and if we had enough slots to cover both, it's a complete
> waste of resources if two people start coding exactly the same thing
> and then we need to throw one of the two works away after the end of
> the summer.
>
> If one works on the buildbot side and the other one on the website,
> that's OK because the outcome would be a different code serving a
> different purpose (even if we have an excellent website with build
> statistics, we still hesitate to switch to buildbot 1.x just because
> nobody in the team is currently willing to invest time into learning
> some new javascript frameworks and figure out how to implement the new
> views that we would need; and the code could also be useful as a demo
> to other groups who hesitate to switch to 1.1).
>
> If you want to know more about the buildbot idea, you should ask Piere
> (CC-ed in my initial reply) and CC this list.
>
> To answer the rest of you question:
>
> I don't mind if one document describes two potential project, but
> describing one project well is already time consuming (and keep in
> mind that you are running out of time). If you end up writing two bad
> proposals instead of one excellent, you are doing yourself a
> disservice. (Writing one excellent project and drafting an alternative
> should be ok.)
>
> In any case don't wait for the last day to submit your draft because
> you won't be able to get any feedback and even if you do get feedback,
> you won't have time to improve it later.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Thanks Mojca for your prompt reply.
1) description, version,maintainer, homepage, variants, dependencies,dependent
ports,etc
So for information of each port, my website has to scrape data from such
pages :

https://github.com/macports/macports-ports/blob/master/python/py-amqplib/Portfile

is this correct?

2) Build Summary:

Would scraping buildbot history pose a problem?

and "  modify the buildbot setup to submit data to the server. "
isn't it aldready submitting data to the server?
if not how is the buildbot history currently generated?


On 25 March 2018 at 04:01, Mojca Miklavec <mo...@macports.org> wrote:

> On 24 March 2018 at 22:50, Vishnu wrote:
> > Well that's really nice.
> > I'll be emailing you the proposal very soon.
> > I would really like to work on creating dynamic websites for each port
> with
> > one framework.
> > I think that is what you are also working on currently.Or is it something
> > else?(Umesh Sir)
> >
> > I would like to go with option B.
> >
> > What i understand is .Basically, i will be creating a dynamic
> > website/framework (use some scripting language) to generate/gather all
> the
> > following information for any port that is requested.
> >
> >
> > - all the information about the port (description, version,
> > maintainer, homepage, variants, dependencies, dependent ports, ...)
> > - build summary (as scraped from buildbot history): which version was
> > built at what time on which os, successful or not, ...
> >   - including whether or not a binary package exists
> > - installation statistics (from users who opted in and send a json
> > file every now and then) with some graphs, similar to the stats page
> > above
> > - results of "livecheck": whether there's a newer version of the port
> > available
> > - short git log with links to the last few changes to the Portfile
> > - links to trac tickets for that port
> >
> > https://trac.macports.org/query?0_port=python27&0_port_
> mode=%7E&0_status=%21closed
> > - (maybe links to pull requests for that port)
> >
> > Is my understanding of the project correct?
> >
> > Now to generate:
> > 1) description, version,maintainer, homepage, variants, dependencies,
> > dependent ports,etc:
> > I will need to update this script
> > https://github.com/g5pw/macports-port-tree made by Aljaz for the
> remaining
> > missing information.because not all information is produced by this tool.
>
> This source of information PortIndex which can be generated from
> https://github.com/macports/macports-ports (that's part of MacPorts
> installation process, it should be possible to get it under linux as
> well, but I didn't try, we also mirror it).
>
> PortIndex is some data structure in Tcl which can be converted into
> JSON with an existing script postindex2json, but ideally the Tcl
> script should be fixed to generate data in a better format.
> This data would then need to be imported at the beginning, but the
> application should keep accepting updates about new or deleted ports.
>
> > 2) Build Summary: This can be generated by scraping through buildbot
> > history.
>
> For historic data. For the future it would be better to modify the
> buildbot setup to submit data to the server.
>
> > 3) Installation Statistics: Take help from Stats web app(Gsoc 2011
> project)
> >
> > 4)Results of livecheck: I am not sure how to get that. i'll be needing
> help
> > to how to get this.
>
> Sample output from the package manager:
>
> > port -v livecheck *
> Bear seems to be up to date
> CImg seems to have been updated (port version: 1.7.9, new version: 2.2.1)
> CoreOSMakefiles seems to be up to date
> Error: cannot check if Djinn was updated (regex didn't match)
> Drizzle seems to have been updated (port version: 30, new version: 31)
> FlowUI seems to be up to date
> GASNet seems to have been updated (port version: 1.26.3, new version:
> 1.30.0)
> GitX seems to be up to date
> HaXml seems to have been updated (port version: 1.20.2, new version:
> 1.23.3)
> Error: livecheck failed for LibVNCServer: extracted version '0.4' is
> older than livecheck.version '0.9.9'
> LucenePlusPlus seems to be up to date
> Error: livecheck failed for MongoObjCDriver: extracted version 'test2'
> is older than livecheck.version '1.0.6'
> Error: cannot check if PlasmaShop was updated (regex didn't match)
> ResIL seems to have been updated (port version: 1.7.9, new version: 1.8.3)
> Error: cannot check if SilverCity was updated (regex didn't match)
> SourceKitten seems to have been updated (port version: 0.17.6, new
> version: 0.19.1)
> Error: cannot check if VillainousStyle was updated (regex didn't match)
> aap seems to be up to date
>
> > 5) Link to track tickets : Can be done using scripting,html.
> > 6)Pull request : Can be done.
>
> Mojca
>

Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello all,

Thanks for rectifying the mail issues.

This is Vishnu.
I would Like to participate in GSOC 18 .And would like to work on :
#Collect build statistics
I needed some help in understanding the idea a bit more.
i wanted to understand the idea more deeply.
Can someone elaborate the idea more?

How many web pages do we need to make ?
Can i get some sample structure of webpage for 1 port . what all need to be
there.
Also where to get : collect per-port statistics & success matrix .Can i get
the exact sample link?




I really think this is the project i could work on because i have prior
experience with all the languages required for this project (HTML, Python,
javascript, in fact, did my last intern mainly involving JSON).
I also have good experience in web Scraping.

I am well aware about :
JSON
Html
Css
Python
JAVASCRIPT
Basic idea about perl

Have used macOS.
Access to macos during gsoc would be limited. But possible.
I have certain knowledge of macports too.


Thanks
Vishnu