Re: Gsoc 18 Project | Collect build statistics

[Bradley Giesbrecht]

>>> On 9 May 2018 at 04:06, Vishnu  wrote:
 
 Hi
 
 And also i couldn't figure out any way to hide passwords/ Sensitive
 information while creating app.
> 
> Here's one way:
>https://ultimatedjango.com/learn-django/lessons/handling-sensitive-keys/
> 
> Of course you need that information on the server where you are
> running the application, but the secrets and passwords should not be
> stored in a public repository.
> What I often do is create something like
>settings.py.sample
> or perhaps just
>secrets.py.sample
> and commit that one with a fake password to repository. Then, whoever
> wants to run the app, should first copy the file (removing the .sample
> extension), enter the correct secret data and only then run the app.
> 
> Again: you do need to have this information stored somewhere, it just
> may not leak to a public repository. If you commit settings.py with
> fake passwords and correct the password on that one file, you might
> accidentally commit the change one day, so it's better to have a
> separate file.

I didn’t see it mentioned so I’ll suggest adding secrets.py to .gitignore.

For some Rails apps we used dotenv and I see they have it for python.

https://github.com/theskumar/python-dotenv

Same here, add .env to .gitignore.


Regards,
Bradley Giesbrecht (pixilla)

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 15 May 2018 at 05:46, Vishnu wrote:
> Hi
>
> Yes I was laos thinking about it for some time.
>
> Please someone from infra team create another repository.

This would also remove all the tickets etc., so I don't think it is a
super good idea.

Git is flexible enough to allow force pushing to a branch. We should do that.
I tried to squash some commits, but the history is too strange
(squashing probably doesn't work well across multiple branches) and I
seriously believe that starting from scratch would be the best thing
to do at this point. But we should not create a new repo, simply use
the existing one.

In the meantime, *please* keep working normally on your own copy (the
repository on github under your username) on the tasks, don't wait for
the repository cleanup, else you will run out of time.

> Things I will be doing:
>
> - Limit testing.json to 1000 ports only.

If you put such a file to a repository for testing purposes (for
running unit tests), it should have 10 entries rather than 1000. If
you want the real/realistic data to start with, this would better be
done by putting a script into repository that fetches the real
PortIndex from our server (or simply runs portindex on your machine),
runs portindex2json and then loads that file. Or you can simply have a
local copy of the file, or create another repository with temporary
testing files. 15 MB files shouldn't be part of source repository.

> - Implement Categories table and the modified port table.

Yes, please do this.

> - Implement some sort of test atleast very trivial but will
> - Did not index the port table so far.But will now index the table.
>
>
>
> Things discussed in the last meeting:
>
> Implement these as well
> - Implement ability to mark deleted ports in the port table itself
> - For every port just display the last 5 - 10 commits on the ports

Sure, but this can be done towards the end of the project, no need to
rush into this now. Let's primarily make sure that we have a fully
functional portindex and installation statistics working properly.

> - Just mention the version everywhere. No portversion id.
> - When build started and ended.
>
> Future Goals:
> - implment obsolete
> - Fix portindex2json, atleast maintainers
> - Differential feature to be in stretch goals
> - Port History
> - Port Version table
>
> Also I think finalising the database ticket will be for long.As we are
> continusally evolving it so... Probably the milestone to create Database
> structure within the week may not be possible.

The milestone doesn't mean that you need to create a database that
will serve everyone for eternity. It means that:
- primarily ourselves, but ideally also a wider community needs to be
satisfied with the overall design proposal (we are nearly there)
- that the design should cover most of the functionality planned for
the summer, even if some details change later
- that you ideally have the port index fully working, utilizing all
the fields from portindex json file, but in particular have a sound
understanding of how to efficiently join tables (read as: this mostly
covers support for categories and maintainers, even if maintainers
don't cover 100% of weird cases with discrepancies in our index) and
that you are able to update the tables from a new portindex file
without exploding the database :)

Once this is done, the ticket would be closed and you would end up
with a document that will serve as a reference for the future /
documentation. You are of course free to make further changes to the
database afterwards, but at least the plan should be sound and we need
to be on the same page about what things should go in there and which
ones not.

> Mojca you said something about static files.
> I think we should have all Javascript also in a separate individual file.
> That's why in a seperate individual file.

Sure, javascript goes to a separate file, but the fake data table
should not be there.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Yes I was laos thinking about it for some time.

Please someone from infra team create another repository.
There I will maintain code in a proper way by Using Pull Requests.So that
code gets merged only if clean.


Regarding your general remarks.
I will look into them.
Please make the repo as soon as possible.


Things I will be doing:

- Limit testing.json to 1000 ports only.
- Implement Categories table and the modified port table.
- Implement some sort of test atleast very trivial but will
- Did not index the port table so far.But will now index the table.



Things discussed in the last meeting:

Implement these as well
- Implement ability to mark deleted ports in the port table itself
- For every port just display the last 5 - 10 commits on the ports
- Just mention the version everywhere. No portversion id.
- When build started and ended.

Future Goals:
- implment obsolete
- Fix portindex2json, atleast maintainers
- Differential feature to be in stretch goals
- Port History
- Port Version table

Also I think finalising the database ticket will be for long.As we are
continusally evolving it so... Probably the milestone to create Database
structure within the week may not be possible.

Mojca you said something about static files.
I think we should have all Javascript also in a separate individual file.
That's why in a seperate individual file.


Thanks

On 15 May 2018 at 02:53, Mojca Miklavec  wrote:

> Dear Vishnu,
>
> I just wanted to say that we *really really really* need to fix the
> repository since it's unacceptably large (some commits checked out are
> 240 MB, the size to transfer the contents is 133 MB).
>
> No matter what you think of it regarding breaking links etc., we
> definitely won't be hosting your linux bootloading images in our repo.
> This is one of the reasons where ... pull request make a huge
> difference over simple commits because such errors would be caught
> early on.
>
> Other general remarks:
> - please delete database_design.txt now that you have the markdown document
> - you should not commit __pycache__ folders
> - in dropdown.css you are mixing tabs and spaces; pick one and use it
> consistently, don't mix the two (and if you use spaces, use 2 or 4,
> not 9; identing is totally lacking)
> - by asking you to move code to static I did not mean that you should
> put data points for a graph to static files.
> - do not put 20 MB json files with sample data to the repository
>
> My suggestion would in fact be to start with an empty repository again
> and ask you to create your first pull request with cleaned up data.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 13 May 2018 at 09:37, Vishnu wrote:
> Hey
>
> Mojca you said once that you have lot to comment on the current database.
> So it would be great if you could do that.

I left a comment on
https://github.com/macports/macports-webapp/issues/2
and made a bunch of changes and suggestions to

https://github.com/macports/macports-webapp/blob/master/docs/Database_Design.md
(I would still like to go through installation statistics and build
statistics in slightly more detail.)

Since the idea was to spend the whole week heavily concentrated on the
database design and implementation (which also includes planning,
documenting, changing, testing etc.), my suggestion would be to do the
following (but feel free to adapt it):

(0) Given the limitation in heroku, what you can do is take
approximately 1000 ports for the initial tests. A quick trick could be
to loop through the file with something like "while
lowercase(first_letter_of_portname) < 'f'". No need to spend any extra
time of this, just a simple suggestion for a quick workaround. You can
of course also cut the portindex file manually for now.

(1) So far you have implemented one table with portindex. Before you
end up implementing the full database scheme from the ground up, you
can probably try the first and the easiest many-to-many relationship
"categories". Change the current app to add a new table "categories"
and while you are reading the database of ports:
- create or update categories as ports list them
- create entries in port_category
- the port page should list all categories to which this port belongs
- these categories should be hyperlinks to a page listing all ports
belonging to that particular category
so basically something similar to what https://www.macports.org/ports.php does.

(2) I would strongly suggest to try to figure out how to do unit tests
and start writing some tests. A random URL:
https://realpython.com/testing-in-django-part-1-best-practices-and-examples/
but don't necessarily stick to this URL/suggestion, it's just one of
the first few hits, you might find better resources elsewhere.

What would be nice to test at this stage is for example the following scenario:
- prepare a simple, stripped-down version of portindex.json (maybe
just 5 ports or so)
- import them to a clean database (you should have a different
database for running unit tests than "production" code)
- check that you have the correct number of entries in all tables,
check for some values that are potentially tricky to get right
- prepare another simple portindex.json with one new port, one deleted
port, one updated port, apply that one and make sure that you are
still getting the expected number of entries in all tables, and that
what you are getting is still correct

(3) I didn't check the recent changes in details yet, but you should
make sure that the application keeps working properly when you read
portindex.json multiple times. Last time the database would create a
new entry for the same port. If that port exists, you should update
the entry instead of creating a new one (there's a function
create_or_update, I think).

Of course feel free to continue implementing tables from the design
document, but instead of rushing into creation of all tables at once,
it make most sense to test them incrementally and have a good coverage
with unit tests to avoid breaking stuff as you keep developing.

Since you started working on portindex anyway, it's probably a lot
more important to have a well implemented and tested this part of
database (+ of course having the database design document ready for
further steps) than to rush into setting up all the tables at once.


See also
https://github.com/umeshksingla/macports-stats
which has been created during / immediately after the MacPorts meeting
in Slovenia two months ago as we were brainstorming on this idea. It's
just a quick proof-of-concept repository, but you might find some
ideas there.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Also can i know the status of your getting access to your VM ?

Thanks

On 13 May 2018 at 13:07, Vishnu  wrote:

> Hey
>
> Mojca you said once that you have lot to comment on the current database.
> So it would be great if you could do that.
> As today is the last day of Community bonding period.
> From tommorrow coding starts.
>
> Thanks
>
> On 12 May 2018 at 23:14, Mojca Miklavec  wrote:
>
>> Dear Vishnu,
>>
>> Sorry for being slightly out-of-sync (sending emails I wrote quite a
>> while ago and in the meantime no longer being up-to-date with other
>> responses).
>>
>> On 12 May 2018 at 17:26, Vishnu wrote:
>> > Hi
>> >
>> > "The PortIndex file itself is generated with the 'portindex' command,
>> > which only updates the data for Portfiles that have a modification time
>> > that is newer than the existing PortIndex file."
>> >
>> > So where can i see that code that checks the modification time?
>>
>> https://github.com/macports/macports-base/blob/master/src/po
>> rt/portindex.tcl
>>
>> > So suppose
>> > Our ports tree contains three ports A, B, C last modified time : 1:00
>> UTC
>> > present port index:
>> > [ A
>> >   B
>> >   C
>> > ] @ 1:00 UTC
>> >
>> > There was a edit in Port A @ 1:30 UTC
>> > so something will check all the port files modification time and if
>> > relatively newer it will update the port index.
>> >
>> > so new  port index:
>> > [ A
>> >   B
>> >   C
>> > ] @ 1:30 UTC
>> >
>> >
>> > "The portindex2postgres.sql script merely converts the PortIndex file
>> > from the custom format based on Tcl lists to SQL. The output will always
>> > contain SQL statements with the full data for every port."
>> >
>> > Then after the port index is updated this script runs and flushes the db
>> > then fills it with new portindex data?
>> > or just updates the db with the modified port data?
>>
>> I don't know the answer without looking at it first (you can check
>> what the code does yourself), but from the perspective of your app
>> development this is irrelevant.
>>
>> > So after every commit the entire 15mb portindex being processed and data
>> > being uploaded to the db.
>> > I think this is very inefficient.
>> > rather it would be best if after every commit the buildbot/ webhook
>> updates
>> > the database itself.
>>
>> I totally agree that it might make a lot more sense to submit just
>> differential data, but my suggestion would be to first implement the
>> functionality and optimise it at the end if there will be time. No
>> user will suffer from suboptimal data transfer (even if that database
>> only gets updated once per day, it will still be perfectly
>> acceptable), while nobody will benefit from spending a lot of time
>> figuring out how to properly implement a super efficient solution and
>> then running out of time to actually finish the app.
>>
>> Let's just work with what we have for now.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Mojca you said once that you have lot to comment on the current database.
So it would be great if you could do that.
As today is the last day of Community bonding period.
>From tommorrow coding starts.

Thanks

On 12 May 2018 at 23:14, Mojca Miklavec  wrote:

> Dear Vishnu,
>
> Sorry for being slightly out-of-sync (sending emails I wrote quite a
> while ago and in the meantime no longer being up-to-date with other
> responses).
>
> On 12 May 2018 at 17:26, Vishnu wrote:
> > Hi
> >
> > "The PortIndex file itself is generated with the 'portindex' command,
> > which only updates the data for Portfiles that have a modification time
> > that is newer than the existing PortIndex file."
> >
> > So where can i see that code that checks the modification time?
>
> https://github.com/macports/macports-base/blob/master/src/
> port/portindex.tcl
>
> > So suppose
> > Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
> > present port index:
> > [ A
> >   B
> >   C
> > ] @ 1:00 UTC
> >
> > There was a edit in Port A @ 1:30 UTC
> > so something will check all the port files modification time and if
> > relatively newer it will update the port index.
> >
> > so new  port index:
> > [ A
> >   B
> >   C
> > ] @ 1:30 UTC
> >
> >
> > "The portindex2postgres.sql script merely converts the PortIndex file
> > from the custom format based on Tcl lists to SQL. The output will always
> > contain SQL statements with the full data for every port."
> >
> > Then after the port index is updated this script runs and flushes the db
> > then fills it with new portindex data?
> > or just updates the db with the modified port data?
>
> I don't know the answer without looking at it first (you can check
> what the code does yourself), but from the perspective of your app
> development this is irrelevant.
>
> > So after every commit the entire 15mb portindex being processed and data
> > being uploaded to the db.
> > I think this is very inefficient.
> > rather it would be best if after every commit the buildbot/ webhook
> updates
> > the database itself.
>
> I totally agree that it might make a lot more sense to submit just
> differential data, but my suggestion would be to first implement the
> functionality and optimise it at the end if there will be time. No
> user will suffer from suboptimal data transfer (even if that database
> only gets updated once per day, it will still be perfectly
> acceptable), while nobody will benefit from spending a lot of time
> figuring out how to properly implement a super efficient solution and
> then running out of time to actually finish the app.
>
> Let's just work with what we have for now.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

Sorry for being slightly out-of-sync (sending emails I wrote quite a
while ago and in the meantime no longer being up-to-date with other
responses).

On 12 May 2018 at 17:26, Vishnu wrote:
> Hi
>
> "The PortIndex file itself is generated with the 'portindex' command,
> which only updates the data for Portfiles that have a modification time
> that is newer than the existing PortIndex file."
>
> So where can i see that code that checks the modification time?

https://github.com/macports/macports-base/blob/master/src/port/portindex.tcl

> So suppose
> Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
> present port index:
> [ A
>   B
>   C
> ] @ 1:00 UTC
>
> There was a edit in Port A @ 1:30 UTC
> so something will check all the port files modification time and if
> relatively newer it will update the port index.
>
> so new  port index:
> [ A
>   B
>   C
> ] @ 1:30 UTC
>
>
> "The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port."
>
> Then after the port index is updated this script runs and flushes the db
> then fills it with new portindex data?
> or just updates the db with the modified port data?

I don't know the answer without looking at it first (you can check
what the code does yourself), but from the perspective of your app
development this is irrelevant.

> So after every commit the entire 15mb portindex being processed and data
> being uploaded to the db.
> I think this is very inefficient.
> rather it would be best if after every commit the buildbot/ webhook updates
> the database itself.

I totally agree that it might make a lot more sense to submit just
differential data, but my suggestion would be to first implement the
functionality and optimise it at the end if there will be time. No
user will suffer from suboptimal data transfer (even if that database
only gets updated once per day, it will still be perfectly
acceptable), while nobody will benefit from spending a lot of time
figuring out how to properly implement a super efficient solution and
then running out of time to actually finish the app.

Let's just work with what we have for now.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Rainer Müller]

On 2018-05-12 17:26, Vishnu wrote:
> So where can i see that code that checks the modification time?

https://github.com/macports/macports-base/blob/bb9db185d54c04efd55a3d634518a971bbb530a7/src/port/portindex.tcl#L120-L141

> "The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port."
> 
> Then after the port index is updated this script runs and flushes the db
> then fills it with new portindex data?
> or just updates the db with the modified port data?
> 
> Which one occurs?
> 
> Trying to understand the present mechanism.

https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/portindex2postgres.tcl#L50-L57

> So after every commit the entire 15mb portindex being processed and data
> being uploaded to the db.
> I think this is very inefficient.
> rather it would be best if after every commit the buildbot/ webhook
> updates the database itself.

It does not matter whether the new information about the port appears on
macports-webapp 10 seconds or a full minute after the commit occurred.
Remember, it might even take at least as long as a full hour until the
updated Portfile will actually be available over rsync.

This is not a critical path. The data is compressed for the transfer, so
the 15 MB boil down to an upload size of only about 1 MB. Any algorithm
to figure out the differences here might likely even take longer than
just naively transferring the full data...

I do not see a pressing need for optimization here. The way it is done
at the moment is as robust as it can get to ensure the data is always
up-to-date.

If you really need to get the differences then that should be done in
comparison with the actual database. Doing it before would always make
assumptions about the state of the database instead of using the actual
data.

For example, when feeding the data into the database, make a temporary
in-memory copy of the old table first, then import the data, afterwards
select the differences by joining the old and new version of the tables.

Rainer

Re: Gsoc 18 Project | Collect build statistics

[Rainer Müller]

On 2018-05-12 17:27, Mojca Miklavec wrote:
> I would in fact really really like to see a functionality like
> portindex --mode=diff --format=json
> that would create a table like
> 
> {
> new = [...],
> removed = [...],
> updated = [...]
> }
> 
> which would only provide the difference since the last run of
> portindex. and then we would only push this data to the database.
> Since portindex already knows which files are new, this doesn't sound
> impossible to implement. This sounds a lot more efficient that
> transmitting full data, but sure ... full data can be done as well. I
> assume that the command which updates the ports would need to mark all
> ports as "potentially deleted" before reading and storing the data.

Something like this would be nice-to-have. That could also be used as a
way to present to the user what has changed since the last time they ran
a selfupdate/sync command.

However, incremental updates can be error-prone as they can get out of
sync, even if only a single update was missed for whatever reason.
Therefore, the functionality to transfer the complete portindex should
always be available in case we need to resync the database from scratch.
Transmitting an incremental update can still be added as an optimization
later.

Overall, we did not experience any problems with the database updates
that always transfer the full 15 MB of SQL. Why exactly do we expect
problems with 15 MB of JSON...?

Rainer

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 12 May 2018 at 14:17, Rainer Müller wrote:
> On 2018-05-12 10:34, Vishnu wrote:
>> I am not saying that my db and the exiting Db would be interdependent.
>> Rather i am saying just once in forever I can copy the content to my
>> database.
>> Then do the code to keep on updating it whenever something is added or
>> deleted or modified.
>>
>> Like for exaple maintainers.
>> You already have a table of mainatainers and their ports. So i can copy
>> the data to my database.

No, you should get that data from portindex.json, rather than from
copying it from another database.

>> Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
>> updating my database independent of the existing database.
>
> Our database contains exactly what the portindex2postgres.tcl script
> produces (note the different naming). You can just generate the data
> locally by running it against PortIndex.
>
> You need to have the ports tree in a directory named "ports", then run
> the following command from the parent directory:
>
>   port-tclsh /path/to/macports-infra/jobs/portindex2postgres.tcl
>
> This will create a PortIndex.sql in the current directory. Ignore or
> remove the macports.conf and sources.conf files that are also created.

But the information is also nearly identical to what one gets with
portindex2json.tcl, and if there are any issues (there are some), I
would say that we need to fix portindex2json instead, rather than
using fifty different ways to get nearly the same information to our
database.

>> Again and again processing the 15mb json file would be bothersome.
>> We definitely need some way of getting a differential json.
>> To update only the changes happened because of commit to the port file.
/.../
>> One doubt I still have is that whether  portindex2postgres.tcl again and
>> again runs the entire MacPorts port tree on the database ?
>> Or just the port that has been changed ?
>
> The PortIndex file itself is generated with the 'portindex' command,
> which only updates the data for Portfiles that have a modification time
> that is newer than the existing PortIndex file.
>
> The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port.

I would in fact really really like to see a functionality like
portindex --mode=diff --format=json
that would create a table like

{
new = [...],
removed = [...],
updated = [...]
}

which would only provide the difference since the last run of
portindex. and then we would only push this data to the database.
Since portindex already knows which files are new, this doesn't sound
impossible to implement. This sounds a lot more efficient that
transmitting full data, but sure ... full data can be done as well. I
assume that the command which updates the ports would need to mark all
ports as "potentially deleted" before reading and storing the data.

Of course, Vishnu, you should not wait for this functionality to be
implemented, I'm just saying that it would be super nice if it was
(figuring out how to implement it yourself would get you too
side-tracked, so it's probably not a good idea).

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

"The PortIndex file itself is generated with the 'portindex' command,
which only updates the data for Portfiles that have a modification time
that is newer than the existing PortIndex file."

So where can i see that code that checks the modification time?
So suppose
Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
present port index:
[ A
  B
  C
] @ 1:00 UTC

There was a edit in Port A @ 1:30 UTC
so something will check all the port files modification time and if
relatively newer it will update the port index.

so new  port index:
[ A
  B
  C
] @ 1:30 UTC


"The portindex2postgres.sql script merely converts the PortIndex file
from the custom format based on Tcl lists to SQL. The output will always
contain SQL statements with the full data for every port."

Then after the port index is updated this script runs and flushes the db
then fills it with new portindex data?
or just updates the db with the modified port data?

Which one occurs?

Trying to understand the present mechanism.


So after every commit the entire 15mb portindex being processed and data
being uploaded to the db.
I think this is very inefficient.
rather it would be best if after every commit the buildbot/ webhook updates
the database itself.


Thanks

On 12 May 2018 at 17:47, Rainer Müller  wrote:

> On 2018-05-12 10:34, Vishnu wrote:
> > I am not saying that my db and the exiting Db would be interdependent.
> > Rather i am saying just once in forever I can copy the content to my
> > database.
> > Then do the code to keep on updating it whenever something is added or
> > deleted or modified.
> >
> > Like for exaple maintainers.
> > You already have a table of mainatainers and their ports.So i can copy
> > the data to my database.
> > Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
> > updating my database independent of the existing database.
>
> Our database contains exactly what the portindex2postgres.tcl script
> produces (note the different naming). You can just generate the data
> locally by running it against PortIndex.
>
> You need to have the ports tree in a directory named "ports", then run
> the following command from the parent directory:
>
>   port-tclsh /path/to/macports-infra/jobs/portindex2postgres.tcl
>
> This will create a PortIndex.sql in the current directory. Ignore or
> remove the macports.conf and sources.conf files that are also created.
>
> > Again and again processing the 15mb json file would be bothersome.
> > We definitely need some way of getting a differential json.
> > To update only the changes happened because of commit to the port file.
> >
> > Or maybe we could make some changes in buildbot that.Whenever there is
> > some change in portfile it could update my port table as well with the
> > changes.
> > Not sure how to do this.But just an idea as of now.
>
> Yes, the idea would be that macports-webapp is updated after every commit.
>
> To achieve that, you could use a job in buildbot, but then the result
> needs to be transferred to the macports-webapp. Alternatively, you could
> deliver a WebHook from GitHub to do this within macports-webapp. It
> largely depends on what kind of data you want to gather and how.
> > And regarding working on a seperate branch.Rainer suggested not
> > to.Because daily pull requests have to merged by the mentor.
>
> I did not argue against working on a separate branch. I just wanted to
> point out that pull request reviews on GitHub might pose a problem as
> they cannot depend on each other. It is up to you and your mentors to
> work out the details.
>
> > One doubt I still have is that whether  portindex2postgres.tcl again and
> > again runs the entire MacPorts port tree on the database ?
> > Or just the port that has been changed ?
>
> The PortIndex file itself is generated with the 'portindex' command,
> which only updates the data for Portfiles that have a modification time
> that is newer than the existing PortIndex file.
>
> The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port.
>
> Rainer
>

Re: Gsoc 18 Project | Collect build statistics

[Rainer Müller]

On 2018-05-12 10:34, Vishnu wrote:
> I am not saying that my db and the exiting Db would be interdependent.
> Rather i am saying just once in forever I can copy the content to my
> database.
> Then do the code to keep on updating it whenever something is added or
> deleted or modified.
> 
> Like for exaple maintainers.
> You already have a table of mainatainers and their ports.So i can copy
> the data to my database.
> Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
> updating my database independent of the existing database.

Our database contains exactly what the portindex2postgres.tcl script
produces (note the different naming). You can just generate the data
locally by running it against PortIndex.

You need to have the ports tree in a directory named "ports", then run
the following command from the parent directory:

  port-tclsh /path/to/macports-infra/jobs/portindex2postgres.tcl

This will create a PortIndex.sql in the current directory. Ignore or
remove the macports.conf and sources.conf files that are also created.

> Again and again processing the 15mb json file would be bothersome.
> We definitely need some way of getting a differential json.
> To update only the changes happened because of commit to the port file.
> 
> Or maybe we could make some changes in buildbot that.Whenever there is
> some change in portfile it could update my port table as well with the
> changes.
> Not sure how to do this.But just an idea as of now.

Yes, the idea would be that macports-webapp is updated after every commit.

To achieve that, you could use a job in buildbot, but then the result
needs to be transferred to the macports-webapp. Alternatively, you could
deliver a WebHook from GitHub to do this within macports-webapp. It
largely depends on what kind of data you want to gather and how.
> And regarding working on a seperate branch.Rainer suggested not
> to.Because daily pull requests have to merged by the mentor.

I did not argue against working on a separate branch. I just wanted to
point out that pull request reviews on GitHub might pose a problem as
they cannot depend on each other. It is up to you and your mentors to
work out the details.

> One doubt I still have is that whether  portindex2postgres.tcl again and
> again runs the entire MacPorts port tree on the database ?
> Or just the port that has been changed ?

The PortIndex file itself is generated with the 'portindex' command,
which only updates the data for Portfiles that have a modification time
that is newer than the existing PortIndex file.

The portindex2postgres.sql script merely converts the PortIndex file
from the custom format based on Tcl lists to SQL. The output will always
contain SQL statements with the full data for every port.

Rainer

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I am not saying that my db and the exiting Db would be interdependent.
Rather i am saying just once in forever I can copy the content to my
database.
Then do the code to keep on updating it whenever something is added or
deleted or modified.

Like for exaple maintainers.
You already have a table of mainatainers and their ports.So i can copy the
data to my database.
Then i will make a similar to PortIndex2PGSQL.tcl script to keep on
updating my database independent of the existing database.

Again and again processing the 15mb json file would be bothersome.
We definitely need some way of getting a differential json.
To update only the changes happened because of commit to the port file.

Or maybe we could make some changes in buildbot that.Whenever there is some
change in portfile it could update my port table as well with the changes.
Not sure how to do this.But just an idea as of now.

"You need a separate glue table to connect maintainers with ports. But
that's already part of your database design. You should not store
maintainers in you port table, but perform correct joins (or maybe
create a view first?) instead."

You misunderstood me, Copying the data.I meant once in a lifetime.
Then keep on updating the table with whatever changes happens.
And i will definitely perform join or views.

I will be doing following things :
- Update Ticket descriptions
- Put the gsoc proposal in docs
- update the graph on the static website
- try to handle sensitive keys


Regarding heroku.
Don't worry about that. I got it figured out.

And regarding working on a seperate branch.Rainer suggested not to.Because
daily pull requests have to merged by the mentor.

Would directly commiting to master be a problem ?
As only i will be dealing with the repo.
Probably no one will right now clone the repository.

It is also very easy to directly commit to master than creating a PR.



One doubt I still have is that whether  portindex2postgres.tcl again and
again runs the entire MacPorts port tree on the database ?
Or just the port that has been changed ?


Thanks

On 11 May 2018 at 12:47, Mojca Miklavec  wrote:

> On 11 May 2018 at 04:00, Vishnu wrote:
> > Hi
> >
> > Also i wanted to know
> > For this link
> > https://www.macports.org/ports.php?by=name=python27
> >
> > There is already existing database with loads of information.
> > Is it not updated?
> > is  it static ?
>
> It is not static, the database is updated via an external script from
> PortIndex and the page is rendered with php code, reading information
> from the database.
>
> > If it is updated automatically...Then half of the work for database would
> > already be over.
> >
> > Could i get access to the DB?
>
> Once your application reaches a certain quality & completeness, it
> should render the above page obsolete and the above page will most
> likely be removed.
>
> While we could theoretically give you access to the database (not
> without opening some security holes): first of all this hardly makes
> any sense because you should always be able to reproduce the exact
> state of the database if you run PortIndex2PGSQL.tcl yourself, but
> most importantly we don't want to introduce a complex network of
> interdependencies.
>
> Let's say that I need to change the new app a few months from now.
> Before I can deploy the change I need to test it locally on my
> machine. If you app will require password-protected or ip-limited
> access to some third-party database, this would greatly hinder my
> ability to patch the new app. The app should be as standalone as
> possible. The next very important aspect is that you might want to
> include more information than what's currently available in that
> database. For example, you might decide to include the information
> about which port supports which version of macOS. That information is
> not available in the existing database, so you would then first need
> to patch who-knows-what-other-code in random mixture of tcl, php,
> bash, ... And you would not be able to do table joins etc. It's
> absolutely bad idea to depend on third-party database. You should of
> course collect information from various sources, but you need to keep
> all the required data in a single database somewhere close to your
> app.
>
> Some time ago you had ideas about updating portindex just once per
> year? Well, here's an example why you should ideally do it after each
> commit rather than once per year. That said, we should ideally find a
> way to optimise the PortIndex2json part (probably not your task).
> PortIndex already updates just those ports that have in fact been
> changed, but conversion to json updates the complete file and needs to
> feed the web app with full 15 MB of data after each commit. This is
> quite a bit suboptimal.
>
> > I got the rough idea of the db from here:
> > https://github.com/macports/macports-infrastructure/blob/
> 2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249
>
> 

Re: Gsoc 18 Project | Collect build statistics

[Rainer Müller]

On 2018-05-11 09:17, Mojca Miklavec wrote:
>> I got the rough idea of the db from here:
>> https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249
> 
> Sure, you can take this as inspiration for database design, but not as
> your definite source of data. Your definite source of data should be
> PortIndex from the latest git checkout of macports-ports.
> 
>> So is this updated regularly?How?
> 
> I'm not absolutely sure, but I believe the changes are deployed after
> each commit via
> https://build.macports.org/builders/jobs-portindex

That is correct. The job is using the portindex2postgres.tcl script. The
SQL output is imported using psql into the database.

https://github.com/macports/macports-infrastructure/blob/master/jobs/portindex2postgres.tcl

 On 9 May 2018 at 03:43, Vishnu  wrote:
>
> I had one doubt.
> Should i switch the link in heroku account for integration with macports
> github ?
> 
> I'm not sure what is it that you are asking. If Heroku need special
> priviliges on GitHub (what permissions are required?), it might be
> best to create an additional user on GitHub for now. Can you provide
> some pointers? It's pointless that I theoreticise about the options
> before I know what is required.

We can also provide GitHub API tokens that belong to the macportsbot
user, if that is what you need. You would need to tell us the required
scope for the tokens.

> Because i think then you need to give accesss to heroku of your account.
>
> I think it would be wise for me to do the commit update in my local
> repository itself..
>
> Once every 2 weeks or something ill push all the changes to macports
> repository.
> Do comment .What should be done?
> 
> I wanted to suggest that the app would be developed either in a
> separate branch or in your fork of repository (not in master branch,
> please!), and then you would make pull requests on regular basis and
> we would review the pull request and make sure that any code that ends
> up in repository has been fully reviewed / tested.
> 
> If you commit everything (including tests) straight to the master
> branch of the main repository, it's more challenging to track which
> code reviews have already been taken into account and which ones were
> not. You should not make giant pull requests anyway (ideally you would
> make a PR on approximately daily basis or at least once per week in
> case you need to figure out a number of things and code is not yet
> ready; making one giant PR every two weeks or - god forbid - once per
> month or only at the end of GSOC might cause too many troubles).

Push your code often, not just every two weeks. Pushing early gives
mentors more time to review your code before calls. It also makes it
easier for the mentors to track your progress and also for yourself.

The problem with pull requests would be that GitHub cannot handle
dependencies between pull requests. A daily pull request will probably
not work out, unless mentors also approve and merge the pull request
within a day...

Rainer

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 11 May 2018 at 04:00, Vishnu wrote:
> Hi
>
> Also i wanted to know
> For this link
> https://www.macports.org/ports.php?by=name=python27
>
> There is already existing database with loads of information.
> Is it not updated?
> is  it static ?

It is not static, the database is updated via an external script from
PortIndex and the page is rendered with php code, reading information
from the database.

> If it is updated automatically...Then half of the work for database would
> already be over.
>
> Could i get access to the DB?

Once your application reaches a certain quality & completeness, it
should render the above page obsolete and the above page will most
likely be removed.

While we could theoretically give you access to the database (not
without opening some security holes): first of all this hardly makes
any sense because you should always be able to reproduce the exact
state of the database if you run PortIndex2PGSQL.tcl yourself, but
most importantly we don't want to introduce a complex network of
interdependencies.

Let's say that I need to change the new app a few months from now.
Before I can deploy the change I need to test it locally on my
machine. If you app will require password-protected or ip-limited
access to some third-party database, this would greatly hinder my
ability to patch the new app. The app should be as standalone as
possible. The next very important aspect is that you might want to
include more information than what's currently available in that
database. For example, you might decide to include the information
about which port supports which version of macOS. That information is
not available in the existing database, so you would then first need
to patch who-knows-what-other-code in random mixture of tcl, php,
bash, ... And you would not be able to do table joins etc. It's
absolutely bad idea to depend on third-party database. You should of
course collect information from various sources, but you need to keep
all the required data in a single database somewhere close to your
app.

Some time ago you had ideas about updating portindex just once per
year? Well, here's an example why you should ideally do it after each
commit rather than once per year. That said, we should ideally find a
way to optimise the PortIndex2json part (probably not your task).
PortIndex already updates just those ports that have in fact been
changed, but conversion to json updates the complete file and needs to
feed the web app with full 15 MB of data after each commit. This is
quite a bit suboptimal.

> I got the rough idea of the db from here:
> https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249

Sure, you can take this as inspiration for database design, but not as
your definite source of data. Your definite source of data should be
PortIndex from the latest git checkout of macports-ports.

> So is this updated regularly?How?

I'm not absolutely sure, but I believe the changes are deployed after
each commit via
https://build.macports.org/builders/jobs-portindex

> And there seems to be lot of data already existing. Which i could just copy.
> Existing maintainer table,many other.

No. You should replicate that in your app.

You need a separate glue table to connect maintainers with ports. But
that's already part of your database design. You should not store
maintainers in you port table, but perform correct joins (or maybe
create a view first?) instead.

> On 11 May 2018 at 07:11, Vishnu  wrote:
>>
>> Hi
>>
>> So should I start working on the same database?
>>
>> Community does not have any suggestions I guess.
>> So should i go ahead with the existing structure?

*I* do have lots of comments even if nobody else does. One of the
problems is that the excell table and your md documents were not in
sync last time I checked (some tables were missing etc.)

While I'm working on them, can you please:

- Update ticket descriptions (some of them were empty when I checked,
some would say "Implement this feature" without actually telling
anything else). Super short descriptions are in principle ok, but only
if we have an additional document describing what will actually be
done. You already spent quite some time writing your proposal, but the
proposal is only visible to GSOC mentors. Can we put your proposal
(probably in markdown?) next to database design document? The idea is
that anyone looking at the repository should have an understanding of
what will be done (and could in principle pick it up himself), but we
might want to update the document during the summer. While you might
know that feature X should be implemented, maybe you don't know yet
how to implement it and before you spend a lot of time on coding
something that would be a bad idea and would need to be dropped
because it would be too inefficient or too insecure, it makes sense to
have a very clear document of what will be done.

- Remove 

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Also i wanted to know
For this link
https://www.macports.org/ports.php?by=name=python27

There is already existing database with loads of information.
Is it not updated?
is  it static ?

If it is updated automatically...Then half of the work for database would
already be over.

Could i get access to the DB ?
I got the rough idea of the db from here:
https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249

So is this updated regularly?How?

And there seems to be lot of data already existing. Which i could just copy.
Existing maintainer table,many other.

Thanks


On 11 May 2018 at 07:11, Vishnu  wrote:

> Hi
>
> So should I start working on the same database?
>
> Community does not have any suggestions I guess.
> So should i go ahead with the existing structure?
>
> I had a doubt regarding version.
>
> How can i check the existing versions of any port.
> Say python27?
>
> Any list of maintainers i can get?
>
> Thanks
>
> On 9 May 2018 at 04:06, Vishnu  wrote:
>
>> Hi
>>
>> And also i couldn't figure out any way to hide passwords/ Sensitive
>> information while creating app.
>>
>> Thanks
>>
>> On 9 May 2018 at 03:43, Vishnu  wrote:
>>
>>> Hi
>>>
>>> I had one doubt.
>>> Should i switch the link in heroku account for integration with macports
>>> github ?
>>>
>>> Because i think then you need to give accesss to heroku of your account.
>>>
>>> I think it would be wise for me to do the commit update in my local
>>> repository itself..
>>>
>>> Once every 2 weeks or something ill push all the changes to macports
>>> repository.
>>> Do comment .What should be done?
>>>
>>> Thanks
>>>
>>> On 8 May 2018 at 11:40, Jackson Isaac  wrote:
>>>
 On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec 
 wrote:
 >
 > On 8 May 2018 at 01:12, Vishnu wrote:
 > > Hi
 > >
 > > I will work on all the points mentioned today..
 > >
 > > But i have been trying many different ways.
 > > To get 20k ports onto the database.. The site is crashing.. Timeout
 error.
 > > Or the site cant be reached.
 > > Maximum i got till 9k ports successfully entered.
 > >
 > > Tried many ways..
 > >
 > > If you have any suggestions do let me know.
 >

 We can even try firebase for the DB backend and host the frontend on
 heroku.

 Firebase free version gives 1GB of storage and 10GB of bandwidth per
 month.
 I guess for testing out the app on complete portindex we could give it
 a try.

 https://firebase.google.com/pricing/

 Then again we would have some kind of limitations over here in long run.

 >
 > I was pretty sure that we would reach the database limit here.
 >
 >
 > This site
 > https://devcenter.heroku.com/articles/heroku-postgres-plans
 > says that the limit is 10k rows which is not anywhere near enough and
 > fully consistent with your observation. The plan that lets you have
 > 10M rows is 9 USD per month which I gladly pay if that would solve the
 > issues (I'm not sure if those are the only ones, we might not have
 > sufficient memory etc.)
 >

 9USD is only for the DB though. We might need to pay for the compute
 separately.

 Looks like free VMs come with only 512MB RAM.

 >
 > (I also remembered that I might have access to create a clean virtual
 > machine, but I need a couple of days to physically reach a place where
 > I have the additional info. That would require setting up the whole
 > machine manually of course, but we need to do that for the final
 > deployment anyway.)
 >
 > > I think i have to break the data entry in separate chunks and do
 it. But not
 > > sure how to do that.
 >
 > In any case Heroku will apparently not let you import the full
 > database anyway until we switch to a payed account. If you still have
 > troubles with performance issues afterwards, this also won't help.
 > What you could do is temporarily copy or move a few folders (say,
 > math, science and python) and run portindex just on those. This will
 > give you a smaller number of ports to work with, but still sufficient
 > to figure out what other issues you'll need to deal with (port listing
 > will have to be paginated etc.)
 >
 > It's a good exercise to know what the limitations are. When you play
 > with new features and database design, it's always easier to play
 > with, say 10-100 ports than with full 15 MB portindex file.
 >

 This plan sounds good. +1

 --
 Jackson Isaac

>>>
>>>
>>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

So should I start working on the same database?

Community does not have any suggestions I guess.
So should i go ahead with the existing structure?

I had a doubt regarding version.

How can i check the existing versions of any port.
Say python27?

Any list of maintainers i can get?

Thanks

On 9 May 2018 at 04:06, Vishnu  wrote:

> Hi
>
> And also i couldn't figure out any way to hide passwords/ Sensitive
> information while creating app.
>
> Thanks
>
> On 9 May 2018 at 03:43, Vishnu  wrote:
>
>> Hi
>>
>> I had one doubt.
>> Should i switch the link in heroku account for integration with macports
>> github ?
>>
>> Because i think then you need to give accesss to heroku of your account.
>>
>> I think it would be wise for me to do the commit update in my local
>> repository itself..
>>
>> Once every 2 weeks or something ill push all the changes to macports
>> repository.
>> Do comment .What should be done?
>>
>> Thanks
>>
>> On 8 May 2018 at 11:40, Jackson Isaac  wrote:
>>
>>> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec 
>>> wrote:
>>> >
>>> > On 8 May 2018 at 01:12, Vishnu wrote:
>>> > > Hi
>>> > >
>>> > > I will work on all the points mentioned today..
>>> > >
>>> > > But i have been trying many different ways.
>>> > > To get 20k ports onto the database.. The site is crashing.. Timeout
>>> error.
>>> > > Or the site cant be reached.
>>> > > Maximum i got till 9k ports successfully entered.
>>> > >
>>> > > Tried many ways..
>>> > >
>>> > > If you have any suggestions do let me know.
>>> >
>>>
>>> We can even try firebase for the DB backend and host the frontend on
>>> heroku.
>>>
>>> Firebase free version gives 1GB of storage and 10GB of bandwidth per
>>> month.
>>> I guess for testing out the app on complete portindex we could give it a
>>> try.
>>>
>>> https://firebase.google.com/pricing/
>>>
>>> Then again we would have some kind of limitations over here in long run.
>>>
>>> >
>>> > I was pretty sure that we would reach the database limit here.
>>> >
>>> >
>>> > This site
>>> > https://devcenter.heroku.com/articles/heroku-postgres-plans
>>> > says that the limit is 10k rows which is not anywhere near enough and
>>> > fully consistent with your observation. The plan that lets you have
>>> > 10M rows is 9 USD per month which I gladly pay if that would solve the
>>> > issues (I'm not sure if those are the only ones, we might not have
>>> > sufficient memory etc.)
>>> >
>>>
>>> 9USD is only for the DB though. We might need to pay for the compute
>>> separately.
>>>
>>> Looks like free VMs come with only 512MB RAM.
>>>
>>> >
>>> > (I also remembered that I might have access to create a clean virtual
>>> > machine, but I need a couple of days to physically reach a place where
>>> > I have the additional info. That would require setting up the whole
>>> > machine manually of course, but we need to do that for the final
>>> > deployment anyway.)
>>> >
>>> > > I think i have to break the data entry in separate chunks and do it.
>>> But not
>>> > > sure how to do that.
>>> >
>>> > In any case Heroku will apparently not let you import the full
>>> > database anyway until we switch to a payed account. If you still have
>>> > troubles with performance issues afterwards, this also won't help.
>>> > What you could do is temporarily copy or move a few folders (say,
>>> > math, science and python) and run portindex just on those. This will
>>> > give you a smaller number of ports to work with, but still sufficient
>>> > to figure out what other issues you'll need to deal with (port listing
>>> > will have to be paginated etc.)
>>> >
>>> > It's a good exercise to know what the limitations are. When you play
>>> > with new features and database design, it's always easier to play
>>> > with, say 10-100 ports than with full 15 MB portindex file.
>>> >
>>>
>>> This plan sounds good. +1
>>>
>>> --
>>> Jackson Isaac
>>>
>>
>>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

And also i couldn't figure out any way to hide passwords/ Sensitive
information while creating app.

Thanks

On 9 May 2018 at 03:43, Vishnu  wrote:

> Hi
>
> I had one doubt.
> Should i switch the link in heroku account for integration with macports
> github ?
>
> Because i think then you need to give accesss to heroku of your account.
>
> I think it would be wise for me to do the commit update in my local
> repository itself..
>
> Once every 2 weeks or something ill push all the changes to macports
> repository.
> Do comment .What should be done?
>
> Thanks
>
> On 8 May 2018 at 11:40, Jackson Isaac  wrote:
>
>> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec 
>> wrote:
>> >
>> > On 8 May 2018 at 01:12, Vishnu wrote:
>> > > Hi
>> > >
>> > > I will work on all the points mentioned today..
>> > >
>> > > But i have been trying many different ways.
>> > > To get 20k ports onto the database.. The site is crashing.. Timeout
>> error.
>> > > Or the site cant be reached.
>> > > Maximum i got till 9k ports successfully entered.
>> > >
>> > > Tried many ways..
>> > >
>> > > If you have any suggestions do let me know.
>> >
>>
>> We can even try firebase for the DB backend and host the frontend on
>> heroku.
>>
>> Firebase free version gives 1GB of storage and 10GB of bandwidth per
>> month.
>> I guess for testing out the app on complete portindex we could give it a
>> try.
>>
>> https://firebase.google.com/pricing/
>>
>> Then again we would have some kind of limitations over here in long run.
>>
>> >
>> > I was pretty sure that we would reach the database limit here.
>> >
>> >
>> > This site
>> > https://devcenter.heroku.com/articles/heroku-postgres-plans
>> > says that the limit is 10k rows which is not anywhere near enough and
>> > fully consistent with your observation. The plan that lets you have
>> > 10M rows is 9 USD per month which I gladly pay if that would solve the
>> > issues (I'm not sure if those are the only ones, we might not have
>> > sufficient memory etc.)
>> >
>>
>> 9USD is only for the DB though. We might need to pay for the compute
>> separately.
>>
>> Looks like free VMs come with only 512MB RAM.
>>
>> >
>> > (I also remembered that I might have access to create a clean virtual
>> > machine, but I need a couple of days to physically reach a place where
>> > I have the additional info. That would require setting up the whole
>> > machine manually of course, but we need to do that for the final
>> > deployment anyway.)
>> >
>> > > I think i have to break the data entry in separate chunks and do it.
>> But not
>> > > sure how to do that.
>> >
>> > In any case Heroku will apparently not let you import the full
>> > database anyway until we switch to a payed account. If you still have
>> > troubles with performance issues afterwards, this also won't help.
>> > What you could do is temporarily copy or move a few folders (say,
>> > math, science and python) and run portindex just on those. This will
>> > give you a smaller number of ports to work with, but still sufficient
>> > to figure out what other issues you'll need to deal with (port listing
>> > will have to be paginated etc.)
>> >
>> > It's a good exercise to know what the limitations are. When you play
>> > with new features and database design, it's always easier to play
>> > with, say 10-100 ports than with full 15 MB portindex file.
>> >
>>
>> This plan sounds good. +1
>>
>> --
>> Jackson Isaac
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I had one doubt.
Should i switch the link in heroku account for integration with macports
github ?

Because i think then you need to give accesss to heroku of your account.

I think it would be wise for me to do the commit update in my local
repository itself..

Once every 2 weeks or something ill push all the changes to macports
repository.
Do comment .What should be done?

Thanks

On 8 May 2018 at 11:40, Jackson Isaac  wrote:

> On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec 
> wrote:
> >
> > On 8 May 2018 at 01:12, Vishnu wrote:
> > > Hi
> > >
> > > I will work on all the points mentioned today..
> > >
> > > But i have been trying many different ways.
> > > To get 20k ports onto the database.. The site is crashing.. Timeout
> error.
> > > Or the site cant be reached.
> > > Maximum i got till 9k ports successfully entered.
> > >
> > > Tried many ways..
> > >
> > > If you have any suggestions do let me know.
> >
>
> We can even try firebase for the DB backend and host the frontend on
> heroku.
>
> Firebase free version gives 1GB of storage and 10GB of bandwidth per month.
> I guess for testing out the app on complete portindex we could give it a
> try.
>
> https://firebase.google.com/pricing/
>
> Then again we would have some kind of limitations over here in long run.
>
> >
> > I was pretty sure that we would reach the database limit here.
> >
> >
> > This site
> > https://devcenter.heroku.com/articles/heroku-postgres-plans
> > says that the limit is 10k rows which is not anywhere near enough and
> > fully consistent with your observation. The plan that lets you have
> > 10M rows is 9 USD per month which I gladly pay if that would solve the
> > issues (I'm not sure if those are the only ones, we might not have
> > sufficient memory etc.)
> >
>
> 9USD is only for the DB though. We might need to pay for the compute
> separately.
>
> Looks like free VMs come with only 512MB RAM.
>
> >
> > (I also remembered that I might have access to create a clean virtual
> > machine, but I need a couple of days to physically reach a place where
> > I have the additional info. That would require setting up the whole
> > machine manually of course, but we need to do that for the final
> > deployment anyway.)
> >
> > > I think i have to break the data entry in separate chunks and do it.
> But not
> > > sure how to do that.
> >
> > In any case Heroku will apparently not let you import the full
> > database anyway until we switch to a payed account. If you still have
> > troubles with performance issues afterwards, this also won't help.
> > What you could do is temporarily copy or move a few folders (say,
> > math, science and python) and run portindex just on those. This will
> > give you a smaller number of ports to work with, but still sufficient
> > to figure out what other issues you'll need to deal with (port listing
> > will have to be paginated etc.)
> >
> > It's a good exercise to know what the limitations are. When you play
> > with new features and database design, it's always easier to play
> > with, say 10-100 ports than with full 15 MB portindex file.
> >
>
> This plan sounds good. +1
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec  wrote:
>
> On 8 May 2018 at 01:12, Vishnu wrote:
> > Hi
> >
> > I will work on all the points mentioned today..
> >
> > But i have been trying many different ways.
> > To get 20k ports onto the database.. The site is crashing.. Timeout error.
> > Or the site cant be reached.
> > Maximum i got till 9k ports successfully entered.
> >
> > Tried many ways..
> >
> > If you have any suggestions do let me know.
>

We can even try firebase for the DB backend and host the frontend on heroku.

Firebase free version gives 1GB of storage and 10GB of bandwidth per month.
I guess for testing out the app on complete portindex we could give it a try.

https://firebase.google.com/pricing/

Then again we would have some kind of limitations over here in long run.

>
> I was pretty sure that we would reach the database limit here.
>
>
> This site
> https://devcenter.heroku.com/articles/heroku-postgres-plans
> says that the limit is 10k rows which is not anywhere near enough and
> fully consistent with your observation. The plan that lets you have
> 10M rows is 9 USD per month which I gladly pay if that would solve the
> issues (I'm not sure if those are the only ones, we might not have
> sufficient memory etc.)
>

9USD is only for the DB though. We might need to pay for the compute separately.

Looks like free VMs come with only 512MB RAM.

>
> (I also remembered that I might have access to create a clean virtual
> machine, but I need a couple of days to physically reach a place where
> I have the additional info. That would require setting up the whole
> machine manually of course, but we need to do that for the final
> deployment anyway.)
>
> > I think i have to break the data entry in separate chunks and do it. But not
> > sure how to do that.
>
> In any case Heroku will apparently not let you import the full
> database anyway until we switch to a payed account. If you still have
> troubles with performance issues afterwards, this also won't help.
> What you could do is temporarily copy or move a few folders (say,
> math, science and python) and run portindex just on those. This will
> give you a smaller number of ports to work with, but still sufficient
> to figure out what other issues you'll need to deal with (port listing
> will have to be paginated etc.)
>
> It's a good exercise to know what the limitations are. When you play
> with new features and database design, it's always easier to play
> with, say 10-100 ports than with full 15 MB portindex file.
>

This plan sounds good. +1

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 8 May 2018 at 01:12, Vishnu wrote:
> Hi
>
> I will work on all the points mentioned today..
>
> But i have been trying many different ways.
> To get 20k ports onto the database.. The site is crashing.. Timeout error.
> Or the site cant be reached.
> Maximum i got till 9k ports successfully entered.
>
> Tried many ways..
>
> If you have any suggestions do let me know.

I was pretty sure that we would reach the database limit here.

This site
https://devcenter.heroku.com/articles/heroku-postgres-plans
says that the limit is 10k rows which is not anywhere near enough and
fully consistent with your observation. The plan that lets you have
10M rows is 9 USD per month which I gladly pay if that would solve the
issues (I'm not sure if those are the only ones, we might not have
sufficient memory etc.)

(I also remembered that I might have access to create a clean virtual
machine, but I need a couple of days to physically reach a place where
I have the additional info. That would require setting up the whole
machine manually of course, but we need to do that for the final
deployment anyway.)

> I think i have to break the data entry in separate chunks and do it. But not
> sure how to do that.

In any case Heroku will apparently not let you import the full
database anyway until we switch to a payed account. If you still have
troubles with performance issues afterwards, this also won't help.
What you could do is temporarily copy or move a few folders (say,
math, science and python) and run portindex just on those. This will
give you a smaller number of ports to work with, but still sufficient
to figure out what other issues you'll need to deal with (port listing
will have to be paginated etc.)

It's a good exercise to know what the limitations are. When you play
with new features and database design, it's always easier to play
with, say 10-100 ports than with full 15 MB portindex file.



Please check if you have commit access to macports-webapp now and try
to start putting there the documentation, plan, database design,
sample html page(s) etc. The repository needs to have a clear
README.md describing its purpose + pointers to additional
documentation inside the repository with more details, like database
design etc.

Once you transfer the database scheme into some markdown tables (you
can use any alternative way as long as it's easy for you to modify and
easy to review), we'll discuss whether anything else needs
modifications.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Tue, May 8, 2018 at 12:01 AM, Vishnu  wrote:

> Hello
>
> Thanks for the repository link.
>
> so my current email should be added as the collaborator for me to
> contribute right?
> or should i use the new macports id.
>

I don't remember exactly but I guess yes, you would need to be added as a
collaborator to start contributing to the repository.

In the meantime, feel free to push your code to your local repository.

in that case do i have to make a new account or what?
> not clear about that..
>
>
No, you just add the macports email id to your existing
github profile (See Links:[6] at the end).

You would also want to update your git config (you could use --local
to keep the config local to the directory that you are working on)
to use your macports id though.

See: https://trac.macports.org/wiki/WorkingWithGit#setup



> Also writing down some points we discussed for the community to know :
>
> 1) make the repository-  already done.
> 2) Upload a file with DB schema .
>

There should be some way that you can create tables/schema
using markdown. But feel free to use whatever works better for you.


> 3) Set milestones in github.
> 4) Learn about unit tests
>

To know more about unit tests in brief I found a brief answer on SO [1].
You can
find a detailed article on Tutorials point [4] and wikipedia [5].


> 5) Really try to implement API feature.
>

It would help to come up with a simple design on what the API would do.
This would
help you during the coding phase. We can always refine it as we progress.

7) Discuss about what commit needs to be shown on the port page.
>

As discussed, we could keep this topic for discussion at a later point of
time.


> 8) Add columns for port and intended OS  , Port and build possible OS
>


Thanks for documenting the points discussed. I have added some of my
thoughts inline.


>
> jackson i would require the milestones page you talked about.
>
>
I was talking about something like [2]. To get more idea about milestones,
you can refer [3].

Links:
[1] https://stackoverflow.com/a/3258768/1970068
[2] https://github.com/pypa/warehouse/milestones
[3] https://help.github.com/articles/about-milestones/
[4] https://www.tutorialspoint.com/software_testing_dictionary/
unit_testing.htm
[5] https://en.wikipedia.org/wiki/Unit_testing
[6] https://help.github.com/articles/adding-an-email-address-to-
your-github-account/

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Thanks for the repository link.

so my current email should be added as the collaborator for me to
contribute right?
or should i use the new macports id.
in that case do i have to make a new account or what?
not clear about that..


Also writing down some points we discussed for the community to know :

1) make the repository-  already done.
2) Upload a file with DB schema .
3) Set milestones in github.
4) Learn about unit tests
5) Really try to implement API feature.
7) Discuss about what commit needs to be shown on the port page.
8) Add columns for port and intended OS  , Port and build possible OS


jackson i would require the milestones page you talked about.


Thanks



On 7 May 2018 at 23:49, Jackson Isaac  wrote:

> Hi Vishnu,
>
> On Mon, May 7, 2018 at 12:22 AM, Bradley Giesbrecht 
> wrote:
>
>> > On Apr 30, 2018, at 10:34 AM, Joshua Root  wrote:
>> >
>> > On 2018-5-1 02:47 , Jackson Isaac wrote:
>> >> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu 
>> wrote:
>> >>> Hey
>> >>>
>> >>> Repository names:
>> >>> MacPorts-features?
>> >>> Macportsinfo?
>> >>> Or just simple
>> >>> gsoc18?
>> >>>
>> >>>
>> >>
>> >> I was thinking about macports-portstatistics or macports-portstats.
>> >
>> > This is going to cover more than just stats; it's intended to fill the
>> > role that was envisioned for MPWA, so I think it would be appropriate to
>> > use that name.
>>
>> macports-webapp?
>>
>>
> New repository for GSoC 2018 - macports-webapp is set up now.
>
> It is accessible at [1]. Let us know if you have any queries. You would
> also
> require @macports id. I'll send a follow up mail on the thread.
>
> Links:
> [1] https://github.com/macports/macports-webapp
>
> @neverpanic, thanks for the quick help :)
>
> Regards,
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Clemens Lang]

Hi,

On Mon, May 07, 2018 at 10:31:09AM +0200, Mojca Miklavec wrote:
> On 6 May 2018 at 20:52, Bradley Giesbrecht wrote:
> > macports-webapp?
> 
> Let's go with that one then.

Created.

-- 
Clemens

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Mon, May 7, 2018 at 12:22 AM, Bradley Giesbrecht 
wrote:

> > On Apr 30, 2018, at 10:34 AM, Joshua Root  wrote:
> >
> > On 2018-5-1 02:47 , Jackson Isaac wrote:
> >> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu  wrote:
> >>> Hey
> >>>
> >>> Repository names:
> >>> MacPorts-features?
> >>> Macportsinfo?
> >>> Or just simple
> >>> gsoc18?
> >>>
> >>>
> >>
> >> I was thinking about macports-portstatistics or macports-portstats.
> >
> > This is going to cover more than just stats; it's intended to fill the
> > role that was envisioned for MPWA, so I think it would be appropriate to
> > use that name.
>
> macports-webapp?
>
>
New repository for GSoC 2018 - macports-webapp is set up now.

It is accessible at [1]. Let us know if you have any queries. You would also
require @macports id. I'll send a follow up mail on the thread.

Links:
[1] https://github.com/macports/macports-webapp

@neverpanic, thanks for the quick help :)

Regards,
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 6 May 2018 at 20:52, Bradley Giesbrecht wrote:
>
> macports-webapp?

Let's go with that one then.

Joshua, or anyone else from portmgr, can you please create a new
repository inside macports org?


In case anyone else wants to join, we'll have an audio meeting tonight
at 17:00 UTC (probably on google hangouts). If you would like to
attend, drop me and/or Jackson an email.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Bradley Giesbrecht]

> On Apr 30, 2018, at 10:34 AM, Joshua Root  wrote:
> 
> On 2018-5-1 02:47 , Jackson Isaac wrote:
>> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu  wrote:
>>> Hey
>>> 
>>> Repository names:
>>> MacPorts-features?
>>> Macportsinfo?
>>> Or just simple
>>> gsoc18?
>>> 
>>> 
>> 
>> I was thinking about macports-portstatistics or macports-portstats.
> 
> This is going to cover more than just stats; it's intended to fill the
> role that was envisioned for MPWA, so I think it would be appropriate to
> use that name.

macports-webapp?

—
Brad

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Yes.
I will remove case sensitive.
Right now the search works with only if any port beginswith those
characters.


Yes 7th may ITC my exams will be over at 5:00.

Time mentioned by you is fine.
I available any day.

Thanks

On Sun, May 6, 2018, 9:45 PM Jackson Isaac  wrote:

> Hi Vishnu,
>
> On Sun, May 6, 2018 at 3:52 PM, Vishnu  wrote:
>
>> Hello
>>
>> Sorry for being inactive.
>> Had back to back exams .
>>
>> Got free today.
>>
>> I implemented a new search feature in my app .
>> Pushed the changes to heroku too.
>> Please go through :
>> sleepy-wave-33400.herokuapp.com/port/
>>
>> There you can search any port with its name.
>> Even just starting few characters would work.
>>
>>
> The search seems to be case-sensitive. For e.g., I entered 'app' but
> didn't get any matches whereas I had to enter 'App' to get some matches
> that too that begin with the entered text. It doesn't look for App in
> between the
> portname i.e., 'BGHUDAppKit'
>
> Also it would be great to populate the DB with the complete portindex. The
> installation
> statistics did you fetch from mpstats or manually entered some values ?
> Looks like they
> are from 2012.
>
> IIRC, your exam gets over tomorrow i.e., 7th May 2018 ? It would be great
> to setup
> an introduction meeting with mentors and student.
>
> I am okay with setting up a meeting any day after 1630 UTC (2200 IST). I
> would not be available the
> coming weekend i.e., from 10th May to 14th May.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Sun, May 6, 2018 at 3:52 PM, Vishnu  wrote:

> Hello
>
> Sorry for being inactive.
> Had back to back exams .
>
> Got free today.
>
> I implemented a new search feature in my app .
> Pushed the changes to heroku too.
> Please go through :
> sleepy-wave-33400.herokuapp.com/port/
>
> There you can search any port with its name.
> Even just starting few characters would work.
>
>
The search seems to be case-sensitive. For e.g., I entered 'app' but
didn't get any matches whereas I had to enter 'App' to get some matches
that too that begin with the entered text. It doesn't look for App in
between the
portname i.e., 'BGHUDAppKit'

Also it would be great to populate the DB with the complete portindex. The
installation
statistics did you fetch from mpstats or manually entered some values ?
Looks like they
are from 2012.

IIRC, your exam gets over tomorrow i.e., 7th May 2018 ? It would be great
to setup
an introduction meeting with mentors and student.

I am okay with setting up a meeting any day after 1630 UTC (2200 IST). I
would not be available the
coming weekend i.e., from 10th May to 14th May.

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Sorry for being inactive.
Had back to back exams .

Got free today.

I implemented a new search feature in my app .
Pushed the changes to heroku too.
Please go through :
sleepy-wave-33400.herokuapp.com/port/

There you can search any port with its name.
Even just starting few characters would work.

Thanks

On 30 April 2018 at 23:30, Mojca Miklavec  wrote:

> On 30 April 2018 at 18:47, Jackson Isaac wrote:
> > On Mon, Apr 30, 2018 at 10:06 PM, Vishnu wrote:
> >> Hey
> >>
> >> Repository names:
> >> MacPorts-features?
> >> Macportsinfo?
> >> Or just simple
> >> gsoc18?
>
> No, gsoc18 is a bad name since the idea is to use the project
> afterwards as well and I don't like to rename it two years from now.
> (We can decide to rename it one month from now, but not once people
> start forking it etc.)
>
> > I was thinking about macports-portstatistics or macports-portstats.
>
> Or "macports-portinfo" as something inbetween? I'm somewhat reluctant
> about "stat[istics]" because it's about much more than statistics.
>
> >> And also in the long term. Parser would be run only once in forever or
> say a
> >> year.
>
> No way. Parser needs to be run after each commit. Initially it could
> be once per hour or so to simplify things and avoid hooks etc., but in
> the long term it needs to be run immediately after GitHub notifies you
> about the update.
>
> >> And before running the original db must be cleared.
> >>
> >> And then obvious question arise. What if a new port is added in
> macports.
> >
> > You are on a right path I would say. We would want it to update after
> > every port is added/updated
> > and built.
>
> Indeed, if a new port is added, you need to update the database. More
> important: also if a port is removed or if version changes etc.
>
> >> For that i would like to know the mechanics of new port addition.
>
> See one of recent pull requests:
>
> https://github.com/macports/macports-ports/pull/1694/files
>
> This commit will create a new Portfile with four subports
> (py-waitress, py27-waitress, py35-waitress, py36-waitress), update a
> bunch of existing ports and add a bunch of new subports to existing
> Portfiles (py35-paste, py36-paste, ...)
>
> We could at the same time declare a few ports obsolete or delete some.
>
> You get all that info from Portindex (it could be useful if we could
> get some kind of "differential Portindex"; I'm not absolutely sure how
> port deletions are handled).
>
> >> I think before the new port is added. It is obviously built before
> right?
>
> No. It is first added. Only once it is added it can be built on the
> buildbot (a commit to repository triggers a build on the buildbot).
>
> >> So
> >> maybe we can add some code there to update the table with a row when new
> >> port is added.
> >
> > When a port is submitted, it is built by Travis CI and also our
> > buildbots,
>
> Travis only builds pull requests, not actually commits.
>
> > which builds
> > the port for mac 10.6 to 10.13 versions and gives the build status. We
> > would be more
> > interested in the status provided by the buildbot.
>
> I would ignore Travis builds for now anyway. It could be interesting
> to add pointers to open pull requests for a particular port, but not
> right now. Evidently your code needs to first update the database to
> get the list of all new ports, information about successful or failed
> builds would come later from the buildbot.
>
> > There could be a trigger or a listener which runs whenever a build job
> > is initiated and then
> > updates the DB on completion. I believe there would be some mechanism
> > to know the status
> > of each job.
>
> You can access the API from the buildbot to ask for status. But if you
> don't have an entry for a particular port first, you might run into
> troubles if you'll want to add build results without being able to
> reference a port with that name.
>
> > Maybe someone more familiar with buildbot could share some insights ?
>
> In any case it would make a lot more sense to start with tens of
> thousands old builds first and only once that is done add the latest
> information as soon as it's available. There is a high chance that
> things will change if we switch to buildbot version one (which might
> happen in the meantime).
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 30 April 2018 at 18:47, Jackson Isaac wrote:
> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu wrote:
>> Hey
>>
>> Repository names:
>> MacPorts-features?
>> Macportsinfo?
>> Or just simple
>> gsoc18?

No, gsoc18 is a bad name since the idea is to use the project
afterwards as well and I don't like to rename it two years from now.
(We can decide to rename it one month from now, but not once people
start forking it etc.)

> I was thinking about macports-portstatistics or macports-portstats.

Or "macports-portinfo" as something inbetween? I'm somewhat reluctant
about "stat[istics]" because it's about much more than statistics.

>> And also in the long term. Parser would be run only once in forever or say a
>> year.

No way. Parser needs to be run after each commit. Initially it could
be once per hour or so to simplify things and avoid hooks etc., but in
the long term it needs to be run immediately after GitHub notifies you
about the update.

>> And before running the original db must be cleared.
>>
>> And then obvious question arise. What if a new port is added in macports.
>
> You are on a right path I would say. We would want it to update after
> every port is added/updated
> and built.

Indeed, if a new port is added, you need to update the database. More
important: also if a port is removed or if version changes etc.

>> For that i would like to know the mechanics of new port addition.

See one of recent pull requests:

https://github.com/macports/macports-ports/pull/1694/files

This commit will create a new Portfile with four subports
(py-waitress, py27-waitress, py35-waitress, py36-waitress), update a
bunch of existing ports and add a bunch of new subports to existing
Portfiles (py35-paste, py36-paste, ...)

We could at the same time declare a few ports obsolete or delete some.

You get all that info from Portindex (it could be useful if we could
get some kind of "differential Portindex"; I'm not absolutely sure how
port deletions are handled).

>> I think before the new port is added. It is obviously built before right?

No. It is first added. Only once it is added it can be built on the
buildbot (a commit to repository triggers a build on the buildbot).

>> So
>> maybe we can add some code there to update the table with a row when new
>> port is added.
>
> When a port is submitted, it is built by Travis CI and also our
> buildbots,

Travis only builds pull requests, not actually commits.

> which builds
> the port for mac 10.6 to 10.13 versions and gives the build status. We
> would be more
> interested in the status provided by the buildbot.

I would ignore Travis builds for now anyway. It could be interesting
to add pointers to open pull requests for a particular port, but not
right now. Evidently your code needs to first update the database to
get the list of all new ports, information about successful or failed
builds would come later from the buildbot.

> There could be a trigger or a listener which runs whenever a build job
> is initiated and then
> updates the DB on completion. I believe there would be some mechanism
> to know the status
> of each job.

You can access the API from the buildbot to ask for status. But if you
don't have an entry for a particular port first, you might run into
troubles if you'll want to add build results without being able to
reference a port with that name.

> Maybe someone more familiar with buildbot could share some insights ?

In any case it would make a lot more sense to start with tens of
thousands old builds first and only once that is done add the latest
information as soon as it's available. There is a high chance that
things will change if we switch to buildbot version one (which might
happen in the meantime).

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Joshua Root]

On 2018-5-1 02:47 , Jackson Isaac wrote:
> On Mon, Apr 30, 2018 at 10:06 PM, Vishnu  wrote:
>> Hey
>>
>> Repository names:
>> MacPorts-features?
>> Macportsinfo?
>> Or just simple
>> gsoc18?
>>
>>
> 
> I was thinking about macports-portstatistics or macports-portstats.

This is going to cover more than just stats; it's intended to fill the
role that was envisioned for MPWA, so I think it would be appropriate to
use that name.

- Josh

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Mon, Apr 30, 2018 at 10:06 PM, Vishnu  wrote:
> Hey
>
> Repository names:
> MacPorts-features?
> Macportsinfo?
> Or just simple
> gsoc18?
>
>

I was thinking about macports-portstatistics or macports-portstats.

>
> I will look into the feedback.
> Also i am currently using create.
>
> And also in the long term. Parser would be run only once in forever or say a
> year.

If we run the parser once a year, wouldn't it be outdated pretty soon ?

> And before running the original db must be cleared.
>
> And then obvious question arise. What if a new port is added in macports.
>

You are on a right path I would say. We would want it to update after
every port is added/updated
and built.

> For that i would like to know the mechanics of new port addition.
>
> I think before the new port is added. It is obviously built before right? So
> maybe we can add some code there to update the table with a row when new
> port is added.

When a port is submitted, it is built by Travis CI and also our
buildbots, which builds
the port for mac 10.6 to 10.13 versions and gives the build status. We
would be more
interested in the status provided by the buildbot.

There could be a trigger or a listener which runs whenever a build job
is initiated and then
updates the DB on completion. I believe there would be some mechanism
to know the status
of each job.

Maybe someone more familiar with buildbot could share some insights ?

>
> Just brainstorming.
>
>
> Thanks
>
> On Mon, Apr 30, 2018, 2:23 PM Mojca Miklavec  wrote:
>>
>> On 30 April 2018 at 02:22, Vishnu wrote:
>> > Hi
>> >
>> > After hours of fixing error in heroku and my ubuntu .
>> > Finally got to run the so far app.
>>
>> Yay! That's great since it will be much easier for the whole community
>> to test the app much more quickly.
>>
>> Now, it would be great to pick some name for the repository and
>> populate it, to simplify further work.
>>
>> > Here all the ports are listed:
>> > sleepy-wave-33400.herokuapp.com/port/
>> >
>> > When you'll open the page..probable the ports would not be there because
>> > i
>> > cleared the DB.
>> >
>> > To add the ports info into DB..we need to open this url:
>> > sleepy-wave-33400.herokuapp.com/port/parser
>> >
>> > Parsing is done using the portindex.json i have.
>> > For simplicity i have only 6 portfile data in it.
>> >
>> > Once you open sleepy-wave-33400.herokuapp.com/port/
>> > you can click on individual port to get further informant about that
>> > port.
>> > infact u can access this information using :
>>
>> Cool. Some feedback:
>> - the css should go to a separate static file
>> - allowing port id might be a bad idea because there might be a port
>> called "123"
>> - calling parser multiple times populates the database with multiple
>> copies of the same port, you should use something like
>> "create_or_update"
>> - we need to figure out the best way to call the "parser"
>>
>> I might have a few other comments, but the initial purpose was to
>> check if we have an easy way to deploy the development version and
>> this has been achieved.
>>
>> Please come up with proposal(s) for project name, so that we can make
>> an explicit request to the infrastructure team to create a new repo.
>>
>> Random brainstorming ideas: webapp, "port guardian" (in whatever
>> spelling), ...
>>
>> Mojca



-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Repository names:
MacPorts-features?
Macportsinfo?
Or just simple
gsoc18?



I will look into the feedback.
Also i am currently using create.

And also in the long term. Parser would be run only once in forever or say
a year.
And before running the original db must be cleared.

And then obvious question arise. What if a new port is added in macports.

For that i would like to know the mechanics of new port addition.

I think before the new port is added. It is obviously built before right?
So maybe we can add some code there to update the table with a row when new
port is added.

Just brainstorming.


Thanks

On Mon, Apr 30, 2018, 2:23 PM Mojca Miklavec  wrote:

> On 30 April 2018 at 02:22, Vishnu wrote:
> > Hi
> >
> > After hours of fixing error in heroku and my ubuntu .
> > Finally got to run the so far app.
>
> Yay! That's great since it will be much easier for the whole community
> to test the app much more quickly.
>
> Now, it would be great to pick some name for the repository and
> populate it, to simplify further work.
>
> > Here all the ports are listed:
> > sleepy-wave-33400.herokuapp.com/port/
> >
> > When you'll open the page..probable the ports would not be there because
> i
> > cleared the DB.
> >
> > To add the ports info into DB..we need to open this url:
> > sleepy-wave-33400.herokuapp.com/port/parser
> >
> > Parsing is done using the portindex.json i have.
> > For simplicity i have only 6 portfile data in it.
> >
> > Once you open sleepy-wave-33400.herokuapp.com/port/
> > you can click on individual port to get further informant about that
> port.
> > infact u can access this information using :
>
> Cool. Some feedback:
> - the css should go to a separate static file
> - allowing port id might be a bad idea because there might be a port
> called "123"
> - calling parser multiple times populates the database with multiple
> copies of the same port, you should use something like
> "create_or_update"
> - we need to figure out the best way to call the "parser"
>
> I might have a few other comments, but the initial purpose was to
> check if we have an easy way to deploy the development version and
> this has been achieved.
>
> Please come up with proposal(s) for project name, so that we can make
> an explicit request to the infrastructure team to create a new repo.
>
> Random brainstorming ideas: webapp, "port guardian" (in whatever
> spelling), ...
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 30 April 2018 at 02:22, Vishnu wrote:
> Hi
>
> After hours of fixing error in heroku and my ubuntu .
> Finally got to run the so far app.

Yay! That's great since it will be much easier for the whole community
to test the app much more quickly.

Now, it would be great to pick some name for the repository and
populate it, to simplify further work.

> Here all the ports are listed:
> sleepy-wave-33400.herokuapp.com/port/
>
> When you'll open the page..probable the ports would not be there because i
> cleared the DB.
>
> To add the ports info into DB..we need to open this url:
> sleepy-wave-33400.herokuapp.com/port/parser
>
> Parsing is done using the portindex.json i have.
> For simplicity i have only 6 portfile data in it.
>
> Once you open sleepy-wave-33400.herokuapp.com/port/
> you can click on individual port to get further informant about that port.
> infact u can access this information using :

Cool. Some feedback:
- the css should go to a separate static file
- allowing port id might be a bad idea because there might be a port
called "123"
- calling parser multiple times populates the database with multiple
copies of the same port, you should use something like
"create_or_update"
- we need to figure out the best way to call the "parser"

I might have a few other comments, but the initial purpose was to
check if we have an easy way to deploy the development version and
this has been achieved.

Please come up with proposal(s) for project name, so that we can make
an explicit request to the infrastructure team to create a new repo.

Random brainstorming ideas: webapp, "port guardian" (in whatever spelling), ...

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Hello

After hours of fixing error in heroku and my ubuntu .
Finally got to run the so far app .



Here all the ports are listed:
sleepy-wave-33400.herokuapp.com/port/

When you'll open the page..probable the ports would not be there because i
cleared the DB.

To add the ports info into DB..we need to open this url:
sleepy-wave-33400.herokuapp.com/port/parser

Parsing is done using the portindex.json i have.
For simplicity i have only 6 portfile data in it.

Once you open sleepy-wave-33400.herokuapp.com/port/
you can click on individual port to get further informant about that port.
infact u can access this information using :

sleepy-wave-33400.herokuapp.com/port/whatever_is_the_port_name
sleepy-wave-33400.herokuapp.com/port/whatever_is_the_port_id


Thanks


On 30 April 2018 at 02:14, Mojca Miklavec  wrote:

> On 29 April 2018 at 19:34, Vishnu wrote:
> > Hey
> >
> > I wanted to work on finalising the database. What all final columns in
> each
> > table.
>
> This is a creative process that should in the first place be carefully
> crafted by you - of course with sufficient input and feedback from our
> side.
>
> > Take your final suggestions.
>
> I'm pretty sure that things will change during the course of the
> summer once you figure out that there might be additional
> requirements, that you might need additional fields to achieve what
> you wanted etc. Django usually lets you transition from one schema to
> a slightly different one in a relatively painless way.
>
> But there's one thing that we should probably discuss together with
> other members, and that is, to what extent to support:
> - historic information about ports (as versions, dependencies,
> maintainers etc. change, ports become obsolete, ...)
> - the fact that different OS versions might support different ports or
> different versions of ports
>
> Vishnu, I would like to suggest to put your Excel schematic to some
> easy-to-read-and-edit document in that github repository. This could
> be some Markdown table, or anything else. You can search a bit for
> programs that let you easily create ERD (Entity Relation Diagram)
> diagrams. This document will then serve at least three purposes:
> - brainstorming with us about the most reasonable schematic, improvements
> etc.
> - enormously help you when you start coding you app
> - as a documentation that will allow anyone else to understand and
> improve the program later on without reading the code
>
> > https://www.macports.org/ports.php?by=platform=linux
> > How does this work?
> > where is the database?
> > how are queries relayed?
> > where is the actual code for this?
>
> There is a PostgreSQL database on the server run by Clemens and Rainer
> in Germany (most likely the same server where you app would eventually
> run).
>
> I would be grateful if someone from the infrastructure team could
> correct me here in case I'm giving you wrong pointers, but I suspect
> that the database gets populated with this script:
> https://github.com/macports/macports-infrastructure/blob/
> master/jobs/PortIndex2PGSQL.tcl
> by converting data from PortIndex to SQL and filling the database with it.
>
> I suspect the rendering is done with
> https://github.com/macports/macports-www/blob/master/ports.php
>
> You may optionally take a glimpse at various repositories at
> https://github.com/macports/
> (ignore user repositories), quickly look at what is there and ask if
> there's something more that you would like to know.
>
> This can give you some ideas, but I assume you would populate the
> database in pretty much the same way as you did during your coding
> challenge, and according to the database design developed during
> proposal writing and during community bonding period.
>
> > Also for my project what database should i use.
> > i recall postgre sql.or something else?
>
> Yes, PostgreSQL is probably the best OpenSource relational database
> (or at least it was some years ago), my preference would be to use
> that one. SQLite is too limited. MySQL (derivatives?) would be
> acceptable, but PG sounds better. Commercial databases are out of
> question.
>
> > So should i start creating tickets?
> > like ..at least putting putting up my basic milestones.
>
> Yes, I would like to see the tickets with milestones once we have the
> repository. Things like "transfer project proposal to documentation"
> with a milestone bonding period could also be stuff that ends up
> there.
>
> > Also you said :
> > (c.3) Plan the API to get the data from the database in JSON format
> > (so that someone else could write an independent app with the same
> > display functionality).
> > More in a separate email, I guess.
> >
> > I'll do this once i finish my database.
>
> OK.
>
> Things that need to be planned are:
> - database schematic (the one from proposal was quite ok already, but
> we need to refine to what extent to support historic entries etc.)
> - the list of different sites (first the list 

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 29 April 2018 at 19:34, Vishnu wrote:
> Hey
>
> I wanted to work on finalising the database. What all final columns in each
> table.

This is a creative process that should in the first place be carefully
crafted by you - of course with sufficient input and feedback from our
side.

> Take your final suggestions.

I'm pretty sure that things will change during the course of the
summer once you figure out that there might be additional
requirements, that you might need additional fields to achieve what
you wanted etc. Django usually lets you transition from one schema to
a slightly different one in a relatively painless way.

But there's one thing that we should probably discuss together with
other members, and that is, to what extent to support:
- historic information about ports (as versions, dependencies,
maintainers etc. change, ports become obsolete, ...)
- the fact that different OS versions might support different ports or
different versions of ports

Vishnu, I would like to suggest to put your Excel schematic to some
easy-to-read-and-edit document in that github repository. This could
be some Markdown table, or anything else. You can search a bit for
programs that let you easily create ERD (Entity Relation Diagram)
diagrams. This document will then serve at least three purposes:
- brainstorming with us about the most reasonable schematic, improvements etc.
- enormously help you when you start coding you app
- as a documentation that will allow anyone else to understand and
improve the program later on without reading the code

> https://www.macports.org/ports.php?by=platform=linux
> How does this work?
> where is the database?
> how are queries relayed?
> where is the actual code for this?

There is a PostgreSQL database on the server run by Clemens and Rainer
in Germany (most likely the same server where you app would eventually
run).

I would be grateful if someone from the infrastructure team could
correct me here in case I'm giving you wrong pointers, but I suspect
that the database gets populated with this script:

https://github.com/macports/macports-infrastructure/blob/master/jobs/PortIndex2PGSQL.tcl
by converting data from PortIndex to SQL and filling the database with it.

I suspect the rendering is done with
https://github.com/macports/macports-www/blob/master/ports.php

You may optionally take a glimpse at various repositories at
https://github.com/macports/
(ignore user repositories), quickly look at what is there and ask if
there's something more that you would like to know.

This can give you some ideas, but I assume you would populate the
database in pretty much the same way as you did during your coding
challenge, and according to the database design developed during
proposal writing and during community bonding period.

> Also for my project what database should i use.
> i recall postgre sql.or something else?

Yes, PostgreSQL is probably the best OpenSource relational database
(or at least it was some years ago), my preference would be to use
that one. SQLite is too limited. MySQL (derivatives?) would be
acceptable, but PG sounds better. Commercial databases are out of
question.

> So should i start creating tickets?
> like ..at least putting putting up my basic milestones.

Yes, I would like to see the tickets with milestones once we have the
repository. Things like "transfer project proposal to documentation"
with a milestone bonding period could also be stuff that ends up
there.

> Also you said :
> (c.3) Plan the API to get the data from the database in JSON format
> (so that someone else could write an independent app with the same
> display functionality).
> More in a separate email, I guess.
>
> I'll do this once i finish my database.

OK.

Things that need to be planned are:
- database schematic (the one from proposal was quite ok already, but
we need to refine to what extent to support historic entries etc.)
- the list of different sites (first the list of all sites with
approximate URLs, then add what content you want to end up on each one
of them)
- ideally the API (probably not that much different from the list of sites)
- potentially refine / go into some more details about different
building blocks use to collect the data for the database

I'll try to explain a bit more about the API ...

Let's just put everything to a (set of) document(s) in the repository.
It could be a markdown or asciidoc or whatever other document that's
easy to edit and can easily be viewed in your repository. As I said, I
feel that this would be the easiest way to collect more feedback,
collect all ideas at a single place, as a reference while you will be
developing and as a documentation and reference that would be useful
to other developers after the GSOC is over.

> I tried to understand a bit about heroku.It'll work when needed.Not an issue
> i guess.

I'm not even sure that heroku is the best choice since I have no
experience with the service. It's just nice to have some 

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Do let me know when the repository is gonna be made. I can start using it.

I wanted to work on finalising the database. What all final columns in each
table.
Take your final suggestions.

https://www.macports.org/ports.php?by=platform=linux
How does this work?
where is the database?
how are queries relayed?
where is the actual code for this?

Also for my project what database should i use.
i recall postgre sql.or something else?

So should i start creating tickets?
like ..at least putting putting up my basic milestones.

Also you said :
(c.3) Plan the API to get the data from the database in JSON format
(so that someone else could write an independent app with the same
display functionality).
More in a separate email, I guess.

I'll do this once i finish my database.


I tried to understand a bit about heroku.It'll work when needed.Not an
issue i guess.


Thanks




On 29 April 2018 at 20:37, Mojca Miklavec  wrote:

> Dear Vishnu,
>
> On 29 April 2018 at 16:45, Vishnu wrote:
> > Hi
> >
> > Jackson i have already replied to the mail regarding username.
> >
> > Mojca, I was able to edit the file and entered values manually in the
> code
> > of mpstats.
> >
> > vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
> show
> > Would submit the following data to
> > http://stats.macports.neverpanic.de/submissions:
> > {
> >   "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
> >   "os": {
> > "macports_version": "2.4.99",
> > "osx_version": "10.15",
> > "os_arch": "i386",
> > "os_platform": "linux",
> > "build_arch": "ppc",
> > "gcc_version": "5.4.0",
> > "xcode_version": "none"
> >   },
> >   "active_ports": [
> > {"name": "ncurses", "version": "6.1_0"},
> > {"name": "libedit", "version": "20170329-3.1_2"},
> > {"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
> >   ],
> >   "inactive_ports": []
> > }
> > vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
> > submit
> > Submitting data to http://stats.macports.neverpanic.de/submissions ...
> > Success.
> >
> > So even upon submitting .There is no update on the site.
> > idk why.
>
> I'm almost equally not familiar with the code as you are (I did make
> some minor changes a while ago). While we could start debugging it now
> (with only Clemens being able to access the logs and deploying any
> changes), I believe this would mostly be a waste of time since a good
> implementation of your project would eventually make this site
> obsolete anyway. Of course it makes sense to get familiar with rails
> application serving the statistics page, but spending a lot of time
> fixing it is simply not worth the trouble in my opinion.
>
> https://github.com/macports/macports-contrib/tree/gsoc11-statistics/
>
> Continue the planning phase (or any other task) would likely be much
> more useful at this point.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 29 April 2018 at 16:45, Vishnu wrote:
> Hi
>
> Jackson i have already replied to the mail regarding username.
>
> Mojca, I was able to edit the file and entered values manually in the code
> of mpstats.
>
> vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
> Would submit the following data to
> http://stats.macports.neverpanic.de/submissions:
> {
>   "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
>   "os": {
> "macports_version": "2.4.99",
> "osx_version": "10.15",
> "os_arch": "i386",
> "os_platform": "linux",
> "build_arch": "ppc",
> "gcc_version": "5.4.0",
> "xcode_version": "none"
>   },
>   "active_ports": [
> {"name": "ncurses", "version": "6.1_0"},
> {"name": "libedit", "version": "20170329-3.1_2"},
> {"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
>   ],
>   "inactive_ports": []
> }
> vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
> submit
> Submitting data to http://stats.macports.neverpanic.de/submissions ...
> Success.
>
> So even upon submitting .There is no update on the site.
> idk why.

I'm almost equally not familiar with the code as you are (I did make
some minor changes a while ago). While we could start debugging it now
(with only Clemens being able to access the logs and deploying any
changes), I believe this would mostly be a waste of time since a good
implementation of your project would eventually make this site
obsolete anyway. Of course it makes sense to get familiar with rails
application serving the statistics page, but spending a lot of time
fixing it is simply not worth the trouble in my opinion.

https://github.com/macports/macports-contrib/tree/gsoc11-statistics/

Continue the planning phase (or any other task) would likely be much
more useful at this point.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Jackson i have already replied to the mail regarding username.

Mojca, I was able to edit the file and entered values manually in the code
of mpstats.

vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
Would submit the following data to
http://stats.macports.neverpanic.de/submissions:
{
  "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
  "os": {
"macports_version": "2.4.99",
"osx_version": "10.15",
"os_arch": "i386",
"os_platform": "linux",
"build_arch": "ppc",
"gcc_version": "5.4.0",
"xcode_version": "none"
  },
  "active_ports": [
{"name": "ncurses", "version": "6.1_0"},
{"name": "libedit", "version": "20170329-3.1_2"},
{"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
  ],
  "inactive_ports": []
}
vishnu@vishnupc:~$ /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
submit
Submitting data to http://stats.macports.neverpanic.de/submissions ...
Success.


So even upon submitting .There is no update on the site.
idk why.

Thanks
Vishnu

On 29 April 2018 at 15:19, Mojca Miklavec  wrote:

> On 29 April 2018 at 10:17, Jackson Isaac wrote:
> > On Sun, Apr 29, 2018 at 1:21 PM, Mojca Miklavec wrote:
> >>
> >>
> >> I would say that the next most reasonable step (apart from any other
> >> thing from the list I sent in the beginning) would be to create a
> >> repository and put all the documentation / plan / milestones / sample
> >> page etc. to that repository.
> >> This means:
> >> - deciding for the name
> >
> > We can go with something like 'macports-buildstatistics' for repo name.
>
> But keep in mind that this project will now contain a lot more than
> just build statistics. It's:
> - portindex
> - build summary
> - usage statistics
> - whether the port is outdated
> - etc.
>
> So "buildstatistics" sounds too narrow.
>
> > If it would be a branch under /macports-base then
> 'gsoc18-buildstatistics'.
>
> Having it under base would be a bad idea since it will be completely
> independent.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 29 April 2018 at 10:17, Jackson Isaac wrote:
> On Sun, Apr 29, 2018 at 1:21 PM, Mojca Miklavec wrote:
>>
>>
>> I would say that the next most reasonable step (apart from any other
>> thing from the list I sent in the beginning) would be to create a
>> repository and put all the documentation / plan / milestones / sample
>> page etc. to that repository.
>> This means:
>> - deciding for the name
>
> We can go with something like 'macports-buildstatistics' for repo name.

But keep in mind that this project will now contain a lot more than
just build statistics. It's:
- portindex
- build summary
- usage statistics
- whether the port is outdated
- etc.

So "buildstatistics" sounds too narrow.

> If it would be a branch under /macports-base then 'gsoc18-buildstatistics'.

Having it under base would be a bad idea since it will be completely
independent.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Sun, Apr 29, 2018 at 1:21 PM, Mojca Miklavec  wrote:
>
>
> I would say that the next most reasonable step (apart from any other
> thing from the list I sent in the beginning) would be to create a
> repository and put all the documentation / plan / milestones / sample
> page etc. to that repository.
> This means:
> - deciding for the name

We can go with something like 'macports-buildstatistics' for repo name.
If it would be a branch under /macports-base then 'gsoc18-buildstatistics'.

Although having another repository under /macports makes more sense, since
the project would be unrelated to what macports-base does.

> - asking someone with permission to create a repository inside
> macports org (and make sure that Vishnu has commit rights on that
> repository)

I believe the PortMgr would be able to create the repository for us ?
I don't have the
sufficient permission to create repo under /macports is what GitHub tells me.

As for commit rights, I had sent out an email on 26 April for getting
@macports id for Vishnu.

Vishnu,
Could you please search for 'Request for macports id for GSoC 2018 student'
thread in your mailbox and reply with your preference for the username ?

> - deciding how exactly this should be done
>
> I suggest to do most of development in a separate development branch
> and make sure that whatever ends up in master gets a review. Or
> something similar. We need a way to distinguish reviewed code from
> playground.
>

Having separate branches for testing and playing around sounds good to me.
We can then cherry pick the commits and have reviewed code on master.

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 29 April 2018 at 08:29, Jackson Isaac wrote:
> On Sun, Apr 29, 2018 at 7:36 AM, Vishnu wrote:
>> Hey
>>
>> Thanks a lot.
>> Now it works.
>> I dont know why it was not working earlier.
>>
>> And jackson . I ran both the commands.
>> root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
>> Would submit the following data to
>> http://stats.macports.neverpanic.de/submissions:
>> {
>>   "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
>>   "os": {
>> "macports_version": "2.4.99",
>> "osx_version": "",
>> "os_arch": "i386",
>> "os_platform": "linux",
>> "build_arch": "",
>> "gcc_version": "5.4.0",
>> "xcode_version": "none"
>>   },
>>   "active_ports": [
>> {"name": "ncurses", "version": "6.1_0"},
>> {"name": "libedit", "version": "20170329-3.1_2"},
>> {"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
>>   ],
>>   "inactive_ports": []
>> }
>> root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats submit
>> Submitting data to http://stats.macports.neverpanic.de/submissions ...
>> Success.

Great, that's good news.

You might want to make sure that /opt/local/bin stays in your PATH (by
modifying ~/.bashrc or ~/.bash_profile or whatever file is used to set
the environment).

> That's great :) But looks like the mpstats page [1] didn't add Linux
> to the statistics.
>
> This is most likely due to the reason that mpstats only checks for mac
> OS version ?
> Try to find a way to read Linux version if the installed os is not darwin 
> based.
>
> Though 'os_platform' does say 'linux'. Probably the statistics code
> [2] doesn't recognize
> linux as a platform ?

In my view the main point of this "exercise" was to:
- make sure that Vishnu is able to play with statistics submission and
also submit to his server
- prepare the necessary tools in case he wants to modify mpstats in
some way in the future

Trying to debug why the old statistics page doesn't recognise the
submission is beyond the scope of this exercise in my opinion and
solving it won't really help anyone. Trying to implement the
functionality of adding version detection to linux in base is also
hopeless since each OS will have a different versioning scheme.

In fact what I wanted to ask was whether it makes sense to file a
ticket to report the correct architecture (x86_64 in this case, I
guess), but again not expecting Vishnu to do it.

What could be a nice exercise though is to modify the Portfile and
mpstats itself to fake both osx_version (let's say 10.15 :) and
build_arch and try to submit again. But ... if it works, it works. If
it doesn't work, I would simply ignore it. But that could just as well
be any other task of changing mpstats in some minor way. Just to get
familiar a bit with port editing & upgrade process. The process is
hopefully described in the macports guide (which might be worth
reading or at least glimpsing through anyway,
https://guide.macports.org).


I would say that the next most reasonable step (apart from any other
thing from the list I sent in the beginning) would be to create a
repository and put all the documentation / plan / milestones / sample
page etc. to that repository.
This means:
- deciding for the name
- asking someone with permission to create a repository inside
macports org (and make sure that Vishnu has commit rights on that
repository)
- deciding how exactly this should be done

I suggest to do most of development in a separate development branch
and make sure that whatever ends up in master gets a review. Or
something similar. We need a way to distinguish reviewed code from
playground.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Sun, Apr 29, 2018 at 7:36 AM, Vishnu  wrote:
> Hey
>
> Thanks a lot.
> Now it works.
> I dont know why it was not working earlier.
>
> And jackson . I ran both the commands.
> root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
> Would submit the following data to
> http://stats.macports.neverpanic.de/submissions:
> {
>   "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
>   "os": {
> "macports_version": "2.4.99",
> "osx_version": "",
> "os_arch": "i386",
> "os_platform": "linux",
> "build_arch": "",
> "gcc_version": "5.4.0",
> "xcode_version": "none"
>   },
>   "active_ports": [
> {"name": "ncurses", "version": "6.1_0"},
> {"name": "libedit", "version": "20170329-3.1_2"},
> {"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
>   ],
>   "inactive_ports": []
> }
> root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats submit
> Submitting data to http://stats.macports.neverpanic.de/submissions ...
> Success.
>

That's great :) But looks like the mpstats page [1] didn't add Linux
to the statistics.

This is most likely due to the reason that mpstats only checks for mac
OS version ?
Try to find a way to read Linux version if the installed os is not darwin based.

Though 'os_platform' does say 'linux'. Probably the statistics code
[2] doesn't recognize
linux as a platform ?


[1] http://stats.macports.neverpanic.de/os_statistics
[2] https://github.com/macports/macports-base/tree/gsoc11-statistics

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

Thanks a lot.
Now it works.
I dont know why it was not working earlier.

And jackson . I ran both the commands.
root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats show
Would submit the following data to
http://stats.macports.neverpanic.de/submissions:
{
  "id": "cc545acb-3f3b-4a93-afbd-8a72ef5e50b9",
  "os": {
"macports_version": "2.4.99",
"osx_version": "",
"os_arch": "i386",
"os_platform": "linux",
"build_arch": "",
"gcc_version": "5.4.0",
"xcode_version": "none"
  },
  "active_ports": [
{"name": "ncurses", "version": "6.1_0"},
{"name": "libedit", "version": "20170329-3.1_2"},
{"name": "mpstats", "version": "0.1.7_0", "requested": "true"}
  ],
  "inactive_ports": []
}
root@vishnupc:/# /opt/local/bin/port-tclsh /opt/local/libexec/mpstats submit
Submitting data to http://stats.macports.neverpanic.de/submissions ...
Success.




On 29 April 2018 at 07:29, Jeremy Lavergne  wrote:

> There's one last thing to try regarding paths; that is to try invoking
> the file this way:
> /opt/local/bin/port-tclsh /opt/local/libexec/mpstats
>
>
> If that doesn't work, it may lead to wondering if all of MacPorts is
> installed properly.
>
> For example, I see you're using the root user: did we ensure that all
> the file permissions were correct for when you're not the root user? Or
> have you been using the root user exclusively?
>
>
> On 04/28/2018 09:28 PM, Vishnu wrote:
> > I dont think its about PATH.
> > root@vishnupc:/# /opt/local/libexec/mpstats
> > can't find package macports
> > while executing
> > "package require macports"
> > (file "/opt/local/libexec/mpstats" line 42)
> > root@vishnupc:/# echo $PATH
> > /usr/local/sbin:/opt/local/bin:/usr/local/bin:/usr/sbin:/
> usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> >
> > I tried all combinations of path.
> > Didn't work.
> >
> > what does it mean exactly by macports package?
>
>

Re: Gsoc 18 Project | Collect build statistics

[Jeremy Lavergne]

There's one last thing to try regarding paths; that is to try invoking
the file this way:
/opt/local/bin/port-tclsh /opt/local/libexec/mpstats


If that doesn't work, it may lead to wondering if all of MacPorts is
installed properly.

For example, I see you're using the root user: did we ensure that all
the file permissions were correct for when you're not the root user? Or
have you been using the root user exclusively?


On 04/28/2018 09:28 PM, Vishnu wrote:
> I dont think its about PATH.
> root@vishnupc:/# /opt/local/libexec/mpstats 
> can't find package macports
>     while executing
> "package require macports"
>     (file "/opt/local/libexec/mpstats" line 42)
> root@vishnupc:/# echo $PATH
> /usr/local/sbin:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> 
> I tried all combinations of path.
> Didn't work.
> 
> what does it mean exactly by macports package?

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I dont think its about PATH.
root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)
root@vishnupc:/# echo $PATH
/usr/local/sbin:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

I tried all combinations of path.
Didn't work.

what does it mean exactly by macports package?


On 29 April 2018 at 06:47, Jeremy Lavergne  wrote:

> But also notice how you already have /opt/local/bin in your PATH...
>
> On 04/28/2018 09:16 PM, Jeremy Lavergne wrote:
> > A missing / on /opt
> >
> > On 04/28/2018 08:49 PM, Vishnu wrote:
> >> I added the path.But still same error.
> >>
> >> root@vishnupc:/# /opt/local/libexec/mpstats
> >> can't find package macports
> >> while executing
> >> "package require macports"
> >> (file "/opt/local/libexec/mpstats" line 42)
> >> root@vishnupc:/# echo $PATH
> >> /usr/local/sbin:opt/local/bin/port-tclsh:/opt/local/bin:/
> usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> >>
> >>
> >> On 29 April 2018 at 05:56, Ken Cunningham
> >>  >> > wrote:
> >>
> >>
> >>
> >>> On Apr 28, 2018, at 5:06 PM, Vishnu  >>> > wrote:
> >>>
> >>> am i supposed to install port tclsh ?
> >>>
> >>> There's no port named tclsh.
> >>>
> >>
> >> you should have it
> >>
> >> /opt/local/bin/port-tclsh
> >>
> >> but your path is probably calling
> >>
> >> /usr/bin/tclsh
> >>
> >>
> >> Ken
> >>
> >>
> >
>
>

Re: Gsoc 18 Project | Collect build statistics

[Jeremy Lavergne]

A missing / on /opt

On 04/28/2018 08:49 PM, Vishnu wrote:
> I added the path.But still same error.
> 
> root@vishnupc:/# /opt/local/libexec/mpstats 
> can't find package macports
>     while executing
> "package require macports"
>     (file "/opt/local/libexec/mpstats" line 42)
> root@vishnupc:/# echo $PATH
> /usr/local/sbin:opt/local/bin/port-tclsh:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> 
> 
> On 29 April 2018 at 05:56, Ken Cunningham
>  > wrote:
> 
> 
> 
>> On Apr 28, 2018, at 5:06 PM, Vishnu > > wrote:
>>
>> am i supposed to install port tclsh ?
>>
>> There's no port named tclsh.
>>
> 
> you should have it
> 
> /opt/local/bin/port-tclsh
> 
> but your path is probably calling 
> 
> /usr/bin/tclsh
> 
> 
> Ken
> 
> 

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I added the path.But still same error.

root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)
root@vishnupc:/# echo $PATH
/usr/local/sbin:opt/local/bin/port-tclsh:/opt/local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games


On 29 April 2018 at 05:56, Ken Cunningham 
wrote:

>
>
> On Apr 28, 2018, at 5:06 PM, Vishnu  wrote:
>
> am i supposed to install port tclsh ?
>
> There's no port named tclsh.
>
>
> you should have it
>
> /opt/local/bin/port-tclsh
>
> but your path is probably calling
>
> /usr/bin/tclsh
>
>
> Ken
>
>

Re: Gsoc 18 Project | Collect build statistics

[Ken Cunningham]

> On Apr 28, 2018, at 5:06 PM, Vishnu  wrote:
> 
> am i supposed to install port tclsh ?
> 
> There's no port named tclsh.
> 

you should have it

/opt/local/bin/port-tclsh

but your path is probably calling 

/usr/bin/tclsh


Ken

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

am i supposed to install port tclsh ?

There's no port named tclsh.




On 29 April 2018 at 05:24, Joshua Root  wrote:

> On 2018-4-29 09:36 , Vishnu wrote:
> > Hi
> >
> > I had created the user.
> >
> > But when trying /opt/local/libexec/mpstats
> >
> > root@vishnupc:/# /opt/local/libexec/mpstats
> > can't find package macports
> > while executing
> > "package require macports"
> > (file "/opt/local/libexec/mpstats" line 42)
> >
> > What's wrong.
>
> Your tclsh can't find the 'macports' package. Probably the location of
> port-tclsh is missing from your PATH and it's using /usr/bin/tclsh.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I had created the user.

But when trying /opt/local/libexec/mpstats

root@vishnupc:/# /opt/local/libexec/mpstats
can't find package macports
while executing
"package require macports"
(file "/opt/local/libexec/mpstats" line 42)

What's wrong.


On 26 April 2018 at 01:24, Jackson Isaac  wrote:

> On Thu, Apr 26, 2018 at 1:02 AM, Vishnu  wrote:
> > these are the errors.
> >
> > On 26 April 2018 at 00:59, Vishnu  wrote:
> >>
> >> Yes I did add  'revupgrade_autorun no' line to
> >> /opt/local/etc/macports/macports.conf
> >> with sudo..
> >> I cross checked it..it has been updated.
> >>
>
> It's been quite some time I have touched Ubuntu. From Joshua's mail
> and your logs I understand that macports user was not created.
>
> Could you try running
>
> $ sudo addgroup macports
>
> $ sudo adduser macports macports
>
> I am not exactly sure though if the group name is correct or not.
>
> Let us know if you still get the same errors after following these steps.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 26 April 2018 at 06:44, Jackson Isaac wrote:
> On Thu, Apr 26, 2018 at 3:42 AM, Vishnu wrote:
>> Finally was able to install it.
>> And successfully run mpstats.
>
> Great to hear that! You are making a good progress. Keep it up.
>
>> root@vishnupc:~# /opt/local/bin/port content  mpstats
>> Port mpstats contains:
>>   /Library/LaunchDaemons/org.macports.mpstats.plist
>>
>> /opt/local/etc/LaunchDaemons/org.macports.mpstats/org.macports.mpstats.plist.default
>>   /opt/local/etc/macports/stats.conf
>>   /opt/local/libexec/mpstats
>>
>> Mojca you said there will be a change in os statistics
>> (http://stats.macports.neverpanic.de/os_statistics#os_platform) .
>> I couldn't see any change.

I'm sorry, I forgot about one important aspect. On macOS the port
installs an equivalent of a cron job. That's the file
/Library/LaunchDaemons/org.macports.mpstats.plist
This file has absolutely no meaning or functionality on Linux.

The proper fix would be to:
- enclose the part of functionality that's responsible for this into a
"darwin { ... }" section (not that hard to do, but would not really
solve anything)
- potentially write a replacement functionality, but that's tricky
because each system has a different way of setting up jobs, even
Debian alone has at least two ways (systemd or initd or however that
older one is called, I forgot), not to mention that any *BSD has its
own system in place

> When you run 'port contents ' it lists you the files that
> are installed by that port.
>
> Here you would want to run '/opt/local/libexec/mpstats'.
>
> Basically it shows me two options. But I would like you to explore the
> command and let us know what you infer from them :)
>
> If I give out the exact command you might not be able to try out other
> commands available.

Indeed, here you can run the command manually (feel free to explore
yourself :). It will be by far the easiest workaround and you'll need
to run it manually for predictable testing anyway.

I'm actually pretty curious what exactly the command does on Linux,
I'm not aware of anyone actually testing it on a non-mac platform
earlier. It might be that mpstats will need minor tweaks to work
correctly on Linux, but I hope that at least the platform is reported
correctly.

Take a look at the contents of sysutils/mpstats in the git repository
and try to figure out what each file does. The *.plist files are meant
to tell the OS how to periodically run the commands and can probably
"safely be ignored" on any other OS.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Thu, Apr 26, 2018 at 3:42 AM, Vishnu  wrote:
> Finally was able to install it.
> And successfully run mpstats.

Great to hear that! You are making a good progress. Keep it up.

> root@vishnupc:~# /opt/local/bin/port content  mpstats
> Port mpstats contains:
>   /Library/LaunchDaemons/org.macports.mpstats.plist
>
> /opt/local/etc/LaunchDaemons/org.macports.mpstats/org.macports.mpstats.plist.default
>   /opt/local/etc/macports/stats.conf
>   /opt/local/libexec/mpstats
>
> Mojca you said there will be a change in os statistics
> (http://stats.macports.neverpanic.de/os_statistics#os_platform) .
> I couldn't see any change.
>


When you run 'port contents ' it lists you the files that
are installed by that port.

Here you would want to run '/opt/local/libexec/mpstats'.

Basically it shows me two options. But I would like you to explore the
command and let us know what you infer from them :)

If I give out the exact command you might not be able to try out other
commands available.

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Finally was able to install it.
And successfully run mpstats.
root@vishnupc:~# /opt/local/bin/port content  mpstats
Port mpstats contains:
  /Library/LaunchDaemons/org.macports.mpstats.plist

/opt/local/etc/LaunchDaemons/org.macports.mpstats/org.macports.mpstats.plist.default
  /opt/local/etc/macports/stats.conf
  /opt/local/libexec/mpstats

Mojca you said there will be a change in os statistics (
http://stats.macports.neverpanic.de/os_statistics#os_platform) .
I couldn't see any change.

Thanks

On 26 April 2018 at 01:24, Jackson Isaac  wrote:

> On Thu, Apr 26, 2018 at 1:02 AM, Vishnu  wrote:
> > these are the errors.
> >
> > On 26 April 2018 at 00:59, Vishnu  wrote:
> >>
> >> Yes I did add  'revupgrade_autorun no' line to
> >> /opt/local/etc/macports/macports.conf
> >> with sudo..
> >> I cross checked it..it has been updated.
> >>
>
> It's been quite some time I have touched Ubuntu. From Joshua's mail
> and your logs I understand that macports user was not created.
>
> Could you try running
>
> $ sudo addgroup macports
>
> $ sudo adduser macports macports
>
> I am not exactly sure though if the group name is correct or not.
>
> Let us know if you still get the same errors after following these steps.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Thu, Apr 26, 2018 at 1:02 AM, Vishnu  wrote:
> these are the errors.
>
> On 26 April 2018 at 00:59, Vishnu  wrote:
>>
>> Yes I did add  'revupgrade_autorun no' line to
>> /opt/local/etc/macports/macports.conf
>> with sudo..
>> I cross checked it..it has been updated.
>>

It's been quite some time I have touched Ubuntu. From Joshua's mail
and your logs I understand that macports user was not created.

Could you try running

$ sudo addgroup macports

$ sudo adduser macports macports

I am not exactly sure though if the group name is correct or not.

Let us know if you still get the same errors after following these steps.

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Yes I did add  'revupgrade_autorun no' line to
/opt/local/etc/macports/macports.conf
with sudo..
I cross checked it..it has been updated.

On 26 April 2018 at 00:54, Jackson Isaac  wrote:

> On Thu, Apr 26, 2018 at 12:49 AM, Vishnu  wrote:
> > It is exactly the same error as in previous one.
> >
> >> --->  Cleaning mpstats
> >> --->  Updating database of binaries
> >> Warning: Error determining file type of `/opt/local/bin/captoinfo':
> >> invalid command name "fileIsBinary"
> >> Warning: A file belonging to the `ncurses' port is missing or
> >> unreadable. Consider reinstalling it.
> >> Warning: Error determining file type of `/opt/local/bin/clear': invalid
> >> command name "fileIsBinary"
> >> Warning: A file belonging to the `ncurses' port is missing or
> >> unreadable. Consider reinstalling it.
> >> Warning: Error determining file type of `/opt/local/bin/infocmp':
> >> invalid command name "fileIsBinary"
> >
>
> Did you try adding 'revupgrade_autorun no' line to
> /opt/local/etc/macports/macports.conf ?
> You might need to open the file with sudo.
>
> > vishnu@macports would be fine i guess.
> >
> > i'll join as soon as i get the id.
> >
>
> You can join IRC without @macports id.
>
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

It is exactly the same error as in previous one.

> --->  Cleaning mpstats
> --->  Updating database of binaries
> Warning: Error determining file type of `/opt/local/bin/captoinfo':
> invalid command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/clear': invalid
> command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/infocmp':
> invalid command name "fileIsBinary"

vishnu@macports would be fine i guess.

i'll join as soon as i get the id.

Thanks

On 26 April 2018 at 00:37, Jackson Isaac  wrote:

> Hi Vishnu,
>
> On Thu, Apr 26, 2018 at 12:33 AM, Vishnu  wrote:
> > anyone?
> >
> > On 25 April 2018 at 21:49, Vishnu  wrote:
> >>
> >> Hi
> >>
> >> Still same error..Dont know what to do.
> >>
>
> Could you please paste the error that you are getting somewhere e.g.,
> dpaste.de ?
>
> It would be really helpful if you join IRC channel #macports on freenode.
> I use IRCCloud to stay connected -
> https://www.irccloud.com/irc/freenode/channel/macports
> The link should help you get started.
>
> You would also need a @macports id. Do you have any preference for an
> username ?
>
> >>
> >> On 25 April 2018 at 21:24, Joshua Root  wrote:
> >>>
> >>> On 2018-4-26 01:39 , Vishnu wrote:
> >>> > Hi
> >>> >
> >>> > I think its best if you can create a repository under Macports.
> >>> > I'll be coding there.
> >>> >
> >>> > Also while trying to install mpstats .
> >>> > I am getting these error
> >>> >
> >>> > Warning: configured user/group macports does not exist, will build as
> >>> > root
> >>>
> >>> Our build system only knows how to create users on macOS. You'll have
> to
> >>> create the macports user manually.
> >>>
> >>> > --->  Cleaning mpstats
> >>> > --->  Updating database of binaries
> >>> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
> >>> > invalid command name "fileIsBinary"
> >>> > Warning: A file belonging to the `ncurses' port is missing or
> >>> > unreadable. Consider reinstalling it.
> >>> > Warning: Error determining file type of `/opt/local/bin/clear':
> invalid
> >>> > command name "fileIsBinary"
> >>> > Warning: A file belonging to the `ncurses' port is missing or
> >>> > unreadable. Consider reinstalling it.
> >>> > Warning: Error determining file type of `/opt/local/bin/infocmp':
> >>> > invalid command name "fileIsBinary"
> >>> >
> >>> > Didnt find a solution on the net either.
> >>>
> >>> That's rev-upgrade failing. It also only works on macOS. It's not
> >>> essential though, so you can set 'revupgrade_autorun no' in your
> >>> macports.conf.
> >>>
> >>> - Josh
> >>
> >>
> >
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Thu, Apr 26, 2018 at 12:33 AM, Vishnu  wrote:
> anyone?
>
> On 25 April 2018 at 21:49, Vishnu  wrote:
>>
>> Hi
>>
>> Still same error..Dont know what to do.
>>

Could you please paste the error that you are getting somewhere e.g.,
dpaste.de ?

It would be really helpful if you join IRC channel #macports on freenode.
I use IRCCloud to stay connected -
https://www.irccloud.com/irc/freenode/channel/macports
The link should help you get started.

You would also need a @macports id. Do you have any preference for an username ?

>>
>> On 25 April 2018 at 21:24, Joshua Root  wrote:
>>>
>>> On 2018-4-26 01:39 , Vishnu wrote:
>>> > Hi
>>> >
>>> > I think its best if you can create a repository under Macports.
>>> > I'll be coding there.
>>> >
>>> > Also while trying to install mpstats .
>>> > I am getting these error
>>> >
>>> > Warning: configured user/group macports does not exist, will build as
>>> > root
>>>
>>> Our build system only knows how to create users on macOS. You'll have to
>>> create the macports user manually.
>>>
>>> > --->  Cleaning mpstats
>>> > --->  Updating database of binaries
>>> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
>>> > invalid command name "fileIsBinary"
>>> > Warning: A file belonging to the `ncurses' port is missing or
>>> > unreadable. Consider reinstalling it.
>>> > Warning: Error determining file type of `/opt/local/bin/clear': invalid
>>> > command name "fileIsBinary"
>>> > Warning: A file belonging to the `ncurses' port is missing or
>>> > unreadable. Consider reinstalling it.
>>> > Warning: Error determining file type of `/opt/local/bin/infocmp':
>>> > invalid command name "fileIsBinary"
>>> >
>>> > Didnt find a solution on the net either.
>>>
>>> That's rev-upgrade failing. It also only works on macOS. It's not
>>> essential though, so you can set 'revupgrade_autorun no' in your
>>> macports.conf.
>>>
>>> - Josh
>>
>>
>

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

anyone?

On 25 April 2018 at 21:49, Vishnu  wrote:

> Hi
>
> Still same error..Dont know what to do.
>
>
> On 25 April 2018 at 21:24, Joshua Root  wrote:
>
>> On 2018-4-26 01:39 , Vishnu wrote:
>> > Hi
>> >
>> > I think its best if you can create a repository under Macports.
>> > I'll be coding there.
>> >
>> > Also while trying to install mpstats .
>> > I am getting these error
>> >
>> > Warning: configured user/group macports does not exist, will build as
>> root
>>
>> Our build system only knows how to create users on macOS. You'll have to
>> create the macports user manually.
>>
>> > --->  Cleaning mpstats
>> > --->  Updating database of binaries
>> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
>> > invalid command name "fileIsBinary"
>> > Warning: A file belonging to the `ncurses' port is missing or
>> > unreadable. Consider reinstalling it.
>> > Warning: Error determining file type of `/opt/local/bin/clear': invalid
>> > command name "fileIsBinary"
>> > Warning: A file belonging to the `ncurses' port is missing or
>> > unreadable. Consider reinstalling it.
>> > Warning: Error determining file type of `/opt/local/bin/infocmp':
>> > invalid command name "fileIsBinary"
>> >
>> > Didnt find a solution on the net either.
>>
>> That's rev-upgrade failing. It also only works on macOS. It's not
>> essential though, so you can set 'revupgrade_autorun no' in your
>> macports.conf.
>>
>> - Josh
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Still same error..Dont know what to do.


On 25 April 2018 at 21:24, Joshua Root  wrote:

> On 2018-4-26 01:39 , Vishnu wrote:
> > Hi
> >
> > I think its best if you can create a repository under Macports.
> > I'll be coding there.
> >
> > Also while trying to install mpstats .
> > I am getting these error
> >
> > Warning: configured user/group macports does not exist, will build as
> root
>
> Our build system only knows how to create users on macOS. You'll have to
> create the macports user manually.
>
> > --->  Cleaning mpstats
> > --->  Updating database of binaries
> > Warning: Error determining file type of `/opt/local/bin/captoinfo':
> > invalid command name "fileIsBinary"
> > Warning: A file belonging to the `ncurses' port is missing or
> > unreadable. Consider reinstalling it.
> > Warning: Error determining file type of `/opt/local/bin/clear': invalid
> > command name "fileIsBinary"
> > Warning: A file belonging to the `ncurses' port is missing or
> > unreadable. Consider reinstalling it.
> > Warning: Error determining file type of `/opt/local/bin/infocmp':
> > invalid command name "fileIsBinary"
> >
> > Didnt find a solution on the net either.
>
> That's rev-upgrade failing. It also only works on macOS. It's not
> essential though, so you can set 'revupgrade_autorun no' in your
> macports.conf.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Joshua Root]

On 2018-4-26 01:39 , Vishnu wrote:
> Hi
> 
> I think its best if you can create a repository under Macports.
> I'll be coding there.
> 
> Also while trying to install mpstats .
> I am getting these error
> 
> Warning: configured user/group macports does not exist, will build as root

Our build system only knows how to create users on macOS. You'll have to
create the macports user manually.

> --->  Cleaning mpstats
> --->  Updating database of binaries
> Warning: Error determining file type of `/opt/local/bin/captoinfo':
> invalid command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/clear': invalid
> command name "fileIsBinary"
> Warning: A file belonging to the `ncurses' port is missing or
> unreadable. Consider reinstalling it.
> Warning: Error determining file type of `/opt/local/bin/infocmp':
> invalid command name "fileIsBinary"
> 
> Didnt find a solution on the net either.

That's rev-upgrade failing. It also only works on macOS. It's not
essential though, so you can set 'revupgrade_autorun no' in your
macports.conf.

- Josh

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

I think its best if you can create a repository under Macports.
I'll be coding there.

Also while trying to install mpstats .
I am getting these error

Warning: configured user/group macports does not exist, will build as root
--->  Cleaning mpstats
--->  Updating database of binaries
Warning: Error determining file type of `/opt/local/bin/captoinfo': invalid
command name "fileIsBinary"
Warning: A file belonging to the `ncurses' port is missing or unreadable.
Consider reinstalling it.
Warning: Error determining file type of `/opt/local/bin/clear': invalid
command name "fileIsBinary"
Warning: A file belonging to the `ncurses' port is missing or unreadable.
Consider reinstalling it.
Warning: Error determining file type of `/opt/local/bin/infocmp': invalid
command name "fileIsBinary"

Didnt find a solution on the net either.

On 25 April 2018 at 19:09, Mojca Miklavec  wrote:

> On 25 April 2018 at 14:45, Vishnu wrote:
> > Hello
> >
> > Succesfully ran the portindex2json.
> > and saved it ino a json.
> >
> > So what next?
>
> Wonderful, that's very good news, that will make it much easier to
> proceed with the rest of the tasks.
>
> Now:
> - play a bit with "port search python", "port info python27", ...
> - try to install some super simple packages[***]
> - try to run "port -v livecheck category:math" to see if you can get
> livecheck running correctly (you'll need that info)
> - try to install "sudo port install mpstats", then "port contents
> mpstats"; if successful, you should see a new entry at
> http://stats.macports.neverpanic.de/os_statistics#os_platform
>   I forgot the exact command, but there's a way to show what will be
> submitted, I guess you can find that out by inspecting the source or
> by running help or something
> (I'm not behind my mac at the moment)
>
>
> Just try to get familiar with the "port" command and then proceed to
> whatever other point you want, like
>
>
> - trying to get buildbot master and slave installed (I guess you can
> install them from the native package manager; alternatively via
> virtualenv which gives you slightly more flexibility in picking up the
> exact version that you want); the next step would be to try to
> replicate our setup, see
> https://github.com/macports/macports-infrastructure/tree/
> master/buildbot
>
> - converting the application (without personal details) into
> documentation/plan in a git(hub) repository (something like a markdown
> document)
>
> - (lower priority) proceeding with the simple html page that you
> started working on as a coding challenge, as part of improving &
> finalizing the plan for the summer
>
> - any other item from suggestions sent earlier
>
> Mojca
>
> [***] It's hard to tell what "simple packages" are, but if "platforms"
> contains "linux", there's at least a somewhat bigger chance that it
> will compile correctly. I guess you can expect a number of failures
> because the number of macports users on linux is super limited. Also,
> the build time might be somewhat high since everything needs to be
> compiled from source. A package like "mpstats" should be simple enough
> though, I assume you can find some others. Avoid anything that depends
> on Qt.
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 25 April 2018 at 14:45, Vishnu wrote:
> Hello
>
> Succesfully ran the portindex2json.
> and saved it ino a json.
>
> So what next?

Wonderful, that's very good news, that will make it much easier to
proceed with the rest of the tasks.

Now:
- play a bit with "port search python", "port info python27", ...
- try to install some super simple packages[***]
- try to run "port -v livecheck category:math" to see if you can get
livecheck running correctly (you'll need that info)
- try to install "sudo port install mpstats", then "port contents
mpstats"; if successful, you should see a new entry at
http://stats.macports.neverpanic.de/os_statistics#os_platform
  I forgot the exact command, but there's a way to show what will be
submitted, I guess you can find that out by inspecting the source or
by running help or something
(I'm not behind my mac at the moment)


Just try to get familiar with the "port" command and then proceed to
whatever other point you want, like


- trying to get buildbot master and slave installed (I guess you can
install them from the native package manager; alternatively via
virtualenv which gives you slightly more flexibility in picking up the
exact version that you want); the next step would be to try to
replicate our setup, see
https://github.com/macports/macports-infrastructure/tree/master/buildbot

- converting the application (without personal details) into
documentation/plan in a git(hub) repository (something like a markdown
document)

- (lower priority) proceeding with the simple html page that you
started working on as a coding challenge, as part of improving &
finalizing the plan for the summer

- any other item from suggestions sent earlier

Mojca

[***] It's hard to tell what "simple packages" are, but if "platforms"
contains "linux", there's at least a somewhat bigger chance that it
will compile correctly. I guess you can expect a number of failures
because the number of macports users on linux is super limited. Also,
the build time might be somewhat high since everything needs to be
compiled from source. A package like "mpstats" should be simple enough
though, I assume you can find some others. Avoid anything that depends
on Qt.

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Succesfully ran the portindex2json.
and saved it ino a json.

So what next?

On 25 April 2018 at 17:22, Mojca Miklavec  wrote:

> On 25 April 2018 at 13:45, Vishnu wrote:
> > Hi
> >
> > I ran portindex.
> > It gives me 2 files Portindex and Portindex.quick
>
> I suggest to keep the discussion on the list, so that others can jump
> in and help when needed.
>
> But in short: https://github.com/macports/macports-contrib/blob/master/
> portindex2json/portindex2json.tcl
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Wed, Apr 25, 2018 at 3:36 AM, Mojca Miklavec  wrote:
> See last email from Ryan. Can you try to clone macports-ports and run
> portindex there? (Sorry, typing on my phone.) You can also go to our IRC
> channel for more real-time help.
>

For specific steps you could refer to step 3. at
https://guide.macports.org/#installing.macports.git

For building portindex you can do the following steps
$ cd /opt/mports/macports-ports
$ portindex

It would take some time to create the whole portindex.

P.S: git clone might also take a lot of time if you are on a slow internet.
You could try a shallow clone [1] to make the checkout fast. You will
have only the latest commit and not the complete changeset.

Links:
[1]: https://www.perforce.com/blog/git-beyond-basics-using-shallow-clones

--
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

See last email from Ryan. Can you try to clone macports-ports and run
portindex there? (Sorry, typing on my phone.) You can also go to our IRC
channel for more real-time help.

Mojca

V tor., 24. apr. 2018 23:37 je oseba Vishnu 
napisala:

> Hi all
>
> was succesfully able to install macports using  |$| *|cd
> /opt/mports/macports-base|*
> > |$| *|./configure --enable-readline|*
> > |$| *|make|*
> > |$| *|sudo make install|*
> > |$| *|make distclean|*
>
> but right now problem is
>
> root@vishnupc:~# /opt/local/bin/port -v selfupdate
> --->  Updating MacPorts base sources using rsync
> rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by
> peer (104)
> rsync error: error in rsync protocol data stream (code 12) at io.c(276)
> [Receiver=3.1.1]
> Command failed: /usr/bin/rsync -rtzvl --delete-after rsync://
> rsync.macports.org/macports/release/tarballs/base.tar
> /opt/local/var/macports/sources/
> rsync.macports.org/macports/release/tarballs
> Exit code: 1
>
> what to do now?
>
> On 24 April 2018 at 22:14, Joshua Root  wrote:
>
>> On 2018-4-25 02:36 , Vishnu wrote:
>> > I tried using these
>> >
>> >  1.
>> >
>> > *|mkdir -p /opt/mports|*
>> > |$| *|cd /opt/mports|*
>> > |$| *|git clone https://github.com/macports/macports-base.git|*
>> > |$| *|git checkout v2.4.3 # skip this if you want to use the
>> development
>> > version|*
>> >
>> >  2.
>> >
>> > Build and Install MacPorts
>> >
>> > MacPorts uses autoconf and makefiles for installation. These
>> > commands will build and install MacPorts to |/opt/local|. You can
>> > add |--prefix| to |./configure| to relocate MacPorts to another
>> > directory if needed.
>> >
>> > |$| *|cd /opt/mports/macports-base|*
>> > |$| *|./configure --enable-readline|*
>> > |$| *|make|*
>> > |$| *|sudo make install|*
>> > |$| *|make distclean|*
>> >
>> > while trying to configure i am getting :configure: error: mtree not
>> > found.
>> >
>> > How to proceed?
>> >
>> > I think i need to install additional files.
>> > Can someone help?
>>
>> See the list of prerequisites at
>> .
>>
>> Look up which package provides mtree on your distro and install it. The
>> executable is sometimes called 'fmtree' on Linux, so you may need to set
>> MTREE=fmtree (or whatever it's actually called) when running configure.
>>
>> - Josh
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi all

was succesfully able to install macports using  |$| *|cd
/opt/mports/macports-base|*
> |$| *|./configure --enable-readline|*
> |$| *|make|*
> |$| *|sudo make install|*
> |$| *|make distclean|*

but right now problem is

root@vishnupc:~# /opt/local/bin/port -v selfupdate
--->  Updating MacPorts base sources using rsync
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by
peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(276)
[Receiver=3.1.1]
Command failed: /usr/bin/rsync -rtzvl --delete-after rsync://
rsync.macports.org/macports/release/tarballs/base.tar
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs
Exit code: 1

what to do now?

On 24 April 2018 at 22:14, Joshua Root  wrote:

> On 2018-4-25 02:36 , Vishnu wrote:
> > I tried using these
> >
> >  1.
> >
> > *|mkdir -p /opt/mports|*
> > |$| *|cd /opt/mports|*
> > |$| *|git clone https://github.com/macports/macports-base.git|*
> > |$| *|git checkout v2.4.3 # skip this if you want to use the
> development
> > version|*
> >
> >  2.
> >
> > Build and Install MacPorts
> >
> > MacPorts uses autoconf and makefiles for installation. These
> > commands will build and install MacPorts to |/opt/local|. You can
> > add |--prefix| to |./configure| to relocate MacPorts to another
> > directory if needed.
> >
> > |$| *|cd /opt/mports/macports-base|*
> > |$| *|./configure --enable-readline|*
> > |$| *|make|*
> > |$| *|sudo make install|*
> > |$| *|make distclean|*
> >
> > while trying to configure i am getting :configure: error: mtree not
> > found.
> >
> > How to proceed?
> >
> > I think i need to install additional files.
> > Can someone help?
>
> See the list of prerequisites at
> .
>
> Look up which package provides mtree on your distro and install it. The
> executable is sometimes called 'fmtree' on Linux, so you may need to set
> MTREE=fmtree (or whatever it's actually called) when running configure.
>
> - Josh
>

Re: Gsoc 18 Project | Collect build statistics

[Joshua Root]

On 2018-4-25 02:36 , Vishnu wrote:
> I tried using these
> 
>  1.
> 
> *|mkdir -p /opt/mports|*
> |$| *|cd /opt/mports|*
> |$| *|git clone https://github.com/macports/macports-base.git|*
> |$| *|git checkout v2.4.3 # skip this if you want to use the development
> version|*
> 
>  2.
> 
> Build and Install MacPorts
> 
> MacPorts uses autoconf and makefiles for installation. These
> commands will build and install MacPorts to |/opt/local|. You can
> add |--prefix| to |./configure| to relocate MacPorts to another
> directory if needed.
> 
> |$| *|cd /opt/mports/macports-base|*
> |$| *|./configure --enable-readline|*
> |$| *|make|*
> |$| *|sudo make install|*
> |$| *|make distclean|*
> 
> while trying to configure i am getting :    configure: error: mtree not
> found.
> 
> How to proceed?
> 
> I think i need to install additional files.
> Can someone help?

See the list of prerequisites at
.

Look up which package provides mtree on your distro and install it. The
executable is sometimes called 'fmtree' on Linux, so you may need to set
MTREE=fmtree (or whatever it's actually called) when running configure.

- Josh

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Tue, Apr 24, 2018 at 10:06 PM, Vishnu  wrote:
> I tried using these
>
> mkdir -p /opt/mports
> $ cd /opt/mports
> $ git clone https://github.com/macports/macports-base.git
> $ git checkout v2.4.3  # skip this if you want to use the development
> version
>
> Build and Install MacPorts
>
> MacPorts uses autoconf and makefiles for installation. These commands will
> build and install MacPorts to /opt/local. You can add --prefix to
> ./configure to relocate MacPorts to another directory if needed.
>
> $ cd /opt/mports/macports-base
> $ ./configure --enable-readline
> $ make
> $ sudo make install
> $ make distclean
>
> while trying to configure i am getting :configure: error: mtree not
> found.
>
> How to proceed?
>
> I think i need to install additional files.
> Can someone help?

Looks like mtree[1] is not installed in your machine.

Could you try running 'sudo apt-get install tree' before configure ?

I also found one answer at [2]. Haven't checked though.

[1] http://manpages.ubuntu.com/manpages/bionic/man8/fmtree.8.html
[2] https://serverfault.com/a/690089



-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Tue, Apr 24, 2018 at 9:58 PM, Vishnu  wrote:
> Hey
>
> I have been trying to install Macports on ubuntu.
> And am unable to do so.
>
> Downloaded the tar.
> Somehelp would be appreciated.
>
>

Could you please try to build it from source [1] ?

Are you getting some error related to XCode ?

It would give some insights if you could paste[2] what kind of error
you are getting.

Reference:
[1] https://guide.macports.org/#installing.macports.source
[2] https://dpaste.de/

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

I tried using these

   1.

   *mkdir -p /opt/mports*$ *cd /opt/mports*$ *git clone
https://github.com/macports/macports-base.git
*$ *git checkout v2.4.3
 # skip this if you want to use the development version*

   2.

   Build and Install MacPorts

   MacPorts uses autoconf and makefiles for installation. These commands
   will build and install MacPorts to /opt/local. You can add --prefix to
   ./configure to relocate MacPorts to another directory if needed.

   $ *cd /opt/mports/macports-base*$ *./configure --enable-readline*$
*make*$ *sudo make install*$ *make distclean*


while trying to configure i am getting :configure: error: mtree not
found.

How to proceed?

I think i need to install additional files.
Can someone help?

On 24 April 2018 at 21:58, Vishnu  wrote:

> Hey
>
> I have been trying to install Macports on ubuntu.
> And am unable to do so.
>
> Downloaded the tar.
> Somehelp would be appreciated.
>
>
> On 24 April 2018 at 08:55, Jackson Isaac  wrote:
>
>> Hi Vishnu,
>>
>> Welcome to The MacPorts Project!
>>
>> On Tue, Apr 24, 2018 at 4:15 AM, Vishnu  wrote:
>> > Hey,
>> >
>> > So first thing.
>> > (a.1) I am planning to install ubuntu on my pc in a day.
>> > I guess We can install Macports on that.
>>
>> That's great. Let us know if you face any issues and if we could be of
>> any help.
>>
>> I was also thinking about cloud platform (specifically Google Cloud)
>> which gives
>> you around 300$ of credits for one year where you could spin up a VM and
>> test
>> the installation.
>>
>> Note: It might need a Credit Card for authentication.
>>
>> > (a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot
>> too.
>> >
>> > (b.1) I went and completed week 1.But i am unable to access week 3 ,
>> week
>> > 4.Because the course starts on April 30 i guess.
>> > I went through the topics.
>> > Week 3 : Subqueries and Join
>> > Week 4 : Modifying and Analyzing Data with SQL
>> >
>>
>> I guess you can join the upcoming session and complete all the
>> assignments and weeks
>> before hand too (I was able to do so few weeks back).
>>
>> > So now would learn more about these topics through youtube, many other
>> > resources.
>> >
>> > (b.2) Will look into them.
>> > (b.3) Django hopefully wouldn't be a problem.
>> >
>> > (c.0)  I would suggest that we create a new repository for your project
>> > under
>> > MacPorts, you fork it and use your own fork for endlessly playing with
>> > it, testing etc
>> > This sounds good.
>> > I think i would be more comfortable with Github.
>> >
>> > (c.2) Can be done.
>> > (c.3) You are talking about Build History data ..right?
>> > There is already a JSON API for it as you said..so cant we use it? (
>> > https://build.macports.org/json/help )
>> >
>> > (d.1 , d.2) Yes will do the research .
>> >
>> >
>> > (e.1) I think when i will be changing Buildbot to update the database
>> after
>> > every build ..Then i would need Tcl.
>> > Will learn them as i get time.
>> >
>>
>> Speaking from experience, I would suggest go through the basics of Tcl
>> and write some small sample programs to get the gist of Tcl.
>> This would help you recall whatever you tried now at a later point
>> of time when you actually touch macports-base.
>>
>> https://www.tcl.tk/man/tcl8.5/tutorial/Tcl0.html helped me a lot.
>> I always keep this window open while developing base.
>>
>> > Will see Project Description.
>> >
>> > Also i have my Semester End exams from 27 April - 7 may.
>> > So maybe i would not be able to reply on time.
>> >
>>
>> No issues. Do well for the exams :) Keep us updated.
>>
>> Looking forward to a great summer this year :)
>>
>> --
>> Jackson Isaac
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey

I have been trying to install Macports on ubuntu.
And am unable to do so.

Downloaded the tar.
Somehelp would be appreciated.


On 24 April 2018 at 08:55, Jackson Isaac  wrote:

> Hi Vishnu,
>
> Welcome to The MacPorts Project!
>
> On Tue, Apr 24, 2018 at 4:15 AM, Vishnu  wrote:
> > Hey,
> >
> > So first thing.
> > (a.1) I am planning to install ubuntu on my pc in a day.
> > I guess We can install Macports on that.
>
> That's great. Let us know if you face any issues and if we could be of any
> help.
>
> I was also thinking about cloud platform (specifically Google Cloud) which
> gives
> you around 300$ of credits for one year where you could spin up a VM and
> test
> the installation.
>
> Note: It might need a Credit Card for authentication.
>
> > (a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot
> too.
> >
> > (b.1) I went and completed week 1.But i am unable to access week 3 , week
> > 4.Because the course starts on April 30 i guess.
> > I went through the topics.
> > Week 3 : Subqueries and Join
> > Week 4 : Modifying and Analyzing Data with SQL
> >
>
> I guess you can join the upcoming session and complete all the
> assignments and weeks
> before hand too (I was able to do so few weeks back).
>
> > So now would learn more about these topics through youtube, many other
> > resources.
> >
> > (b.2) Will look into them.
> > (b.3) Django hopefully wouldn't be a problem.
> >
> > (c.0)  I would suggest that we create a new repository for your project
> > under
> > MacPorts, you fork it and use your own fork for endlessly playing with
> > it, testing etc
> > This sounds good.
> > I think i would be more comfortable with Github.
> >
> > (c.2) Can be done.
> > (c.3) You are talking about Build History data ..right?
> > There is already a JSON API for it as you said..so cant we use it? (
> > https://build.macports.org/json/help )
> >
> > (d.1 , d.2) Yes will do the research .
> >
> >
> > (e.1) I think when i will be changing Buildbot to update the database
> after
> > every build ..Then i would need Tcl.
> > Will learn them as i get time.
> >
>
> Speaking from experience, I would suggest go through the basics of Tcl
> and write some small sample programs to get the gist of Tcl.
> This would help you recall whatever you tried now at a later point
> of time when you actually touch macports-base.
>
> https://www.tcl.tk/man/tcl8.5/tutorial/Tcl0.html helped me a lot.
> I always keep this window open while developing base.
>
> > Will see Project Description.
> >
> > Also i have my Semester End exams from 27 April - 7 may.
> > So maybe i would not be able to reply on time.
> >
>
> No issues. Do well for the exams :) Keep us updated.
>
> Looking forward to a great summer this year :)
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 24 April 2018 at 00:45, Vishnu wrote:
>
>> (c.3) Plan the API to get the data from the database in JSON format
>> (so that someone else could write an independent app with the same
>> display functionality).
>> More in a separate email, I guess.
>
> (c.3) You are talking about Build History data ..right?

No. After the proposal deadline was already over, Bradley pointed out
that it would be really helpful if the website would also *serve* the
API rather than just read if from elsewhere.

That is: the fact that Buildbot has a pretty nice API (version one has
a way more extensive one than version 0.8) allows us to easily access
some of its data, otherwise we might have needed to parse raw HTML.

When you create a view to, say, display all ports in the math
category, you would probably serve the page via something like
https://www.macports.org/ports/list/category/math
which would render a nice website with hyperlinks to all ports in this
category. It would be cool if the website would also allow something
like
https://www.macports.org/api/list?category=math
that would basically provide the same piece of information, only in
json format instead of rendered html.

Note that I'm making up the exact URLs.

The rationale would be that anyone who might need some basic
information about the ports could easily access the data. What Bradley
envisioned in the long run is that provided how much the frontend
technologies change, if we had a solid database and API, anyone could
simply create a one-page javascript file at some point in future and
get an awesome site without having to touch the backend. Or maybe we
would need part of this information to render views on the buildbot.
Or perhaps "port arethereanyknownproblems python27" could at some
point in future tell the user whether one can expect a failure etc.

I imagine that this should be possible to without too much additonal
effort, but we can see.

By far the most important part of the website is to have a solid
database and data collection strategies in the background to allow
efficiently querying various data. Even if the frontend won't be
perfect just yet, that's always possible to change later.

You will certainly need to fetch and interpret various JSON data from
various sources. This addition would be about being able to generate
one. But we can still discuss that.

> There is already a JSON API for it as you said..so cant we use it? (
> https://build.macports.org/json/help )

Yes, we can (and will) use that one. But we would need to create one ourselves.

See also http://docs.buildbot.net/latest/developer/rest.html for the
latest Buildbot API.

> (e.1) I think when i will be changing Buildbot to update the database after
> every build ..Then i would need Tcl.

Not really. Buildbot itself is written in python. There's no Tcl
involved in that step.
One thing about the buildbot: we need to be careful not to invest too
much effort into buildbot 0.8 because plenty of stuff changes with
buildbot 1.x. It's certainly vital to be able to collect build
statistics for the old builds, and the script can run once per hour or
so, but when it comes to interaction, we should probably check if we
could perhaps update our own buildbot to version 1.1 before starting
implementing too meany features that are specific to one particular
buildbot version.

> Also i have my Semester End exams from 27 April - 7 may.
> So maybe i would not be able to reply on time.

Good luck with your exams. Do them well, in order to get more free
time and less worries during the summer :)

In this case I suggest that you try to get at least a working MacPorts
installation before exams and we get back to planning with full speed
after your exams are done.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

Welcome to The MacPorts Project!

On Tue, Apr 24, 2018 at 4:15 AM, Vishnu  wrote:
> Hey,
>
> So first thing.
> (a.1) I am planning to install ubuntu on my pc in a day.
> I guess We can install Macports on that.

That's great. Let us know if you face any issues and if we could be of any help.

I was also thinking about cloud platform (specifically Google Cloud) which gives
you around 300$ of credits for one year where you could spin up a VM and test
the installation.

Note: It might need a Credit Card for authentication.

> (a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot too.
>
> (b.1) I went and completed week 1.But i am unable to access week 3 , week
> 4.Because the course starts on April 30 i guess.
> I went through the topics.
> Week 3 : Subqueries and Join
> Week 4 : Modifying and Analyzing Data with SQL
>

I guess you can join the upcoming session and complete all the
assignments and weeks
before hand too (I was able to do so few weeks back).

> So now would learn more about these topics through youtube, many other
> resources.
>
> (b.2) Will look into them.
> (b.3) Django hopefully wouldn't be a problem.
>
> (c.0)  I would suggest that we create a new repository for your project
> under
> MacPorts, you fork it and use your own fork for endlessly playing with
> it, testing etc
> This sounds good.
> I think i would be more comfortable with Github.
>
> (c.2) Can be done.
> (c.3) You are talking about Build History data ..right?
> There is already a JSON API for it as you said..so cant we use it? (
> https://build.macports.org/json/help )
>
> (d.1 , d.2) Yes will do the research .
>
>
> (e.1) I think when i will be changing Buildbot to update the database after
> every build ..Then i would need Tcl.
> Will learn them as i get time.
>

Speaking from experience, I would suggest go through the basics of Tcl
and write some small sample programs to get the gist of Tcl.
This would help you recall whatever you tried now at a later point
of time when you actually touch macports-base.

https://www.tcl.tk/man/tcl8.5/tutorial/Tcl0.html helped me a lot.
I always keep this window open while developing base.

> Will see Project Description.
>
> Also i have my Semester End exams from 27 April - 7 may.
> So maybe i would not be able to reply on time.
>

No issues. Do well for the exams :) Keep us updated.

Looking forward to a great summer this year :)

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hey,

So first thing.
(a.1) I am planning to install ubuntu on my pc in a day.
I guess We can install Macports on that.
(a.2 , a.3) Once macports is installed i'll install mpstats ,buildbot too.

(b.1) I went and completed week 1.But i am unable to access week 3 , week
4.Because the course starts on April 30 i guess.
I went through the topics.
Week 3 : Subqueries and Join
Week 4 : Modifying and Analyzing Data with SQL

So now would learn more about these topics through youtube, many other
resources.

(b.2) Will look into them.
(b.3) Django hopefully wouldn't be a problem.

(c.0)  I would suggest that we create a new repository for your project
under
MacPorts, you fork it and use your own fork for endlessly playing with
it, testing etc
This sounds good.
I think i would be more comfortable with Github.

(c.2) Can be done.
(c.3) You are talking about Build History data ..right?
There is already a JSON API for it as you said..so cant we use it? (
*https://build.macports.org/json/help
* )

(d.1 , d.2) Yes will do the research .


(e.1) I think when i will be changing Buildbot to update the database after
every build ..Then i would need Tcl.
Will learn them as i get time.

Will see Project Description.

Also i have my Semester End exams from 27 April - 7 may.
So maybe i would not be able to reply on time.

Thanks
Vishnu







On 24 April 2018 at 03:37, Mojca Miklavec  wrote:

> Dear Vishnu,
>
> I have a bunch of suggestions written below. Community bonding should
> usually be used to getting familiar with our codebase, our tools, our
> communication channels, getting to know people, ...
>
> Since you are creating a standalone product, this community bonding
> could be slightly different (not so much need to learn our base code
> by heart), but it should warm you up and get you up to full speed by
> the time the coding period starts.
>
> The most important parts are:
> - getting MacPorts installed somewhere where you have regular access
> - creating a repository and finish/improve the planning, I would like
> to see the database design finalized, finish the sample HTML document,
> ...
> - creating tickets with milestones
>
> I hope others will be able to comment on my suggestions (they are not
> universal truths :)
>
>
>
> INSTALLING MACPORTS, BUILDBOT
>
> (a.1) We explained that owning a Mac was not a requirement for working
> on this project, but it would be orders of magnitude easier to work on
> some tasks if you had a regular access to MacPorts. This could either
> be:
>   - SSH access to a remote Mac or VM
>   - A Linux machine (even if Raspberry PI) or a VM with Linux
>
> You should start working on this as soon as possible because it would
> be a pity to waste precious time during the coding period. MacPorts
> can theoretically be installed on a UNIX machine, but there might be
> problems of one kind or the other. We can help you circumvent the
> problems, but you need to star working on this ASAP.
>
> Please let us know what your plan is.
>
> (a.2) With MacPorts you should be able to install mpstats and use it
> to submit statistics (to either the official server or your new server
> once ready).
>
> (a.3) It would probably be quite helpful if you manage to install
> buildbot and replicate our setup (at least to some extent). But (a.1)
> is a strong requirement for that, else it makes no sense.
>
>
> SQL & OTHER TOOLS
>
> (b.1) This is an absolute must:
> https://www.coursera.org/learn/sql-for-data-science#
> along with some further understanding of primary keys, indices,
> foreign keys, ...
>
> Let us know whether you managed to subscribe and how you are
> progressing, whether you have any further questions etc.
>
> (b.2) I found two (fully optional) courses on web design:
> - a very lightweight one with very clear explanations and adorable
> teacher, good for listening to while going for a walk :)
>   https://www.coursera.org/learn/responsivedesign/
> - a more comprehensive one about bootstrap 4:
>   https://www.coursera.org/learn/bootstrap-4/
> You might know these things already and even if not, you probably know
> enough to come up with some basic styles for what we need; we don't
> need perfect design anyway.
>
> (b.3) Django (which you should know from inside out :)
>
>
> PLANNING
>
> (c.0) Open for discussion/suggestions ...
> I would suggest that we create a new repository for your project under
> MacPorts, you fork it and use your own fork for endlessly playing with
> it, testing etc. (No need to fork though, you can just start
> somewhere.) Once you think that part of your code is ready for review,
> you submit a pull request, we review it and merge it. That way we
> would end up with clean code upstream, all of it being properly
> reviewed.
>
> Once the code ends up in the MacPorts repository, that code should
> basically be ready for deployment.
>
> We use Trac as our main issue tracker. It depends on what others
> 

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

Let me congratulate you for being accepted to the GSOC program with MacPorts.

Since your project is of vital importance for MacPorts, we will
require pretty high coding standards, including good documentation and
test cases, and strive towards early deployment, leaving sufficient
room for feedback and improvements.

Apart from regular email flow on the mailing list or IRC, I suggest to
have weekly live chat with mentors for brainstorming, ease of planning
& progress tracking, problem resolution, ...

We have three weeks of community bonding period in front of us. Making
good use of this time is equally important as coding itself, so we
hope you'll use this time well. I have a pretty long email nearly
ready with more instructions and suggestion about what I envisioned
for those three weeks. You might get some greeting from other members
of the team as well. You can expect help from me, Jackson and Umesh
(I'm at UTC+2, the other two are in the same time zone as you are),
but others should be able to help you as well (I don't always have all
the answers), final deployment can be done by Clemens or Rainer who
also have a lot more expertise in MacPorts internals than all the
three of us combined :).

We are looking forward to an awesome summer working together,
Mojca

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

To address some of your earlier non-technical questions with respect
to GSOC, here's what our admin wrote:

As per GSOC policy, we cannot comment on selection process before
April 23rd. We are evaluating all the proposals and might ask some
clarifications if required. There are no set steps or guideline which
confirms a student's acceptance. Students' self motivation to open
pull request and commit changes are always welcome and improves the
chances. Communication also plays a key role.



The technical feedback written below is valid independent of gsoc
schedule (that is: I'm not expecting you to implement what I wrote
below right away as a requirement for anything, this does not in any
way reflect feedback or questions from other mentors etc. and until
the official announcements there is not even a guarantee that
implementing all of the suggestions would guarantee a slot).

On 5 April 2018 at 16:08, Vishnu wrote:
>
> I further improved my demo project based on feedback.

Thank you.

> I am now able to parse the ports(stored in json locally) using the website
> itself(by accessing  {url}/port/parser).
> This will read the json file and fill the port table using django models.

Thank you, that's better than writing raw sql statements in any case.

I already wanted to suggest putting the material in a repository, but
you already did that in the meantime
(https://github.com/VishnuM-sopho/demo-macports).


Some feedback related to repository layout:

1.) It's nice if a project contains README.md explaining what the
project does and how someone who's not familiar with django can launch
it to test himself.

2.) It's not clear what test.db is at all. OK, I know it's a sqlite
database after checking the contents, but it makes sense to put the
result of such individual standalone tasks to a different folder in
order to distinguish between the Django app itself and some unrelated
playing with the code, so put python code that's not related to django
to a separate location.

3.) You have some strange file names in the middle of your django app.
Like "aljaz script output.json". And then another file
"testing2.json". The first one is 15 MB, the other one is 5 MB. It's
not clear what file does what, what is the difference etc. In any case
this is data that changes over time. I would generally put it to a
different folder called "data" or something. In the final deployed
apllication such a file should not be included in the repository, it
also gets outdated pretty soon, but it's certainly fine to have such a
file at hand for testing etc., so it is still valuable, it just makes
sense to put it to a different top folder.

4.) Your repository contains some sqlite3 databases. This might be ok
for the initial demonstration of your ability to fill in a small
database, but generally the database itself should not be part of code
repository. Instead, the instructions how to run your code are much
more valuable.


Some feedback related to your app:

1.) I understand this was the fastest way to get the simple site done.
But once your are out of "get this done quickly" mode, you need to put
the css and js out of port/templates/port/port.html (some minor js
might occasionally be needed, that's fine, but don't put all the
zillion of css lines there). Static files should be separate, also for
efficiency reasons.

2.) Use more descriptive names than p1n, d1e, v1a, ...
(portname=p1n, description=d1e, variant=v1a, portdir=p1d)

3.) Your code works for single time import:
c = Port.objects.create(portname=p1n, description=d1e,
variant=v1a, portdir=p1d)
what happens when you do data import for the second, third, ... time?

4.) In port/models.py you need to make the port name "unique"
(https://docs.djangoproject.com/en/2.0/ref/models/fields/) and you
probably need an index on portdir, so that you can later group ports
with the same portdir together and show all ports with the same
portdir together.

5.) In many other projects on your github account you included/left
database passwords in repository. You should absolutely never do that.
In the django project you left some key inside the code. Not sure
where that one is used, but you need to make sure that the production
server uses something else. I don't know yet how Django handle this.

I was surprised that you used some made-up names in the database in
your screenshots. Since you had the full database, I would imagine
there was zero extra effort to simply import them all real data there
eventually :)


Some potential further steps could be:

(a) Static site (I don't see the page itself in the repo, only in the
form of template?)
It might make sense to continue working on that static sample site.
Add all the missing graphs (not the graphs themselves, but specify
what to put there), maybe some further work on layout (just the
basics, not to the perfection).

(b) Study how to do unit tests in Django and try to write some for
that sample site.

One example of a unit 

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Yes i wanted to get some feedback on the updated demo project.
And what should i do now?
Thanks

On Fri, Apr 6, 2018, 10:27 PM Jackson Isaac  wrote:

> Hi Vishnu,
>
>
> On Fri 6 Apr, 2018, 22:19 Vishnu,  wrote:
>
>> Could i get a reply please.
>>
>
> Was there any question or query that I might have missed ?
>
>
> On Thu, Apr 5, 2018, 11:23 PM Vishnu  wrote:
>>
>>> Hello
>>>
>>> Here is the git repository i made and am working on for demo project.
>>> https://github.com/VishnuM-sopho/demo-macports
>>>
>>> Thanks
>>>
>>> On 5 April 2018 at 21:31, Vishnu  wrote:
>>>
 Hello Jackson


 I am sorry .
 I was in a hurry to finish the demo as early as possible. So, I didnt yet
 make a git rep for this folder so you cant see any history.
 Right now we can only compare the files i sent earlier thrugh mails
 earlier to see changes.

 Thanks

 On 5 April 2018 at 19:53, Jackson Isaac  wrote:

> Hi,
>
> On Thu 5 Apr, 2018, 19:38 Vishnu,  wrote:
>
>> Dear mentors,
>>
>> I further improved my demo project based on feedback.
>>
>> I am now able to parse the ports(stored in json locally) using the
>> website itself(by accessing  {url}/port/parser).
>> This will read the json file and fill the port table using django
>> models.
>>
>> also upon adding the ports to the table we get to see how many ports
>> were added .
>>
>> Please go through the images attached in the link below in sequential
>> order.
>> And do provide some feedback.What else should i do?
>>
>>
>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>
>
> Could you please push the code on git so that we can track changes ?
>


>>>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

Here is the git repository i made and am working on for demo project.
https://github.com/VishnuM-sopho/demo-macports

Thanks

On 5 April 2018 at 21:31, Vishnu  wrote:

> Hello Jackson
>
>
> I am sorry .
> I was in a hurry to finish the demo as early as possible. So, I didnt yet
> make a git rep for this folder so you cant see any history.
> Right now we can only compare the files i sent earlier thrugh mails
> earlier to see changes.
>
> Thanks
>
> On 5 April 2018 at 19:53, Jackson Isaac  wrote:
>
>> Hi,
>>
>> On Thu 5 Apr, 2018, 19:38 Vishnu,  wrote:
>>
>>> Dear mentors,
>>>
>>> I further improved my demo project based on feedback.
>>>
>>> I am now able to parse the ports(stored in json locally) using the
>>> website itself(by accessing  {url}/port/parser).
>>> This will read the json file and fill the port table using django models.
>>>
>>> also upon adding the ports to the table we get to see how many ports
>>> were added .
>>>
>>> Please go through the images attached in the link below in sequential
>>> order.
>>> And do provide some feedback.What else should i do?
>>>
>>>
>>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>>
>>
>> Could you please push the code on git so that we can track changes ?
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello Jackson


I am sorry .
I was in a hurry to finish the demo as early as possible. So, I didnt yet
make a git rep for this folder so you cant see any history.
Right now we can only compare the files i sent earlier thrugh mails earlier
to see changes.

Thanks

On 5 April 2018 at 19:53, Jackson Isaac  wrote:

> Hi,
>
> On Thu 5 Apr, 2018, 19:38 Vishnu,  wrote:
>
>> Dear mentors,
>>
>> I further improved my demo project based on feedback.
>>
>> I am now able to parse the ports(stored in json locally) using the
>> website itself(by accessing  {url}/port/parser).
>> This will read the json file and fill the port table using django models.
>>
>> also upon adding the ports to the table we get to see how many ports were
>> added .
>>
>> Please go through the images attached in the link below in sequential
>> order.
>> And do provide some feedback.What else should i do?
>>
>>
>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>
>
> Could you please push the code on git so that we can track changes ?
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello Jackson


I am sorry .
I was in a hurry to finish the demo as early as possible. So, I didnt yet
make a git rep for this folder so you cant see any history.
Right now we can only compare the files i sent earlier through mails
earlier to see changes.

Will be making one now.So that the changes can be tracked.

Thanks



On 5 April 2018 at 19:53, Jackson Isaac  wrote:

> Hi,
>
> On Thu 5 Apr, 2018, 19:38 Vishnu,  wrote:
>
>> Dear mentors,
>>
>> I further improved my demo project based on feedback.
>>
>> I am now able to parse the ports(stored in json locally) using the
>> website itself(by accessing  {url}/port/parser).
>> This will read the json file and fill the port table using django models.
>>
>> also upon adding the ports to the table we get to see how many ports were
>> added .
>>
>> Please go through the images attached in the link below in sequential
>> order.
>> And do provide some feedback.What else should i do?
>>
>>
>> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>>
>
> Could you please push the code on git so that we can track changes ?
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi,

On Thu 5 Apr, 2018, 19:38 Vishnu,  wrote:

> Dear mentors,
>
> I further improved my demo project based on feedback.
>
> I am now able to parse the ports(stored in json locally) using the website
> itself(by accessing  {url}/port/parser).
> This will read the json file and fill the port table using django models.
>
> also upon adding the ports to the table we get to see how many ports were
> added .
>
> Please go through the images attached in the link below in sequential
> order.
> And do provide some feedback.What else should i do?
>
>
> https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS
>

Could you please push the code on git so that we can track changes ?

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Dear mentors,

I further improved my demo project based on feedback.

I am now able to parse the ports(stored in json locally) using the website
itself(by accessing  {url}/port/parser).
This will read the json file and fill the port table using django models.

also upon adding the ports to the table we get to see how many ports were
added .

Please go through the images attached in the link below in sequential order.
And do provide some feedback.What else should i do?


https://drive.google.com/open?id=1brwBCNoz9STWakvLZ7h-JNrnKFAdvMTS

Thanks
Vishnu

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello

I wanted to get feedback on the demo project i submitted

https://drive.google.com/open?id=1LbMnyHjaRCbg7IrAUMM042eD7O2Br-jU


Thanks


On Wed, Apr 4, 2018, 11:08 AM Mojca Miklavec  wrote:

> Dear Vishnu,
>
> On 3 April 2018 at 23:54, Vishnu wrote:
> >
> > Hello
> >
> > I have finished the mini project given to me.
> > I am now successfully able to scrape data from json into the database.
> > And have integrated the database and website to display all the ports
> and their basic information as it was required
> > All the images and files are attached.
> > Please go through the link.
> > https://drive.google.com/open?id=1LbMnyHjaRCbg7IrAUMM042eD7O2Br-jU
>
> Thank you very much.
>
> As far as I am concerned this serves ok as a demo.
> (The database would ideally be filled with Django as well, but that's
> ok for now.)
> I don't know if other mentors have any further questions.
>
> > On 4 April 2018 at 03:19, Vishnu wrote:
> >>
> >> Hello
> >>
> >> I have finished the mini project given to me.
> >> I am successfully able to scrape data from json into the database.
> >> And have integrated the database and website to display all the ports
> and their basic information as it was required
> >> All the images and files are attached.
> >>
> >> My last 3 mails got rejected due to large size.
> >> Hence uploaded all the images as attachment..
>
> You can see which emails reached the mailing list by looking at
>
> https://lists.macports.org/pipermail/macports-dev/2018-April/thread.html
>
> I'm not absolutely sure, but I think this email did not reach others
> either because the images were still present inline (in quoted
> messages) when you hit the "reply to" button and thus the email size
> was still prohibitively large.
>
> In the future try not to send anything but trivial/small text
> attachments (or super small images like a logotype), any screenshots
> should better go to some other place where only those interested in
> the topic will go to check them out, while hundreds of other
> subscribers to the mailing list would not get their mailboxes filled.
>
> Mojca
>
>
>
> >> On 4 April 2018 at 03:06, Vishnu  wrote:
> >>>
> >>> In the last mail it was working but by taking portid from url.
> >>>
> >>> Now i created my custom view which will take the text after "port/"
> and search it in port table and return all its properties.
> >>>
> >>> Please look at the url.This is what our final result would want.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On 4 April 2018 at 02:06, Vishnu  wrote:
> 
>  Hello
> 
>  I have succesfully integrated django & sqlite3 .
>  Also am able to display the content on web.
>  Following Screenshots will explain :
> 
> 
> 
>  Databse :
> 
> 
> 
>  Is my progress fine?
> 
>  I think this would be enough as my proof of skill.
> 
>  Thanks
> 
> 
> 
>  On 4 April 2018 at 01:12, Vishnu  wrote:
> >
> > Hello
> >
> >
> > Yes that's what I have been doing.
> > Its almost ready.. Now integrating it with my front-end website.so
> that table is updated automatically.
> > Will be mailing you very soon. All the screenshots.
> >
> > On Wed, Apr 4, 2018, 1:05 AM Mojca Miklavec 
> wrote:
> >>
> >> Dear Vishnu,
> >>
> >> On 3 April 2018 at 20:10, Vishnu wrote:
> >> >
> >> > Hello
> >> >
> >> > Yes portindex.json is very useful.
> >> >
> >> > I tried my hands with python and sql.
> >> > And am successfully able to parse the json and store the data
> into the table .
> >>
> >> Thank you very much. The initial steps look promising.
> >>
> >> Now, try to integrate this into a simple django app with the
> database
> >> abstraction layer. A simple tutorial is here to demonstrate how to
> >> define simple python classes that automatically interoperate with
> >> database (so no need to write direct sql code at this step):
> >> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
> >>
> >> Using SQLite is fine (for the production we would probably want to
> go
> >> for PostgreSQL, but now it doesn't make any difference).
> >>
> >> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 3 April 2018 at 23:54, Vishnu wrote:
>
> Hello
>
> I have finished the mini project given to me.
> I am now successfully able to scrape data from json into the database.
> And have integrated the database and website to display all the ports and 
> their basic information as it was required
> All the images and files are attached.
> Please go through the link.
> https://drive.google.com/open?id=1LbMnyHjaRCbg7IrAUMM042eD7O2Br-jU

Thank you very much.

As far as I am concerned this serves ok as a demo.
(The database would ideally be filled with Django as well, but that's
ok for now.)
I don't know if other mentors have any further questions.

> On 4 April 2018 at 03:19, Vishnu wrote:
>>
>> Hello
>>
>> I have finished the mini project given to me.
>> I am successfully able to scrape data from json into the database.
>> And have integrated the database and website to display all the ports and 
>> their basic information as it was required
>> All the images and files are attached.
>>
>> My last 3 mails got rejected due to large size.
>> Hence uploaded all the images as attachment..

You can see which emails reached the mailing list by looking at
https://lists.macports.org/pipermail/macports-dev/2018-April/thread.html

I'm not absolutely sure, but I think this email did not reach others
either because the images were still present inline (in quoted
messages) when you hit the "reply to" button and thus the email size
was still prohibitively large.

In the future try not to send anything but trivial/small text
attachments (or super small images like a logotype), any screenshots
should better go to some other place where only those interested in
the topic will go to check them out, while hundreds of other
subscribers to the mailing list would not get their mailboxes filled.

Mojca



>> On 4 April 2018 at 03:06, Vishnu  wrote:
>>>
>>> In the last mail it was working but by taking portid from url.
>>>
>>> Now i created my custom view which will take the text after "port/" and 
>>> search it in port table and return all its properties.
>>>
>>> Please look at the url.This is what our final result would want.
>>>
>>> Thanks
>>>
>>>
>>> On 4 April 2018 at 02:06, Vishnu  wrote:

 Hello

 I have succesfully integrated django & sqlite3 .
 Also am able to display the content on web.
 Following Screenshots will explain :



 Databse :



 Is my progress fine?

 I think this would be enough as my proof of skill.

 Thanks



 On 4 April 2018 at 01:12, Vishnu  wrote:
>
> Hello
>
>
> Yes that's what I have been doing.
> Its almost ready.. Now integrating it with my front-end website.so that 
> table is updated automatically.
> Will be mailing you very soon. All the screenshots.
>
> On Wed, Apr 4, 2018, 1:05 AM Mojca Miklavec  wrote:
>>
>> Dear Vishnu,
>>
>> On 3 April 2018 at 20:10, Vishnu wrote:
>> >
>> > Hello
>> >
>> > Yes portindex.json is very useful.
>> >
>> > I tried my hands with python and sql.
>> > And am successfully able to parse the json and store the data into the 
>> > table .
>>
>> Thank you very much. The initial steps look promising.
>>
>> Now, try to integrate this into a simple django app with the database
>> abstraction layer. A simple tutorial is here to demonstrate how to
>> define simple python classes that automatically interoperate with
>> database (so no need to write direct sql code at this step):
>> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
>>
>> Using SQLite is fine (for the production we would probably want to go
>> for PostgreSQL, but now it doesn't make any difference).
>>
>> Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello


Yes that's what I have been doing.
Its almost ready.. Now integrating it with my front-end website.so that
table is updated automatically.
Will be mailing you very soon. All the screenshots.

On Wed, Apr 4, 2018, 1:05 AM Mojca Miklavec  wrote:

> Dear Vishnu,
>
> On 3 April 2018 at 20:10, Vishnu wrote:
> >
> > Hello
> >
> > Yes portindex.json is very useful.
> >
> > I tried my hands with python and sql.
> > And am successfully able to parse the json and store the data into the
> table .
>
> Thank you very much. The initial steps look promising.
>
> Now, try to integrate this into a simple django app with the database
> abstraction layer. A simple tutorial is here to demonstrate how to
> define simple python classes that automatically interoperate with
> database (so no need to write direct sql code at this step):
> https://docs.djangoproject.com/en/2.0/intro/tutorial01/
>
> Using SQLite is fine (for the production we would probably want to go
> for PostgreSQL, but now it doesn't make any difference).
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 3 April 2018 at 20:10, Vishnu wrote:
>
> Hello
>
> Yes portindex.json is very useful.
>
> I tried my hands with python and sql.
> And am successfully able to parse the json and store the data into the table .

Thank you very much. The initial steps look promising.

Now, try to integrate this into a simple django app with the database
abstraction layer. A simple tutorial is here to demonstrate how to
define simple python classes that automatically interoperate with
database (so no need to write direct sql code at this step):
https://docs.djangoproject.com/en/2.0/intro/tutorial01/

Using SQLite is fine (for the production we would probably want to go
for PostgreSQL, but now it doesn't make any difference).

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 3 April 2018 at 16:56, Vishnu wrote:
> Regarding mini django project. What would you suggest i do?
>
> I was thinking on code to populate the port table from portindex.json
>
> Would that act as a sample for my Coding skill?
> Or should i do something else?

That sounds ok and is in fact needed for any other step of the project.

Don't bother too much about proper implementation of maintainers,
variants and dependencies. (And in particular don't even try to bother
about storing historic changes of port versions that you planned in
the spreadsheet.) If you do need an additional challenge, you can try
to populate another table "categories" with many-to-many relation to
ports, but that's bonus (only if you have extra time), so don't worry
too much about it either.

If you can populate the database and get a basic port listing at
{url}/port/{name},
like for example
http://localhost/port/python27
giving you a subset of the HTML that you sent us earlier, that should be fine.

I sent you a portindex a while ago, so I hope you have all the
information you need.

> Not clear about what document you are talking about perfecting.

I sent you feedback about that HTML, so you could improve that
document, but it doesn't make too much sense to do this now.

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello,

First thing. I am. Not clear about what document you are talking about
perfecting.

And i will now focus attention on django as mentioned in my earlier mails.

And thanks a lot For that in depth reply. Will work on all things related
to Html after I complete working with django.

Thanks

On Tue, Apr 3, 2018, 9:04 PM Mojca Miklavec  wrote:

> Dear Vishnu,
>
> Thank you very much for sharing the document. The purpose of this HTML
> was two-fold:
> - demonstrating your skills
> - first step of the planning phase for the actual implementation
>
> Below I'm providing some feedback, but I would suggest to concentrate
> on a simple django app at this moment and then return back to this
> html once you are "done" with Django to address (some of) the
> comments. In short: if selected, I'll insist to make this document
> "perfect" before proceeding (and to address all the feedback + more I
> didn't yet bother writing), but there's no point in asking you to
> spend a week making this document ten times longer and fixing tiny
> unimportant details that don't really demonstrate the skillset :)
>
> On 2 April 2018 at 23:32, Mojca Miklavec wrote:
> > V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:
> >>
> >> In the database.
> >> Because then it would be very easy to count the number of os for that
> >> port.
> >
> > I'll explain tomorrow why this is suboptimal. (But there's no need to
> > further optimise the database design right now.)
>
> There are probably better resources that explain this, but here's the
> first hit from Google:
>
> https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
> https://en.wikipedia.org/wiki/Database_normalization
>
> In extreme case, imagine that we decide to send a questionnaire to our
> participants of statistics collection, asking them some 100 optional
> questions, including anything from gender, age, country of origin,
> country of current residence, education, favourite animal, ... Then we
> decide that we would want to compare the age distribution of users of
> package A vs. age distribution of users of package B.
>
> Your idea that allows "very easy number counting" would mean that:
>
> At the moment you only have (submission id, port, port version,
> variants) in the table. You would need to extend the table to contain
> (submission id, submission time, user id, port, port version,
> variants, os version, stdlib, xcode version, age, gender, country,
> education, favourite animal, ...)
> And if the user has 1000 ports installed, you would need to store
> 100x1000 cells (repeat that same information one thousand times and
> then again in any subsequent submission from the same user) instead of
> having a single copy in a separate "questionnaire" table. Multiply
> that with 10.000 users submitting statistics and you end up with tens
> of gigabytes of data each month, just to store results of that
> one-time questionnaire.
>
> On top of that, once the user submits a questionnaire, if you keep
> those answers in a separate table and use proper SQL queries, you
> could easily get the answer to question "what was the prevailing
> gender of users of package A" even for submissions that were made many
> months ago. If you store everything into a single monstrous table, you
> would either need to modify plenty of old submissions or you would not
> be able to get that information for old submissions at all.
>
> Additionally, it could happen that while you are updating old
> submissions, the database crashes. You could end up with half of the
> entries updated and the other half left at their old value, in
> inconsistent state. There are plenty of problems if you don't make
> sure that you keep your database design in a good shape from the very
> beginning.
>
>
> That's a super common use case in databases that has already been
> solved. One should use table joins and views. Random link (I'm sure
> there are better ones):
> https://db.grussell.org/sql3.html
>
> I don't know how Django handles joins and views (some hints I skimmed
> through are here https://stackoverflow.com/a/1281051/585897), but one
> should certainly make sure that the database design is done well.
> Learning more about that topic is part of the process.
>
>
> On 2 April 2018 at 23:50, Vishnu wrote:
> >
> > Please go through this https://jsfiddle.net/vishnum98/3r4vL4L3/21/
> >
> > I did some changes.
>
> Thank you very much. The chart looks ok. For the remaining (missing)
> charts just add a section (and optionally an empty box) and describe
> what kind of chart goes there (no need for a long paragraph, just make
> it clear what's on the Y axis).
>
> I don't think we need a drop-down to select a version, but now that
> you put it there, what I think would be helpful to have there is
> something to switch between:
> - absolute number of installations in that month
> - number of installations of that port divided by total number of
> submissions in that month
> That is: having both 

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

Thank you very much for sharing the document. The purpose of this HTML
was two-fold:
- demonstrating your skills
- first step of the planning phase for the actual implementation

Below I'm providing some feedback, but I would suggest to concentrate
on a simple django app at this moment and then return back to this
html once you are "done" with Django to address (some of) the
comments. In short: if selected, I'll insist to make this document
"perfect" before proceeding (and to address all the feedback + more I
didn't yet bother writing), but there's no point in asking you to
spend a week making this document ten times longer and fixing tiny
unimportant details that don't really demonstrate the skillset :)

On 2 April 2018 at 23:32, Mojca Miklavec wrote:
> V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:
>>
>> In the database.
>> Because then it would be very easy to count the number of os for that
>> port.
>
> I'll explain tomorrow why this is suboptimal. (But there's no need to
> further optimise the database design right now.)

There are probably better resources that explain this, but here's the
first hit from Google:

https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
https://en.wikipedia.org/wiki/Database_normalization

In extreme case, imagine that we decide to send a questionnaire to our
participants of statistics collection, asking them some 100 optional
questions, including anything from gender, age, country of origin,
country of current residence, education, favourite animal, ... Then we
decide that we would want to compare the age distribution of users of
package A vs. age distribution of users of package B.

Your idea that allows "very easy number counting" would mean that:

At the moment you only have (submission id, port, port version,
variants) in the table. You would need to extend the table to contain
(submission id, submission time, user id, port, port version,
variants, os version, stdlib, xcode version, age, gender, country,
education, favourite animal, ...)
And if the user has 1000 ports installed, you would need to store
100x1000 cells (repeat that same information one thousand times and
then again in any subsequent submission from the same user) instead of
having a single copy in a separate "questionnaire" table. Multiply
that with 10.000 users submitting statistics and you end up with tens
of gigabytes of data each month, just to store results of that
one-time questionnaire.

On top of that, once the user submits a questionnaire, if you keep
those answers in a separate table and use proper SQL queries, you
could easily get the answer to question "what was the prevailing
gender of users of package A" even for submissions that were made many
months ago. If you store everything into a single monstrous table, you
would either need to modify plenty of old submissions or you would not
be able to get that information for old submissions at all.

Additionally, it could happen that while you are updating old
submissions, the database crashes. You could end up with half of the
entries updated and the other half left at their old value, in
inconsistent state. There are plenty of problems if you don't make
sure that you keep your database design in a good shape from the very
beginning.


That's a super common use case in databases that has already been
solved. One should use table joins and views. Random link (I'm sure
there are better ones):
https://db.grussell.org/sql3.html

I don't know how Django handles joins and views (some hints I skimmed
through are here https://stackoverflow.com/a/1281051/585897), but one
should certainly make sure that the database design is done well.
Learning more about that topic is part of the process.


On 2 April 2018 at 23:50, Vishnu wrote:
>
> Please go through this https://jsfiddle.net/vishnum98/3r4vL4L3/21/
>
> I did some changes.

Thank you very much. The chart looks ok. For the remaining (missing)
charts just add a section (and optionally an empty box) and describe
what kind of chart goes there (no need for a long paragraph, just make
it clear what's on the Y axis).

I don't think we need a drop-down to select a version, but now that
you put it there, what I think would be helpful to have there is
something to switch between:
- absolute number of installations in that month
- number of installations of that port divided by total number of
submissions in that month
That is: having both absolute and relative numbers available.

To make it clear: don't bother actually implementing this now. You can
add a placeholder to remind you about that later (or just change the
contents of that drop-down to do this instead), nothing else.

We are mainly interested in the cumulative number of installations of
a particular. Version does tell something, but not *that* much, except
that the user did not update the ports for at least a month. We could
potentially make a cumulative diagram listing all versions, random
example:

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Sure thanks a lot for the prompt reply.
Will start working on the idea and produce results very quickly.

On Tue, Apr 3, 2018, 8:32 PM Jackson Isaac  wrote:

> Hi Vishnu,
>
> On Tue 3 Apr, 2018, 20:26 Vishnu,  wrote:
>
>> Regarding mini django project. What would you suggest i do?
>>
>> I was thinking on code to populate the port table from portindex.json
>>
>
> Please go ahead and implement your idea :)
>
>
>> Would that act as a sample for my Coding skill?
>> Or should i do something else?
>>
>
> For me, it should be enough to get a gist of your skills and thought
> process. We can probably add some mini tasks on the fly, if required.
>
> Feel free to reach out to us in case you have any queries.
>
> --
> Jackson Isaac
>

Re: Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

Hi Vishnu,

On Tue 3 Apr, 2018, 20:26 Vishnu,  wrote:

> Regarding mini django project. What would you suggest i do?
>
> I was thinking on code to populate the port table from portindex.json
>

Please go ahead and implement your idea :)


> Would that act as a sample for my Coding skill?
> Or should i do something else?
>

For me, it should be enough to get a gist of your skills and thought
process. We can probably add some mini tasks on the fly, if required.

Feel free to reach out to us in case you have any queries.

-- 
Jackson Isaac

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Regarding mini django project. What would you suggest i do?

I was thinking on code to populate the port table from portindex.json

Would that act as a sample for my Coding skill?
Or should i do something else?

On Tue, Apr 3, 2018, 8:03 PM Mojca Miklavec  wrote:

> On 3 April 2018 at 15:55, Vishnu  wrote:
> > Hello
> >
> >
> > Is the graph fine?
>
> Yes, the one graph is ok. I'm writing the feedback right now (I'm just
> back from work) with further suggestions.
>
> But you should probably start working on the django part and I'll send
> you feedback about the html site in the meantime.
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Please go through this https://jsfiddle.net/vishnum98/3r4vL4L3/21/

I did some changes.

Thanks

On 3 April 2018 at 03:02, Mojca Miklavec  wrote:

>
> V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:
>
>> In the database.
>> Because then it would be very easy to count the number of os for that
>> port.
>>
>
> I'll explain tomorrow why this is suboptimal. (But there's no need to
> further optimise the database design right now.)
>
> Just plan the final result now rather than how the database queries should
> look like.
>
> Mojca
>
>>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

V pon., 2. apr. 2018 19:49 je oseba Vishnu napisala:

> In the database.
> Because then it would be very easy to count the number of os for that port.
>

I'll explain tomorrow why this is suboptimal. (But there's no need to
further optimise the database design right now.)

Just plan the final result now rather than how the database queries should
look like.

Mojca

>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

In the database.
Because then it would be very easy to count the number of os for that port.

Ok.
would be updating the html.


On 2 April 2018 at 22:54, Mojca Miklavec  wrote:

> On 2 April 2018 at 18:50, Vishnu wrote:
> > Hi
> >
> > Here is the basic structure i prepared.
> >
> > https://jsfiddle.net/3r4vL4L3/5/
> >
> > And i wanted to know what all am i supposed to show in installation
> > statistics mandatorily?
> >
> > Top port Versions of a month?
> > Top os of that port?
> > Number of installations versus time ? (of that particular port)
> >
> > what else?
>
> Just get an inspiration from
> http://stats.macports.neverpanic.de/installed_ports
> https://trac.macports.org/wiki/StatisticsIdeas
> or from one particular port (which is what your port page should most
> likely show):
> http://stats.macports.neverpanic.de/categories/8/ports/587
>
> With those few exceptions:
> - The "All versions" cumulated since the beginning of statistics
> collection is pretty useless.
> - The "All variants" is somewhat controversial (I'm not even sure what
> to do with it), but if, then it should be shown as a line chart rather
> than pie chart
> - I would add OS version and build architecture like you suggested.
>
> I would vote for line charts with time on the x axis for nearly everything.
>
> (Of course we'll need another page for statistics, like OS version
> distribution etc that doesn't depend on individual ports, but let's
> not bother about that now.)
>
> > Also in the port history table. Should we add another coloumn named os .
> > which will store the current os version??
>
> Database table or the table on the website?
>
> If you were asking about the table on the website with build summary,
> then I would just replace builder name in that table with OS version +
> arch (10.5/ppc, 10.6/i386, 10.6/x86_64, ...). I would slightly change
> the wording in "Reason", but that's nitpicking. (I could add some
> other minor changes on the list, but they don't feel so important
> right now.)
>
> Mojca
>

Re: Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 2 April 2018 at 18:50, Vishnu wrote:
> Hi
>
> Here is the basic structure i prepared.
>
> https://jsfiddle.net/3r4vL4L3/5/
>
> And i wanted to know what all am i supposed to show in installation
> statistics mandatorily?
>
> Top port Versions of a month?
> Top os of that port?
> Number of installations versus time ? (of that particular port)
>
> what else?

Just get an inspiration from
http://stats.macports.neverpanic.de/installed_ports
https://trac.macports.org/wiki/StatisticsIdeas
or from one particular port (which is what your port page should most
likely show):
http://stats.macports.neverpanic.de/categories/8/ports/587

With those few exceptions:
- The "All versions" cumulated since the beginning of statistics
collection is pretty useless.
- The "All variants" is somewhat controversial (I'm not even sure what
to do with it), but if, then it should be shown as a line chart rather
than pie chart
- I would add OS version and build architecture like you suggested.

I would vote for line charts with time on the x axis for nearly everything.

(Of course we'll need another page for statistics, like OS version
distribution etc that doesn't depend on individual ports, but let's
not bother about that now.)

> Also in the port history table. Should we add another coloumn named os .
> which will store the current os version??

Database table or the table on the website?

If you were asking about the table on the website with build summary,
then I would just replace builder name in that table with OS version +
arch (10.5/ppc, 10.6/i386, 10.6/x86_64, ...). I would slightly change
the wording in "Reason", but that's nitpicking. (I could add some
other minor changes on the list, but they don't feel so important
right now.)

Mojca

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hi

Here is the basic structure i prepared.

https://jsfiddle.net/3r4vL4L3/5/

And i wanted to know what all am i supposed to show in installation
statistics mandatorily?

Top port Versions of a month?
Top os of that port?
Number of installations versus time ? (of that particular port)

what else?

Also in the port history table. Should we add another coloumn named os
. which will store the current os version??

Databse Spreadsheet :
*https://docs.google.com/spreadsheets/d/1Kgbpl1aHn-10fjXh-f4Wz24-TQw4Coaucidpxp7ZqOQ/edit?usp=sharing
*

Thanks

On 2 April 2018 at 19:44, Vishnu  wrote:

> Hello,
>
> I would be getting back on the project from now.
> Sorry i could not deliver soon because i was having my exams going on.
> Now i would deliver my best.
>
> Thanks
>
> On 2 April 2018 at 00:36, Mojca Miklavec  wrote:
>
>> On 1 April 2018 at 20:33, Vishnu  wrote:
>> > Hello mojca
>> >
>> > I am planning on submitting the simple static form of the webpage(for
>> port).
>> >
>> > Had some doubts for the same. Is css required for that? As of now while
>> > showing you the structure?
>> >
>> > Right now its in plain html.
>> > That's  why.
>>
>> The very first step is to have a sound understanding and clear plan
>> for the content, there's no need to prepare css for that.
>>
>> Then we expect some proof of coding skills to build a simple django app.
>>
>> If you have time left, it would be nice to see a nicely looking html,
>> but you should concentrate on content for now and only add design at
>> the end.
>>
>> Mojca
>>
>
>

Re: Gsoc 18 Project | Collect build statistics

[Vishnu]

Hello,

I would be getting back on the project from now.
Sorry i could not deliver soon because i was having my exams going on.
Now i would deliver my best.

Thanks

On 2 April 2018 at 00:36, Mojca Miklavec  wrote:

> On 1 April 2018 at 20:33, Vishnu  wrote:
> > Hello mojca
> >
> > I am planning on submitting the simple static form of the webpage(for
> port).
> >
> > Had some doubts for the same. Is css required for that? As of now while
> > showing you the structure?
> >
> > Right now its in plain html.
> > That's  why.
>
> The very first step is to have a sound understanding and clear plan
> for the content, there's no need to prepare css for that.
>
> Then we expect some proof of coding skills to build a simple django app.
>
> If you have time left, it would be nice to see a nicely looking html,
> but you should concentrate on content for now and only add design at
> the end.
>
> Mojca
>