Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Ralph Seichter]

* Mojca Miklavec:

> I would not dismiss the initial effort as a GSOC project as long as
> the student can prove the competence to do some good work and
> reasonable potential to stick with the project afterwards.

The way I understand GSoC projects is that smaller, self-contained work
is preferred. Rewriting MacPorts is way out of scope for that. It is too
big a task for a "summer" project, requires solid knowledge about the
existing MacPorts environment and experience in software design. I would
not entrust a student with that type of work.

-Ralph

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Mojca Miklavec]

Hi,

On Thu, 2 Apr 2020 at 20:37, Joshua Root wrote:
>
> I would say it's OK for a GSoC project to be completely original and
> never discussed before; the problem with this one is just the scope. You
> would need to spend the entire summer (if not more) just doing design in
> order to do it right, but the primary assessable output of the project
> is code. Naturally you need to do some design before you code, and other
> tasks like documentation of the code, but you need to write the code.
>
> It's also OK for a GSoC project to have research elements. A successful
> project may produce code that never actually gets put into production --
> maybe in the course of implementation it becomes clear that the entire
> approach is flawed. Learning that is still valuable.
>
> In this particular case, if the student can suggest a subset of the
> project with a small enough scope to be completed over the summer, that
> will still produce working code, it could be considered. I have no idea
> what that would look like, but it may be possible. As Ryan implied, an
> important question is "Which parts should be rewritten in Python, and why?"
>
> - Josh

I just want to say that I completely agree with Josh here.

I would love to see MacPorts (re)written in Python, this has been my
wish since years, but I realize that I neither have sufficient
expertise nor time to do it (I guess I could gain the expertise, but
that would again require insane amounts of time). And it's not like we
have an army of developers willing to do this at this very moment, but
if it was written in Python (in a nicely structured way), I guess that
the number of contributors to base would grow quite a bit.

I would say that the main reason why this hasn't been discussed (loud)
before is that developers understand how much work this is, and it
takes quite some expertise and quite some work to get it done right.
And rewriting is probably even more challenging than writing it from
scratch, since one would want to keep compatibility with the existing
packages, and ideally reproduce the exact same results. (We did in
fact discuss this during the last meeting, but not as realistic to
happen any time soon.)

But it's not entirely impossible to do it either. I mean ... MacPorts
base has been written at some point. HomeBrew has been written at some
point. Plenty of other package managers have been written from scratch
at some point. I would not dismiss the initial effort as a GSOC
project as long as the student can prove the competence to do some
good work and reasonable potential to stick with the project
afterwards. GSOC could be done partially as a research project. But
it's of course critical to have a very good student working on this.

If we would ever want to have the base rewritten (independent of
whether or not it happens in the scope of GSOC) and not want to end up
with yet another competitor to MacPorts, we would want to keep the
Portfile syntax working (one could later remove some hard-core tcl
from ports if there was an alternative to achieve the same). One could
potentially start with using Python to add useful new functionality
that's still missing and is either very annoying to write in Tcl, or
has a bigger contributor potential if written in Python; we could for
example support something like "port health " that would
fetch the data from the web api and report to the user whether the
port and its dependencies are know to build or break, whether there
are any critical bugs open in Trac, ... (just as a random example).
Some further action could later be replaced step by step, and in
parallel some bottom-up design would need to be built in Python to
slowly migrate over. (I'm not competent enough to mentor such a
project though.)

It's definitely not an easy task, but if the student can prove
competence to solve some problems in a superior way ...

Mojca

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Joshua Root]

On 2020-4-3 04:52 , Jackson Isaac wrote:
> Hi Ryan,
> 
> Thanks for your valuable insights and I really appreciate the detailed
> feedback. I didn't realize these things before and the efforts that
> went into creating and maintaining such a great project.
> 
> I sincerely apologize to the community and also to the student for my
> ignorance and the fact that I didn't discuss this before in detail on
> the mailing list and understand the implications that it would make
> before putting anything on the ideas page.
> 
> It was a pre-mature action from my side, and my intention was not to
> hurt anyone. This was also a learning for me personally to not put
> things in place without much thought, discussion and understanding.

I would say it's OK for a GSoC project to be completely original and
never discussed before; the problem with this one is just the scope. You
would need to spend the entire summer (if not more) just doing design in
order to do it right, but the primary assessable output of the project
is code. Naturally you need to do some design before you code, and other
tasks like documentation of the code, but you need to write the code.

It's also OK for a GSoC project to have research elements. A successful
project may produce code that never actually gets put into production --
maybe in the course of implementation it becomes clear that the entire
approach is flawed. Learning that is still valuable.

In this particular case, if the student can suggest a subset of the
project with a small enough scope to be completed over the summer, that
will still produce working code, it could be considered. I have no idea
what that would look like, but it may be possible. As Ryan implied, an
important question is "Which parts should be rewritten in Python, and why?"

- Josh

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Jackson Isaac]

Hi Ryan,

On Thu, Apr 2, 2020 at 3:44 AM Ryan Schmidt  wrote:
>
> On Apr 1, 2020, at 18:20, Ryan Schmidt wrote:
>
> > On Mar 31, 2020, at 16:14, Alex Ionkov wrote:
> >
> >> I submitted a proposal this year for rewriting parts of MacPorts in 
> >> Python. The eventual goal is to rewrite all of MacPorts in Python to 
> >> increase modularity and make integration of other APIs with MacPorts 
> >> easier.
> >
> > Is this an idea you just came up with or has there been some discussion of 
> > this before? I didn't see this idea on our GSoC page [1] and it comes as a 
> > surprise to me.
> >
> > [1] https://trac.macports.org/wiki/SummerOfCode#Projects
>
> I see now that it is actually on the list, as just a one-line item:
>
> https://trac.macports.org/wiki/SummerOfCode?action=diff=337_version=336
>
> So then my comments are not so much about your proposal, Alex, as it is about 
> the fact that this item got onto the list at all.
>
>
> MacPorts has 18 years worth of Tcl code which not even us long-time MacPorts 
> developers fully understand. It works as is, and in the spirit of not 
> modifying a working system, my inclination would be not to change it. I may 
> not know where everything is or how everything works in MacPorts base, but I 
> know where many things are and how many things work, and I fear that 
> knowledge would be lost if it were rewritten.  Certainly we should fix bugs. 
> Certainly we should make incremental improvements like whatever is necessary 
> to allow us to update to Tcl 8.6 (i.e. fix the "catch" problem). But 
> rewriting MacPorts wholesale in another programming language seems completely 
> infeasible to me.
>
> I'm sure that rewriting MacPorts base would introduce new bugs, and I'm not 
> confident that our existing test suite would be adequate to catch them. In 
> fact, our test suite is written in Tcl, so there's double the possibility of 
> missing bugs: either because the test suite didn't cover a particular 
> scenario or because rewriting the test suite in Python introduced new bugs 
> into it.
>
> It's not just rewriting MacPorts base either. The more than 10,000 portfiles 
> and the portgroups are also written in Tcl. Should they all be rewritten in 
> Python as well? Doing that manually would take colossal effort, and doing it 
> automatically would require writing a fairly full-featured Tcl-to-Python 
> converter. Or if you propose that the portfiles should be kept in Tcl, then 
> that means Python would have to launch Tcl to interpret each portfile, which 
> seems unnecessarily more complex than what we have now. Using yet another 
> programming language in MacPorts makes it that much harder for new developers 
> to get up to speed.
>
> Note that Tcl was deliberately chosen because of its simple syntax. If we 
> convert portfiles to Python, would that make basic everyday usage more 
> verbose and if so do we really want that? As one specific example, setting 
> the name of a port to "foo" is currently as simple as writing:
>
> name foo
>
> If portfiles become Python files, I am guessing that would have to be 
> rewritten as:
>
> name = 'foo'
>
> and that kind of superfluous punctuational verbosity is what the original 
> designers of MacPorts were trying to avoid.
>
> There are also a lot of little utilities in the macports-contrib repository 
> that hook into MacPorts via Tcl. Our buildbot setup uses several more such 
> scripts developed just for that purpose. All of these would then need to be 
> rewritten in Python.
>
> Needless to say, this also requires a thorough understanding of Python, both 
> to perform this massive one-time conversion and then on an ongoing basis by 
> all MacPorts developers and maintainers. Maybe many people are already 
> familiar with Python, but some MacPorts developers like myself are more 
> familiar with Tcl at this point. Perhaps we could all retrain ourselves to 
> learn Python. But I can't help but recall how four years ago we undertook the 
> massive conversion of our source code from Subversion to GitHub, from which I 
> still haven't fully recovered; I still can't contribute to MacPorts as 
> effectively anymore as I did when we used Subversion. I'm probably in the 
> minority there as other developers were already familiar with git at the 
> time, and if everybody else wants MacPorts rewritten in Python and it can be 
> done then I won't stand in the way.
>
> I don't recall any of this discussion occurring on the mailing list before. I 
> usually stay away from Google Summer of Code issues, but I would have hoped 
> that GSoC would be used as an opportunity for us to finally implement changes 
> that we have wanted to do for years but never got around to doing, rather 
> than to propose new projects whose ramifications have never been discussed 
> before.

Thanks for your valuable insights and I really appreciate the detailed
feedback. I didn't realize these things before and the efforts that
went into creating and maintaining 

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Ralph Seichter]

* Ryan Schmidt:

> [...] I would have hoped that GSoC would be used as an opportunity for
> us to finally implement changes that we have wanted to do for years
> but never got around to doing, rather than to propose new projects
> whose ramifications have never been discussed before.

Quite so. Not meaning any disrespect to Alex (I have never seen any of
his code), but if the developer team would actually let him have a shot
at rewriting parts of MacPorts in Python, I'd look for ways to sever my
connections with MacPorts. Homebrew seems nice.

My own experience in software development spans more than 35 years, some
of them focused on Python, and think that rewriting significant aspects
of MacPorts in Python is neither desirable nor an achievable goal for a
"summer of code". Proper design alone with all its necessary peer review
stages could take months.

My advice to GSoC participants: Choose a smallish task with clearly
defined goals that can be completed and documented in the alotted time.
Work to meet the highest quality standards, and actually make life a
little better for end users. Don't aim for the stars. MacPorts devs
have spent many, many years on the existing ecosystem, and while there
is of course room for improvement, proposing a rewrite is, as far as I
am personally concerned, a mere demonstration of bravado that does not
inspire confidence in me.

-Ralph

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Ryan Schmidt]

On Apr 1, 2020, at 18:20, Ryan Schmidt wrote:

> On Mar 31, 2020, at 16:14, Alex Ionkov wrote:
> 
>> I submitted a proposal this year for rewriting parts of MacPorts in Python. 
>> The eventual goal is to rewrite all of MacPorts in Python to increase 
>> modularity and make integration of other APIs with MacPorts easier. 
> 
> Is this an idea you just came up with or has there been some discussion of 
> this before? I didn't see this idea on our GSoC page [1] and it comes as a 
> surprise to me.
> 
> [1] https://trac.macports.org/wiki/SummerOfCode#Projects

I see now that it is actually on the list, as just a one-line item:

https://trac.macports.org/wiki/SummerOfCode?action=diff=337_version=336

So then my comments are not so much about your proposal, Alex, as it is about 
the fact that this item got onto the list at all.


MacPorts has 18 years worth of Tcl code which not even us long-time MacPorts 
developers fully understand. It works as is, and in the spirit of not modifying 
a working system, my inclination would be not to change it. I may not know 
where everything is or how everything works in MacPorts base, but I know where 
many things are and how many things work, and I fear that knowledge would be 
lost if it were rewritten.  Certainly we should fix bugs. Certainly we should 
make incremental improvements like whatever is necessary to allow us to update 
to Tcl 8.6 (i.e. fix the "catch" problem). But rewriting MacPorts wholesale in 
another programming language seems completely infeasible to me.

I'm sure that rewriting MacPorts base would introduce new bugs, and I'm not 
confident that our existing test suite would be adequate to catch them. In 
fact, our test suite is written in Tcl, so there's double the possibility of 
missing bugs: either because the test suite didn't cover a particular scenario 
or because rewriting the test suite in Python introduced new bugs into it.

It's not just rewriting MacPorts base either. The more than 10,000 portfiles 
and the portgroups are also written in Tcl. Should they all be rewritten in 
Python as well? Doing that manually would take colossal effort, and doing it 
automatically would require writing a fairly full-featured Tcl-to-Python 
converter. Or if you propose that the portfiles should be kept in Tcl, then 
that means Python would have to launch Tcl to interpret each portfile, which 
seems unnecessarily more complex than what we have now. Using yet another 
programming language in MacPorts makes it that much harder for new developers 
to get up to speed.

Note that Tcl was deliberately chosen because of its simple syntax. If we 
convert portfiles to Python, would that make basic everyday usage more verbose 
and if so do we really want that? As one specific example, setting the name of 
a port to "foo" is currently as simple as writing:

name foo

If portfiles become Python files, I am guessing that would have to be rewritten 
as:

name = 'foo'

and that kind of superfluous punctuational verbosity is what the original 
designers of MacPorts were trying to avoid.

There are also a lot of little utilities in the macports-contrib repository 
that hook into MacPorts via Tcl. Our buildbot setup uses several more such 
scripts developed just for that purpose. All of these would then need to be 
rewritten in Python.

Needless to say, this also requires a thorough understanding of Python, both to 
perform this massive one-time conversion and then on an ongoing basis by all 
MacPorts developers and maintainers. Maybe many people are already familiar 
with Python, but some MacPorts developers like myself are more familiar with 
Tcl at this point. Perhaps we could all retrain ourselves to learn Python. But 
I can't help but recall how four years ago we undertook the massive conversion 
of our source code from Subversion to GitHub, from which I still haven't fully 
recovered; I still can't contribute to MacPorts as effectively anymore as I did 
when we used Subversion. I'm probably in the minority there as other developers 
were already familiar with git at the time, and if everybody else wants 
MacPorts rewritten in Python and it can be done then I won't stand in the way.

I don't recall any of this discussion occurring on the mailing list before. I 
usually stay away from Google Summer of Code issues, but I would have hoped 
that GSoC would be used as an opportunity for us to finally implement changes 
that we have wanted to do for years but never got around to doing, rather than 
to propose new projects whose ramifications have never been discussed before.

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Ryan Schmidt]

On Mar 31, 2020, at 16:14, Alex Ionkov wrote:

> I submitted a proposal this year for rewriting parts of MacPorts in Python. 
> The eventual goal is to rewrite all of MacPorts in Python to increase 
> modularity and make integration of other APIs with MacPorts easier. 

Is this an idea you just came up with or has there been some discussion of this 
before? I didn't see this idea on our GSoC page [1] and it comes as a surprise 
to me.

[1] https://trac.macports.org/wiki/SummerOfCode#Projects

Re: GSoC Proposal: Rewrite key parts of MacPorts in Python

[Ruben Di Battista]

I have no experience to gave you any valuable help on this Alex, just wanted to 
drop some ideas I have in mind. 

I work in the frame of HPC computing and I’m a heavy user of spack 
(https://spack.readthedocs.io/en/latest/). It’s a package manager tailored for 
HPC that shares more than few features and design approach with Macports.
 
- It compiles for Unixes on different architectures (Including Power) and on 
Mac.
- has lots of optimized packages for specific architectures 
- Isolated spack environments and filesystem views

I would argue is a bit less stable for daily use as we do with Macports, but I 
find the packages very pleasant to write and the community is vibrant. So I 
would like to tell you to consider, maybe, to base your eventual work on that. 
Maybe making Macports packages compatible with their API or also using their 
infrastructure and strip most of the HPC-related features replacing them with 
the few, unique, feature Macports provides. Maybe discussing with them on the 
best approach to use their work in order to avoid to reinvent the wheel could 
be beneficial. I think the approach would be probably based on their spack 
environments/view (https://spack.readthedocs.io/en/latest/environments.html).

One of the proposed issues over there is to turn the actual spack “binary” into 
a library that other projects can leverage. Could it be feasible way to 
integrate a mature ecosystem with Macports? Maybe other people from the 
community could give it a look and express their opinion on that. 

Good luck! 

  _   
-. .´  |
  ',  ;|∞∞
˜˜ |∞ RdB
,.,|∞∞
  .'   '.  |
-'   `’

https://rdb.is

On 31 March 2020 at 23:15:02, Alex Ionkov (aion...@gmail.com) wrote:

Dear MacPorts community, 

I submitted a proposal this year for rewriting parts of MacPorts in Python. The 
eventual goal is to rewrite all of MacPorts in Python to increase modularity 
and make integration of other APIs with MacPorts easier. 

I've attached my proposal. As for some edits that have already been recommended 
to me for more measurable goals, by each evaluation I would want to have 
rewritten an X amount of functions or functionality. 

What I would request from the community is advice on which functions would be 
the most valuable and useful and how to split them among the evaluation 
periods. Some recommendations that I have received already include getting 
information from the webapp to implement functionality that is not yet 
available and also rewriting functions such as fetch, dependency calculation, 
livecheck and install.

I'm currently working on integrating a small Python script which tells you the 
latest successful build of a port with the Tcl source as a proof of concept.

Thank you very much for your time,
Alex Ionkov

signature.asc
Description: Message signed with OpenPGP using AMPGpg

GSoC Proposal: Rewrite key parts of MacPorts in Python

[Alex Ionkov]

Dear MacPorts community,

I submitted a proposal this year for rewriting parts of MacPorts in Python.
The eventual goal is to rewrite all of MacPorts in Python to increase
modularity and make integration of other APIs with MacPorts easier.

I've attached my proposal. As for some edits that have already been
recommended to me for more measurable goals, by each evaluation I would
want to have rewritten an X amount of functions or functionality.

What I would request from the community is advice on which functions would
be the most valuable and useful and how to split them among the evaluation
periods. Some recommendations that I have received already include getting
information from the webapp to implement functionality that is not yet
available and also rewriting functions such as fetch, dependency
calculation, livecheck and install.

I'm currently working on integrating a small Python script which tells you
the latest successful build of a port with the Tcl source as a proof of
concept.

Thank you very much for your time,
Alex Ionkov


MacPorts Summer Of Code Application.pdf
Description: Adobe PDF document

GSoC Proposal: Enhancing the webapp

[Arjun Salyan]

Dear Macports community,

Last year during GSoC 19, I wrote the ports webapp under the mentorship of
Mojca and Umesh, we had a functional app by the end of the GSoC period. The
app still needs a lot of improvements- codebase needs to be improved so
that new maintainers can start quickly, more features have to be added,
page load time needs to be taken care of, API and documentation also need
to be completed.

I am also planning to deploy Buildbot 2 to carry out real builds as a part
of this project. The main priority of the project would be to have a robust
webapp by the end of the period. My familiarity with the codebase could be
helpful.

I have shared the draft proposal through the GSoC portal and also attaching
the link below. Any feedback would be constructive- the deadline is
tomorrow (31st March, 18:00 UTC).

https://docs.google.com/document/d/1gFxE-8fhZUm98IZ4R0X0Q6-BkHiYG0R4xj3WtKLbt8E/edit?usp=sharing

Thank You

Re: Submitted the GSoC Proposal

[Vibhansh Gupta]

Here's the link to the document by the way :
A
https://docs.google.com/document/d/1c2DkaHLueqMrY66n-l3nvHecVgsezD2XOR8iaF-kkH4/edit?usp=drivesdk

On Tue, Mar 24, 2020, 4:04 AM Vibhansh Gupta  wrote:

> Hey everyone !
>
> I finally submitted my proposal on the Google Sumer of Code website.
> Please do let me know your thoughts on it, either on the Google Doc as
> comments or via mail.
>
> Either way, I will be grateful for any and for all of your thoughts and
> advice on it. I am super pumped to work and hope you like the ideas I’ve
> presented. Any additional ideas are of course always welcome.
>
> PS, should I attach the link to the Document or upload the PDF here?
>
> Hoping to Hear from you super soon,
> Vibhansh.

Submitted the GSoC Proposal

[Vibhansh Gupta]

Hey everyone !

I finally submitted my proposal on the Google Sumer of Code website. Please do 
let me know your thoughts on it, either on the Google Doc as comments or via 
mail.

Either way, I will be grateful for any and for all of your thoughts and advice 
on it. I am super pumped to work and hope you like the ideas I’ve presented. 
Any additional ideas are of course always welcome.

PS, should I attach the link to the Document or upload the PDF here?

Hoping to Hear from you super soon,
Vibhansh.

Re: GSoC Proposal

[Karan Sheth via macports-dev]

Hey,

I really want to thank you guys for selecting me.
I am really overjoyed and excited about working with you guys!:):)

I will make sure to always put in my best efforts and to do the job
I'm assigned to the best of my abilities. I look forward to an awesome
summer, coding and getting to know the community better and to
contributing as much as possible now and also after the completion of
GSoC.

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

On Fri, 3 May 2019 at 23:27, Karan Sheth wrote:
>
> If PEAR ports are useful to the community then I would like to take
> this as stretch goals / after GSoC goal.

For anyone putting the GSOC mails to "ignore mode" due to heavy
traffic ... maybe leave the comment inside
https://github.com/macports/macports-ports/pull/4192
or in the first ticket.

I have absolutely no clue what they are and whether they are
important, so someone else needs to decide.

To me personally php is kind of a dying language. Not exactly the
case, but I can imagine that any "serious developers" would take the
individual modules from whatever internal build/install system it
offers anyway as opposed to relying on MacPorts, similar to using
"pip". I didn't do the analysis on any of those ports, I would look at
the number of commits and the number of tickets filed. If there is or
was a serious demand, I would expect some activity. Admittedly Haskell
is way more obscure and much less used. But probably the only reason
why there's demand for it (from a more substantial number of people)
is Pandoc.

That said, the decision about whether or not to remove ports now is
relatively orthogonal to whether or not it makes sense to create an
import later.

Mojca

Re: GSoC Proposal

[Karan Sheth via macports-dev]

Hey,

So I came across this recent mail in mailing list[1] stating the PR
for deletion of PHP PEAR ports.
I went through a few of the ports and it seems that once the update
functionality of UPT is completed the updation of these ports could be
easily automated just by adding a PEAR frontend to upt which won't
take much time.
If PEAR ports are useful to the community then I would like to take
this as stretch goals / after GSoC goal.

Thanks,
Karan Sheth

[1] https://lists.macports.org/pipermail/macports-dev/2019-May/040664.html

-- 

           
      

Re: Feedback Request for final GSoC Proposal [Speed Up Trace Mode]

[Mihir Luthra]

Hi,

Sorry for re-mailing. I know it's almost the last moment.
But if possible, please provide any possible feedback.

https://docs.google.com/document/d/14eSXwZ6N1vRcBudaJWlwO5G17dY5B1nHgfhh52tSnv0/edit#

Regards,
Mihir

Feedback Request for final GSoC Proposal [Speed Up Trace Mode]

[Mihir Luthra]

Hi everyone,

Kindly check my final proposal and provide any possible feedback.

https://docs.google.com/document/d/14eSXwZ6N1vRcBudaJWlwO5G17dY5B1nHgfhh52tSnv0/edit#


Regards,
Mihir

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

Just a gentle reminder, If I could get a reply to my previous mail as
the deadline for submission is now less than a day and I would like to
submit the proposal well before the deadline.
Sorry for remailing.

Thanks,
Karan Sheth

On Mon, Apr 8, 2019 at 12:05 AM KARAN SHETH  wrote:
>
> Hey,
>
> > There's nothing specific that I would want to comment on.
> > Well, to me it's not entirely clear how the automatic updates would be
> > supposed to work, but ...
>
> I made some changes to the update part, is it better now or am I still
> missing something?
>
> > And the stretch goals are somewhat short :)
>
> Ya I am aware of this and am waiting for some good idea to put in there.
> Should I add PortGroup for NPM as stretch goal?
>
> > The only thing (mentioned yesterday) is that you either try to support
> > Haskell, or you don't even try. If we want to try (and spend the exact
> > same amount of time as you initially planned), then it makes sense to
> > spend a bit of time earlier than in week 11. If you only start
> > exploring the options in week 11, I would say that it's nearly
> > guaranteed that you would run out of time while potentially waiting
> > for someone else to do something, or waiting for some good ideas to
> > pop up.
> > If you would want to do Haskell, you would probably need to plant a
> > seed early (not spend more than half a day or a day in any case), let
> > it ripe while working on other important features, and then see if
> > this is feasible / worth finishing or not. I still don't know what the
> > major blocks are, but just because something is not written in json
> > should not be the reason to give up. The format still seems parsable
> > to me (not sure how many complications would arise).
> >
> > http://hackage.haskell.org/package/cabal2arch
> > http://hackage.haskell.org/package/cabal2nix
> > http://hackage.haskell.org/package/cabal2spec
> > https://github.com/ddssff/cabal-debian
> >
> > Please note that I'm not implying that this is of vital importance to
> > the project. A well written support for automatic updates of
> > python/ruby/perl/npm ports would be worth a lot more than a bunch of
> > semi-broken haskell/cabal packages.
>
> I have changed the timeline for Haskell now and the basic structure of
> it starts at week3.
>
> Should I add the PR links somewhere in the proposal ?
>
> Thanks,
> Karan Sheth

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

> There's nothing specific that I would want to comment on.
> Well, to me it's not entirely clear how the automatic updates would be
> supposed to work, but ...

I made some changes to the update part, is it better now or am I still
missing something?

> And the stretch goals are somewhat short :)

Ya I am aware of this and am waiting for some good idea to put in there.
Should I add PortGroup for NPM as stretch goal?

> The only thing (mentioned yesterday) is that you either try to support
> Haskell, or you don't even try. If we want to try (and spend the exact
> same amount of time as you initially planned), then it makes sense to
> spend a bit of time earlier than in week 11. If you only start
> exploring the options in week 11, I would say that it's nearly
> guaranteed that you would run out of time while potentially waiting
> for someone else to do something, or waiting for some good ideas to
> pop up.
> If you would want to do Haskell, you would probably need to plant a
> seed early (not spend more than half a day or a day in any case), let
> it ripe while working on other important features, and then see if
> this is feasible / worth finishing or not. I still don't know what the
> major blocks are, but just because something is not written in json
> should not be the reason to give up. The format still seems parsable
> to me (not sure how many complications would arise).
>
> http://hackage.haskell.org/package/cabal2arch
> http://hackage.haskell.org/package/cabal2nix
> http://hackage.haskell.org/package/cabal2spec
> https://github.com/ddssff/cabal-debian
>
> Please note that I'm not implying that this is of vital importance to
> the project. A well written support for automatic updates of
> python/ruby/perl/npm ports would be worth a lot more than a bunch of
> semi-broken haskell/cabal packages.

I have changed the timeline for Haskell now and the basic structure of
it starts at week3.

Should I add the PR links somewhere in the proposal ?

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

On Sat, 6 Apr 2019 at 04:06, Cyril Roelandt wrote:
>
> Upt gives you three kinds of dependencies: build, runtime and test. This
> should be done in the backend.

MacPorts also splits "runtime" dependencies into two types. One are
proper runtime dependencies (not needed for building the package
itself), and others are needed all the time (say, libpng): both during
build and while running the software.

This allows optimisation to the extent that the package builder
doesn't really need to install that dependency when creating the
binaries.

> > 5) please add "supported_archs noarch” after “platforms"; since there 
> > are no architecture dependent files installed (as is often the case for 
> > Python scripts)
>
> This could probably be added to the backend as well!

However it might be non-trivial to figure it out whether a python
module needs to build some sources or not.
Figuring this out might be a standalone task on its own (I'm not
saying that this would take the full summer to implement, but it's
certainly not five minutes work). The question there is whether the
installed files depend on architecture (x86_64 vs. i386 etc.) or if a
package is installing just plain text files.

Mojca

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

> Check your dependencies; this should only happen if your port does not
> depend on the port that provides this file, i.e. python37. You might be
> overwriting the dependency by setting depends_lib rather than appending
> to it.

Changing from this :

if {${name} ne ${subport}} {
depends_build-append \
port:py${python.version}-setuptools

depends_lib-append  port:py${python.version}-spdx

livecheck.type  none
}


to this:

depends_build-append \
port:py${python.version}-setuptools

depends_lib-append  port:py${python.version}-spdx

livecheck.type  none


solved the issue.

I have submitted a PR[1] for spdx, spdx-lookup, upt, upt-pypi,
upt-cpan and upt-macports.

Please review the same.

I would also like to ask if anything else needs to be added or updated
in the proposal as the deadline is nearing.

Thanks,
Karan Sheth

[1] https://github.com/macports/macports-ports/pull/4032

-- 

           
      

Re: GSoC Proposal

[Joshua Root]

On 2019-4-6 11:01 , KARAN SHETH via macports-dev wrote:
> One issue I faced while going over the checklist for PR is:
> When I try installing upt with trace mode (-vst), it fails but other
> than that it installs properly.
> Any way to fix this issue?
> 
> Error:
> 
> --->  Building upt
> Executing:  cd 
> "/opt/local/var/macports/build/_Users_kiteretsu_Downloads_GSoc_upt_testports_python_upt/upt/work/upt-0.6"
> && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
> setup.py --no-user-cfg build
> sh: /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7:
> No such file or directory
> Command failed:  cd
> "/opt/local/var/macports/build/_Users_kiteretsu_Downloads_GSoc_upt_testports_python_upt/upt/work/upt-0.6"
> && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
> setup.py --no-user-cfg build
> Exit code: 127
> Warning: The following existing file was hidden from the build system
> by trace mode:
>   /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7

Check your dependencies; this should only happen if your port does not
depend on the port that provides this file, i.e. python37. You might be
overwriting the dependency by setting depends_lib rather than appending
to it.

- Josh

Re: GSoC Proposal

[Cyril Roelandt]

Hello,

On 2019-04-05 09:22, Renee Otten wrote:
> this looks like a very good start! There are a few things that should be 
> changed and I listed them below (some of them common to all three ports):

What could be interesting here is to improve upt to make sure we never
witness these issues ever again:)

> 
> 1) you’re missing the “license” field (you should have gotten a warning about 
> this when doing “port lint —nitpick py-upt”)

The upt frontend is usually able to gather the license. You need to
write down something that makes sense for Macports, though. This should
be done in the backend.

> 2) the value for “homepage” seems incorrect
> 3) since upt is a tool (i.e., run on the command line and not really used as 
> a library by other packages) only one Python version would suffice. I'd 
> suggest to only add “37” after “python.versions”
> 4) the category of your dependencies is not completely correct. Indeed, 
> py-setuptools is a build-dependency, but py-spdx-lookup is not. The latter is 
> needed when using upt and should, therefore, be listed as 
> “depends_lib-append”.

Upt gives you three kinds of dependencies: build, runtime and test. This
should be done in the backend.

> 5) please add "supported_archs noarch” after “platforms"; since there are 
> no architecture dependent files installed (as is often the case for Python 
> scripts)

This could probably be added to the backend as well!

> 6) a little bit nitpicking: (i) add a new line after the modeline, (2) add 
> “revision 0” after “version”

Same here, this should be fixed in the templates.


Cyril

Re: GSoC Proposal

[Renee Otten]

hi Karan, 


this looks like a very good start! There are a few things that should be 
changed and I listed them below (some of them common to all three ports):

1) you’re missing the “license” field (you should have gotten a warning about 
this when doing “port lint —nitpick py-upt”)
2) the value for “homepage” seems incorrect
3) since upt is a tool (i.e., run on the command line and not really used as a 
library by other packages) only one Python version would suffice. I'd suggest 
to only add “37” after “python.versions”
4) the category of your dependencies is not completely correct. Indeed, 
py-setuptools is a build-dependency, but py-spdx-lookup is not. The latter is 
needed when using upt and should, therefore, be listed as “depends_lib-append”.
5) please add "supported_archs noarch” after “platforms"; since there are 
no architecture dependent files installed (as is often the case for Python 
scripts)
6) a little bit nitpicking: (i) add a new line after the modeline, (2) add 
“revision 0” after “version”

I do agree with adding ports for the frontends and macports-backend. I suppose 
these should be all installed by default in the py-upt port since the program 
isn’t very useful otherwise); one could also add variants but I am not sure if 
that’s all that useful in this case.

Best, 
Renee

> On Apr 5, 2019, at 2:25 AM, KARAN SHETH  wrote:
> 
> Hey,
> 
> So I tried porting upt and in the process had to port spdx-lookup and spdx.
> These are the portfiles for the same [1],[2],[3].
> 
> When I try to install upt there is no error message but while trying
> to run upt via a terminal directly it says the command not found
> whereas if I go to python console and import upt, then it works as expected.
> I guess the problem is related to sys.path
> 
> It would be great if someone could try installing upt using this
> portfiles and check if there's any error.
> 
> Thanks,
> Karan Sheth.
> 
> [1] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-upt/Portfile
> [2] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx-lookup/Portfile
> [3] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx/Portfile
> 
> On Tue, Apr 2, 2019 at 6:11 AM Renee Otten  wrote:
>> 
>> 
>> I generated a portfile for py-upt and tried installing that after a few 
>> manual edits, which will actually be automated in the future.
>> It did install properly without any errors but I guess there's some issue 
>> with the path and because of which it was unuseable, once I figure this out 
>> should I make a PR ?
>> 
>> 
>> Great - you can certainly submit a PR when you have a (mostly) working 
>> version.
>> 
>> Path where pip installs : 
>> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
>> Path where macports installs : 
>> ./opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
>> 
>> 
>> The install path (without the “dot” in front) seems fine. It would probably 
>> be useful for you to take a look at the MacPorts guide 
>> (https://guide.macports.org/) and, more specifically, Chapter 5.
>> 
>> Am I missing anything obvious here cause I have no prior experience 
>> installing local ports.
>> 
>> That’s hard to tell without looking at the Portfile and/or trying it. You 
>> can submit a PR and get feedback over there or, if you’d like some 
>> suggestions before submitting a PR, just point us to your Portfile (I 
>> couldn’t find a clone of the macports-port repository on your GitHub).
>> 
>> Best,
>> Renee
> 
> -- 
> 
>   
>       
> 

Re: GSoC Proposal

[Mojca Miklavec]

On Fri, 5 Apr 2019 at 09:16, KARAN SHETH wrote:
> On Fri, Apr 5, 2019 at 11:59 AM Mojca Miklavec wrote:
> >
> > Dear Karan,
> >
> > No time to test now, but what does "port contents py37-upt" say? It
> > could be called upt-py37 or something. That's a weird consequence of
> > supporting multiple version of python at the same time.
>
>   /opt/local/bin/upt-3.7

Here you have it.

Try running upt-3.7 instead of upt alone.

We could tweak this to be called just "upt", either by putting the
port somewhere else (not call it "py-upt", but plain "upt") and make
it strictly depend just on 3.7. Or make an additional package upt
(outside of python subfolder) which depends on py37-upt and just
creates a symlink. Or ... well, I would not worry about it now.

Mojca

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Fri, Apr 5, 2019 at 11:59 AM Mojca Miklavec  wrote:
>
> Dear Karan,
>
> No time to test now, but what does "port contents py37-upt" say? It
> could be called upt-py37 or something. That's a weird consequence of
> supporting multiple version of python at the same time.

Port py37-upt contains:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/upt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/PKG-INFO
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/SOURCES.txt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/dependency_links.txt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/entry_points.txt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/requires.txt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt-0.6-py3.7.egg-info/top_level.txt
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__init__.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/__init__.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/checksum.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/exceptions.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/licenses.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/log.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/__pycache__/upt.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/checksum.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/exceptions.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/licenses.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/log.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__init__.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__pycache__/__init__.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__pycache__/test_checksum.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__pycache__/test_licenses.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__pycache__/test_log.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/__pycache__/test_upt.cpython-37.pyc
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/test_checksum.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/test_licenses.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/test_log.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/tests/test_upt.py
  
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/upt/upt.py
  /opt/local/bin/upt-3.7

> I would suggest you to submit pull requests with at least py-spdx-*
> (or all three, that's up to you, depending on whether you get upt to
> run), so that we can include them in the repository.

Ok I will do this.
Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

On Fri, 5 Apr 2019 at 08:25, KARAN SHETH  wrote:
>
> Hey,
>
> So I tried porting upt and in the process had to port spdx-lookup and spdx.
> These are the portfiles for the same [1],[2],[3].
>
> When I try to install upt there is no error message but while trying
> to run upt via a terminal directly it says the command not found
> whereas if I go to python console and import upt, then it works as expected.
> I guess the problem is related to sys.path
>
> It would be great if someone could try installing upt using this
> portfiles and check if there's any error.

No time to test now, but what does "port contents py37-upt" say? It
could be called upt-py37 or something. That's a weird consequence of
supporting multiple version of python at the same time.

I would suggest you to submit pull requests with at least py-spdx-*
(or all three, that's up to you, depending on whether you get upt to
run), so that we can include them in the repository.

Mojca

> [1] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-upt/Portfile
> [2] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx-lookup/Portfile
> [3] 
> https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx/Portfile

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

So I tried porting upt and in the process had to port spdx-lookup and spdx.
These are the portfiles for the same [1],[2],[3].

When I try to install upt there is no error message but while trying
to run upt via a terminal directly it says the command not found
whereas if I go to python console and import upt, then it works as expected.
I guess the problem is related to sys.path

It would be great if someone could try installing upt using this
portfiles and check if there's any error.

Thanks,
Karan Sheth.

[1] 
https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-upt/Portfile
[2] 
https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx-lookup/Portfile
[3] 
https://github.com/Korusuke/macports-ports/blob/9d7f1995dfb44f693f90a79f91d8695b9dd7feac/python/py-spdx/Portfile

On Tue, Apr 2, 2019 at 6:11 AM Renee Otten  wrote:
>
>
> I generated a portfile for py-upt and tried installing that after a few 
> manual edits, which will actually be automated in the future.
> It did install properly without any errors but I guess there's some issue 
> with the path and because of which it was unuseable, once I figure this out 
> should I make a PR ?
>
>
> Great - you can certainly submit a PR when you have a (mostly) working 
> version.
>
> Path where pip installs : 
> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
> Path where macports installs : 
> ./opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
>
>
> The install path (without the “dot” in front) seems fine. It would probably 
> be useful for you to take a look at the MacPorts guide 
> (https://guide.macports.org/) and, more specifically, Chapter 5.
>
> Am I missing anything obvious here cause I have no prior experience 
> installing local ports.
>
> That’s hard to tell without looking at the Portfile and/or trying it. You can 
> submit a PR and get feedback over there or, if you’d like some suggestions 
> before submitting a PR, just point us to your Portfile (I couldn’t find a 
> clone of the macports-port repository on your GitHub).
>
> Best,
> Renee

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Wed, Apr 3, 2019 at 2:35 AM Mojca Miklavec  wrote:
>
> Dear Karan,
>
> you may put such statistics and suggestions for action plans in the proposal 
> (or appendix to proposal).

Ok, I'll do that soon

> What about pypackage -> py-pypackage? (I miss such entries on the list, 
> unless those are the ones you intentionally didn't count.)

They are counted under   package ->  py-package (769) as I had just
applied a split on "py-".
Will generate the detailed stats if necessary.

> Did you ignore capitalisation of names?

Yep

> Mojca
>
> On Tue, 2 Apr 2019 at 22:51, KARAN SHETH  wrote:
>>
>> Hey,
>>
>> Just to see how many ports will we have to rename if we take on this
>> path I ran a quick script and following were the results:
>>
>> PyPI NameMacports NameNumber
>>
>> package package   10
>>
>> package py-package   769
>>
>> package py-pypackage1
>>
>> pypackage py-package14
>>
>> --  unclassified30
>>
>> Total PyPI Ports = 824
>>
>> So according to this results it would be best to name all the PyPI as
>> py-package.
>>
>> Note: The script might be a bit off.
>>
>> Thanks,
>> Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

you may put such statistics and suggestions for action plans in the
proposal (or appendix to proposal).

What about pypackage -> py-pypackage? (I miss such entries on the list,
unless those are the ones you intentionally didn't count.)
Did you ignore capitalisation of names?

Mojca

On Tue, 2 Apr 2019 at 22:51, KARAN SHETH  wrote:

> Hey,
>
> Just to see how many ports will we have to rename if we take on this
> path I ran a quick script and following were the results:
>
> PyPI NameMacports NameNumber
>
> package package   10
>
> package py-package   769
>
> package py-pypackage1
>
> pypackage py-package14
>
> --  unclassified30
>
> Total PyPI Ports = 824
>
> So according to this results it would be best to name all the PyPI as
> py-package.
>
> Note: The script might be a bit off.
>
> Thanks,
> Karan Sheth
>

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

Just to see how many ports will we have to rename if we take on this
path I ran a quick script and following were the results:

PyPI NameMacports NameNumber

package package   10

package py-package   769

package py-pypackage1

pypackage py-package14

--  unclassified30

Total PyPI Ports = 824

So according to this results it would be best to name all the PyPI as
py-package.

Note: The script might be a bit off.

Thanks,
Karan Sheth

On Tue, Apr 2, 2019 at 1:10 PM Mojca Miklavec  wrote:
>
>
> On Tue, 2 Apr 2019 at 01:33, KARAN SHETH wrote:
>>
>>
>> Is it possible to rename the existing Python ports as per PYPI so that 
>> issues of duplicates ports can be avoided?
>
>
> Yes, I would be in favour of coming up with an "algorithm" for naming all 
> python ports and then stick to it. Ports which don't follow the convention 
> could and should be renamed.
>
> Pypi packages might be called "foo", "pyfoo", "py-foo", ... and we just need 
> to agree on how to do the name transformation in some well-defined way. Some 
> maintainers changed "pyfoo" into "py-foo", others into "py-pyfoo". This needs 
> to be unified.
>
> Mojca
>

-- 

           
      

Re: Enabled commenting on GSoC Proposal

[Jackson Isaac]

On Tue 2 Apr, 2019, 21:07 Mihir Luthra, <1999mihir.lut...@gmail.com> wrote:

>
>> Mihir,
>>
>> The proposal is view-only. You might need to give comment-access for
>> us to leave feedback on the draft.
>>
>> Done
>

Try to avoid creating multiple mail threads. Thanks

>

Enabled commenting on GSoC Proposal

[Mihir Luthra]

>
>
> Mihir,
>
> The proposal is view-only. You might need to give comment-access for
> us to leave feedback on the draft.
>
> Done

Re: GSoC Proposal

[Mojca Miklavec]

On Tue, 2 Apr 2019 at 01:33, KARAN SHETH wrote:

>
> Is it possible to rename the existing Python ports as per PYPI so that
> issues of duplicates ports can be avoided?
>

Yes, I would be in favour of coming up with an "algorithm" for naming all
python ports and then stick to it. Ports which don't follow the convention
could and should be renamed.

Pypi packages might be called "foo", "pyfoo", "py-foo", ... and we just
need to agree on how to do the name transformation in some well-defined
way. Some maintainers changed "pyfoo" into "py-foo", others into
"py-pyfoo". This needs to be unified.

Mojca

Re: GSoC Proposal

[Renee Otten]

> I generated a portfile for py-upt and tried installing that after a few 
> manual edits, which will actually be automated in the future.
> It did install properly without any errors but I guess there's some issue 
> with the path and because of which it was unuseable, once I figure this out 
> should I make a PR ? 

Great - you can certainly submit a PR when you have a (mostly) working version.

> Path where pip installs : 
> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
> Path where macports installs : 
> ./opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages

The install path (without the “dot” in front) seems fine. It would probably be 
useful for you to take a look at the MacPorts guide 
(https://guide.macports.org/ ) and, more 
specifically, Chapter 5.

> Am I missing anything obvious here cause I have no prior experience 
> installing local ports.

That’s hard to tell without looking at the Portfile and/or trying it. You can 
submit a PR and get feedback over there or, if you’d like some suggestions 
before submitting a PR, just point us to your Portfile (I couldn’t find a clone 
of the macports-port repository on your GitHub).

Best,
Renee

Re: GSoC Proposal

[Cyril Roelandt]

On 2019-04-01 11:39, Mojca Miklavec wrote:
> Cyril: what's your desired destination for such work? I would suggest
> placing the repositories on either
> https://framagit.org/upt
> or create a repository under
> https://github.com/macports-gsoc
> and do the pull request and code review on either of these places
> (just one though), and once the review is done, try to make some
> release (even if a beta one) if that part of the code is ready. Cyril,
> do you have a github account?

I do have a Github account https://github.com/Steap .

I would love for the MacPorts backend to be maintained by the MacPorts
community, so basically, it could live wherever you want :) As for the
NPM backend, it could live in framagit. I really don't mind if
everything goes to github.com/macports-gsoc during GSoC.

> 
> Renee (or anyone else from this list who's familiar with writing
> portfiles etc.), could you please assist in testing the resulting
> packages and providing feedback from the MacPorts point of view?
> 
> What I would like to suggest is for Karan to try to submit a PR for a
> new port py-upt (except that this might be a bit tricky at this point
> before the macports backend is included; thus the suggestion to make a
> beta release).

I think it should not be too hard to release upt-macports 0.1. Then
Karan could install upt, upt-pypi and upt-macports using pip, and
package them using upt itself. 


Cyril

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Mon, Apr 1, 2019 at 8:34 PM Mojca Miklavec  wrote:

> On Mon, 1 Apr 2019 at 16:38, Renee Otten wrote:
> >
> > there is a figure showing a WebUI that is not described (personally I am
> not sure if that would be needed anyway).
>
> My assumption was that Karan planned to submit "an alternative"
> proposal for another project (web app), but I might be wrong.
>
> Ya, I was initially going to submit a alternative proposal for the Build
Static django app but instead I decided to focus on one and make it
excellent.
I had thought that web UI would be a nice addition and would add a visual
touch to creating packages but I see that completing and perfecting the
tool itself is of more importance rather than adding UI which won't be that
useful.

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Mon, Apr 1, 2019 at 8:08 PM Renee Otten  wrote:

> hi Karan, Mojca, Cyril,
>
>
> I am happy to help with testing and provide feedback. I quickly glanced
> over the proposal draft and agree with what Cyril said; in addition, there
> is a figure showing a WebUI that is not described (personally I am not sure
> if that would be needed anyway).
>
> I have removed the webUI from the proposal now.

> I agree that the most important thing is to (i) make the most accurate and
> syntactically correct Portfiles and (ii) provide an update mechanism. See
> below for a few consideration with respect to the MacPorts backend for
> PyPI, but some might apply to other backends as well.
>
> 1) while generating the dependency list, please keep in mind that not all
> Python ports follow the PyPI naming scheme (with respect to capitalization
> amongst others). For example the same package is called “py-codestyle” in
> MacPorts, but “pycodestyle” on PyPI. We probably should follow the PyPI
> naming scheme in the long run, but in cases like this you wouldn’t want to
> (recursively) create a “pycodestyle” package that effectively duplicates a
> port.
>

Is it possible to rename the existing Python ports as per PYPI so that
issues of duplicates ports can be avoided?
If renaming is not possible and if there's a pattern like  “py-codestyle”
in MacPorts, but “pycodestyle” on PyPI then we can query for both
combination to see if the port exists or not and then decide accordingly,
though this would be slower and practically double the number of queries
but just a possible solution.

2) testing whether dependencies are present already in MacPorts could be
> done using “port lint”. If the idea is to use that, it would probably be
> good to extend the functionality of “port lint” to automatically do the
> linting on sub-ports as well (instead of needing to call it on every
> sub-port again).
>

I used port lint today for the first time and still not sure what exactly
it does so first I will understand it.
As far as I understood port lint check for any syntax error in the ports
I guess it would be great to add the auto sub port linting feature.

3) if there are version requirements specified for dependencies it would be
> great to check whether these are met, in addition to just checking whether
> a port for that package is present
>

I guess that would be the plan.

4) ideally the “python_requires” information should be used to only add the
> supported versions that are present in the default “python.versions” field
>

Yeah sorry as of now I had kept it static, it would be dynamic in future
and will only have supported versions.

5) there is no need to add the “md5” checksum
>

Ok

As Cyril said, updating existing packages should be implemented only once
> after considering the best option. I would imagine that the only things the
> “update” function should touch in an existing portfile are “version,
> revision, checksums, depends_build, depends_lib, depends_run”. All other
> lines in the existing portfile should stay the same and care has to be
> taken to minimize formatting changes as not to pollute the diff. The
> easiest way (in my mind) is just to use the “upt package”  command to
> generate a “new" portfile for the package, parse the existing portfile and
> only swap the above-mentioned fields for the newly generated ones.
>

I agree on this and have updated the proposal accordingly.

Again, these are just some initial thoughts and I will take a closer look
> at upt and the work you have done so far. I hope this helps and let me know
> if you have any questions.
>
> Best,
> Renee
>
>
Thanks very much for review.
Is there any other goals or tasks that can be taken under this project?

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Mon, Apr 1, 2019 at 3:09 PM Mojca Miklavec  wrote:

> Dear Karan,
>
> Just a few quick notes (for others as well).
>
> On Sun, 31 Mar 2019 at 22:00, KARAN SHETH via macports-dev wrote:
> >
> > I have uploaded the project.
> > Macports-Backend :  https://github.com/Korusuke/upt-macports
> > NPM-Frontend : https://github.com/Korusuke/upt-npm
>
> Awesome, thank you very much.
>
> Cyril: what's your desired destination for such work? I would suggest
> placing the repositories on either
> https://framagit.org/upt
> or create a repository under
> https://github.com/macports-gsoc
> and do the pull request and code review on either of these places
> (just one though), and once the review is done, try to make some
> release (even if a beta one) if that part of the code is ready. Cyril,
> do you have a github account?
>
> Renee (or anyone else from this list who's familiar with writing
> portfiles etc.), could you please assist in testing the resulting
> packages and providing feedback from the MacPorts point of view?
>
> What I would like to suggest is for Karan to try to submit a PR for a
> new port py-upt (except that this might be a bit tricky at this point
> before the macports backend is included; thus the suggestion to make a
> beta release).
>

I generated a portfile for py-upt and tried installing that after a few
manual edits, which will actually be automated in the future.
It did install properly without any errors but I guess there's some issue
with the path and because of which it was unuseable, once I figure this out
should I make a PR ?

Path where pip installs
: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
Path where macports installs
: 
./opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages

Am I missing anything obvious here cause I have no prior experience
installing local ports.

In any case I would be grateful if some MacPorters could help testing,
> guiding, providing feedback etc.
>
> Thank you,
> Mojca
>

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

I have completed most of the changes that you suggested.
Haskell frontend has been now added to technical details and I guess I got
why you were saying that why it would be tricky to add it as a frontend.
Web UI has been removed seeing that it won't be a good add-on as suggested
by Renne.

Please point out anything that I may have missed or needs to added/updated.
Also sorry for late reply, just being project submission week in college, I
have been busy with it.

Thanks,
Karan Sheth
On Mon, Apr 1, 2019 at 1:16 AM Cyril Roelandt 
wrote:

> Hello,
>
> On 2019-04-01 00:30, KARAN SHETH wrote:
> > I have started drafting my proposal for GSoC 2019. It would be great to
> > have a review on it before I submit the same.
>
> A few comments.
>
> First of all, "upt" is written in lower case, except when found at the
> beginning of a sentence :)
>
> To someone who is new to the subject, it is unclear what cpan2port and
> pypi2port do. Maybe you could write that they automatically write a
> Portile for a piece of software available on CPAN or PyPI, and that it
> is really useful for packagers.
>
> Then it is not really clear why upt would be better suited for this
> task. You could use a few lines to explain that, thanks to its modular
> design, it is easier to maintain, and will allow MacPorts to get
> something to work with minimal effort.
>
> I would probably have a "upt update" subcommand rather than a new option
> to "upt package", but it's nitpicking, and definitely not the most
> challenging part of this feature.
>
> About the "Adding MacPorts Backend to UPT" section: calling it very
> basic may not sound really nice to Mojca :) I think you could also
> mention that the package needs unit tests, which you will happily write!
>
> About the "Adding npm frontend" section. I do not really expect you to
> add support for NPM to all the backends: we should probably stay focused
> on MacPorts. This could be a "bonus" if you find yourself a bit bored at
> the end of the summer, though. You could write "default.nix" instead of
> "nix" in the "Package definitions" column.
>
> About the "Input / Ouptut of UPT" section. The "-o/--output" option
> already exists in upt. It is not always used by the backends though. The
> "-i/--input" option does not exist, and I would rather use the following
> syntax:
>
> $ upt package -f pypi -b macports pkg1,pkg2,pkg3
>
> What do you think?
>
> About the "Support for Package Updates in UPT" section. Some questions
> will need to be answered first. Will all the frontends be able to
> support this? How hard will it be for other backends to implement this?
>
> About the schedule. You mention the support of npm/cpan/pypi for
> Macports in both the pre-GSoC and the Week 1. Just do it in Week 1:)
>
> Week 2: you mention the colored output, but it is not part of the
> "Technical details" section.
>
> Week 3.4: why would you add two methods of updating packages? Shouldn't
> we spend more time thinking about the right way of doing things, rather
> than coding 2 different solutions?
>
> Week 10,11: if you want to add the Haskell frontend to the "Technical
> Details", I can tell you more about why it is going to be a bit more
> difficult than other package managers:)
>
> About the additional questions. Linux is written with a capital "L".
>
>
> I hope this helps!
> Cyril.
>

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

On Mon, 1 Apr 2019 at 16:38, Renee Otten wrote:
>
> there is a figure showing a WebUI that is not described (personally I am not 
> sure if that would be needed anyway).

My assumption was that Karan planned to submit "an alternative"
proposal for another project (web app), but I might be wrong.

Mojca

Re: GSoC Proposal

[Renee Otten]

hi Karan, Mojca, Cyril, 


I am happy to help with testing and provide feedback. I quickly glanced over 
the proposal draft and agree with what Cyril said; in addition, there is a 
figure showing a WebUI that is not described (personally I am not sure if that 
would be needed anyway). 

I agree that the most important thing is to (i) make the most accurate and 
syntactically correct Portfiles and (ii) provide an update mechanism. See below 
for a few consideration with respect to the MacPorts backend for PyPI, but some 
might apply to other backends as well.

1) while generating the dependency list, please keep in mind that not all 
Python ports follow the PyPI naming scheme (with respect to capitalization 
amongst others). For example the same package is called “py-codestyle” in 
MacPorts, but “pycodestyle” on PyPI. We probably should follow the PyPI naming 
scheme in the long run, but in cases like this you wouldn’t want to 
(recursively) create a “pycodestyle” package that effectively duplicates a port.
2) testing whether dependencies are present already in MacPorts could be done 
using “port lint”. If the idea is to use that, it would probably be good to 
extend the functionality of “port lint” to automatically do the linting on 
sub-ports as well (instead of needing to call it on every sub-port again).
3) if there are version requirements specified for dependencies it would be 
great to check whether these are met, in addition to just checking whether a 
port for that package is present
4) ideally the “python_requires” information should be used to only add the 
supported versions that are present in the default “python.versions” field
5) there is no need to add the “md5” checksum

As Cyril said, updating existing packages should be implemented only once after 
considering the best option. I would imagine that the only things the “update” 
function should touch in an existing portfile are “version, revision, 
checksums, depends_build, depends_lib, depends_run”. All other lines in the 
existing portfile should stay the same and care has to be taken to minimize 
formatting changes as not to pollute the diff. The easiest way (in my mind) is 
just to use the “upt package”  command to generate a “new" portfile for the 
package, parse the existing portfile and only swap the above-mentioned fields 
for the newly generated ones.

Again, these are just some initial thoughts and I will take a closer look at 
upt and the work you have done so far. I hope this helps and let me know if you 
have any questions. 

Best,
Renee


> On Apr 1, 2019, at 5:39 AM, Mojca Miklavec  wrote:
> 
> Dear Karan,
> 
> Just a few quick notes (for others as well).
> 
> On Sun, 31 Mar 2019 at 22:00, KARAN SHETH via macports-dev wrote:
>> 
>> I have uploaded the project.
>> Macports-Backend :  https://github.com/Korusuke/upt-macports
>> NPM-Frontend : https://github.com/Korusuke/upt-npm
> 
> Awesome, thank you very much.
> 
> Cyril: what's your desired destination for such work? I would suggest
> placing the repositories on either
>https://framagit.org/upt
> or create a repository under
>https://github.com/macports-gsoc
> and do the pull request and code review on either of these places
> (just one though), and once the review is done, try to make some
> release (even if a beta one) if that part of the code is ready. Cyril,
> do you have a github account?
> 
> Renee (or anyone else from this list who's familiar with writing
> portfiles etc.), could you please assist in testing the resulting
> packages and providing feedback from the MacPorts point of view?
> 
> What I would like to suggest is for Karan to try to submit a PR for a
> new port py-upt (except that this might be a bit tricky at this point
> before the macports backend is included; thus the suggestion to make a
> beta release).
> 
> In any case I would be grateful if some MacPorters could help testing,
> guiding, providing feedback etc.
> 
> Thank you,
>Mojca

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

Just a few quick notes (for others as well).

On Sun, 31 Mar 2019 at 22:00, KARAN SHETH via macports-dev wrote:
>
> I have uploaded the project.
> Macports-Backend :  https://github.com/Korusuke/upt-macports
> NPM-Frontend : https://github.com/Korusuke/upt-npm

Awesome, thank you very much.

Cyril: what's your desired destination for such work? I would suggest
placing the repositories on either
https://framagit.org/upt
or create a repository under
https://github.com/macports-gsoc
and do the pull request and code review on either of these places
(just one though), and once the review is done, try to make some
release (even if a beta one) if that part of the code is ready. Cyril,
do you have a github account?

Renee (or anyone else from this list who's familiar with writing
portfiles etc.), could you please assist in testing the resulting
packages and providing feedback from the MacPorts point of view?

What I would like to suggest is for Karan to try to submit a PR for a
new port py-upt (except that this might be a bit tricky at this point
before the macports backend is included; thus the suggestion to make a
beta release).

In any case I would be grateful if some MacPorters could help testing,
guiding, providing feedback etc.

Thank you,
Mojca

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

I will do the suggested changes and also add the pending stuff and then
submit the draft proposal by tomorrow.
Thanks for the review.

Thanks,
Karan Sheth

On Mon, Apr 1, 2019 at 1:16 AM Cyril Roelandt 
wrote:

> Hello,
>
> On 2019-04-01 00:30, KARAN SHETH wrote:
> > I have started drafting my proposal for GSoC 2019. It would be great to
> > have a review on it before I submit the same.
>
> A few comments.
>
> First of all, "upt" is written in lower case, except when found at the
> beginning of a sentence :)
>
> To someone who is new to the subject, it is unclear what cpan2port and
> pypi2port do. Maybe you could write that they automatically write a
> Portile for a piece of software available on CPAN or PyPI, and that it
> is really useful for packagers.
>
> Then it is not really clear why upt would be better suited for this
> task. You could use a few lines to explain that, thanks to its modular
> design, it is easier to maintain, and will allow MacPorts to get
> something to work with minimal effort.
>
> I would probably have a "upt update" subcommand rather than a new option
> to "upt package", but it's nitpicking, and definitely not the most
> challenging part of this feature.
>
> About the "Adding MacPorts Backend to UPT" section: calling it very
> basic may not sound really nice to Mojca :) I think you could also
> mention that the package needs unit tests, which you will happily write!
>
> About the "Adding npm frontend" section. I do not really expect you to
> add support for NPM to all the backends: we should probably stay focused
> on MacPorts. This could be a "bonus" if you find yourself a bit bored at
> the end of the summer, though. You could write "default.nix" instead of
> "nix" in the "Package definitions" column.
>
> About the "Input / Ouptut of UPT" section. The "-o/--output" option
> already exists in upt. It is not always used by the backends though. The
> "-i/--input" option does not exist, and I would rather use the following
> syntax:
>
> $ upt package -f pypi -b macports pkg1,pkg2,pkg3
>
> What do you think?
>
> About the "Support for Package Updates in UPT" section. Some questions
> will need to be answered first. Will all the frontends be able to
> support this? How hard will it be for other backends to implement this?
>
> About the schedule. You mention the support of npm/cpan/pypi for
> Macports in both the pre-GSoC and the Week 1. Just do it in Week 1:)
>
> Week 2: you mention the colored output, but it is not part of the
> "Technical details" section.
>
> Week 3.4: why would you add two methods of updating packages? Shouldn't
> we spend more time thinking about the right way of doing things, rather
> than coding 2 different solutions?
>
> Week 10,11: if you want to add the Haskell frontend to the "Technical
> Details", I can tell you more about why it is going to be a bit more
> difficult than other package managers:)
>
> About the additional questions. Linux is written with a capital "L".
>
>
> I hope this helps!
> Cyril.
>

-- 

           
      

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

I have uploaded the project.
Macports-Backend :  https://github.com/Korusuke/upt-macports
NPM-Frontend : https://github.com/Korusuke/upt-npm

Note: Macports backend is forked from original repo by Mojca.

Thanks,
Karan Sheth

On Sun, Mar 31, 2019 at 10:56 PM Renee Otten  wrote:

> hi Karan,
>
>
> where is your repository for this project? I wanted to take a look and
> possibly provide some feedback specifically for the Python ports, but have
> a hard time finding it…
>
> Thanks,
> Renee
>
>
> > Date: Sat, 30 Mar 2019 15:46:28 +0530
> > From: KARAN SHETH 
> > To: Cyril Roelandt 
> > Cc: "macports-dev@lists.macports.org"
> >   
> > Subject: Re: GSoC Proposal
> > Message-ID:
> >9wlfhxv...@mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hey Cyril,
> >
> > I have made a merge request as you suggested, please review it if
> anything
> > is wrong.
> >
> > Thanks,
> > Karan Sheth
>
>

-- 

 <https://www.somaiya.edu>        <http://www.somaiya-ayurvihar.org>  
<http://nareshwadi.org>  <http://somaiya.com>  <http://www.helpachild.in>  
<http://nareshwadi.org>

Re: GSoC Proposal

[Cyril Roelandt]

Hello,

On 2019-04-01 00:30, KARAN SHETH wrote:
> I have started drafting my proposal for GSoC 2019. It would be great to
> have a review on it before I submit the same.

A few comments.

First of all, "upt" is written in lower case, except when found at the
beginning of a sentence :)

To someone who is new to the subject, it is unclear what cpan2port and
pypi2port do. Maybe you could write that they automatically write a
Portile for a piece of software available on CPAN or PyPI, and that it
is really useful for packagers.

Then it is not really clear why upt would be better suited for this
task. You could use a few lines to explain that, thanks to its modular
design, it is easier to maintain, and will allow MacPorts to get
something to work with minimal effort.

I would probably have a "upt update" subcommand rather than a new option
to "upt package", but it's nitpicking, and definitely not the most
challenging part of this feature.

About the "Adding MacPorts Backend to UPT" section: calling it very
basic may not sound really nice to Mojca :) I think you could also
mention that the package needs unit tests, which you will happily write!

About the "Adding npm frontend" section. I do not really expect you to
add support for NPM to all the backends: we should probably stay focused
on MacPorts. This could be a "bonus" if you find yourself a bit bored at
the end of the summer, though. You could write "default.nix" instead of
"nix" in the "Package definitions" column.

About the "Input / Ouptut of UPT" section. The "-o/--output" option
already exists in upt. It is not always used by the backends though. The
"-i/--input" option does not exist, and I would rather use the following
syntax:

$ upt package -f pypi -b macports pkg1,pkg2,pkg3

What do you think?

About the "Support for Package Updates in UPT" section. Some questions
will need to be answered first. Will all the frontends be able to
support this? How hard will it be for other backends to implement this?

About the schedule. You mention the support of npm/cpan/pypi for
Macports in both the pre-GSoC and the Week 1. Just do it in Week 1:)

Week 2: you mention the colored output, but it is not part of the
"Technical details" section.

Week 3.4: why would you add two methods of updating packages? Shouldn't
we spend more time thinking about the right way of doing things, rather
than coding 2 different solutions?

Week 10,11: if you want to add the Haskell frontend to the "Technical
Details", I can tell you more about why it is going to be a bit more
difficult than other package managers:)

About the additional questions. Linux is written with a capital "L".


I hope this helps!
Cyril.

Re: GSoC Proposal

[Renee Otten]

Thank Cyril! I have looked at your repository of upt and Mojca’s initial 
backend already; however, it would be helpful to be able to see the 
repository/commits of the GSoC student.


> On Mar 31, 2019, at 2:50 PM, Cyril Roelandt  
> wrote:
> 
> Hello,
> 
> On 2019-03-31 13:26, Renee Otten wrote:
>> where is your repository for this project? I wanted to take a look and 
>> possibly provide some feedback specifically for the Python ports, but have a 
>> hard time finding it…
> 
> You may find the upt code and a lot of modules at
> https://framagit.org/upt . Mojca's backend for MacPorts is available at
> https://github.com/mojca/upt-macports .
> 
> 
> Regards,
> Cyril

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

I have started drafting my proposal for GSoC 2019. It would be great to
have a review on it before I submit the same.
It's still a WIP, once done I will submit the same to GSoC draft proposal.

Link:
https://docs.google.com/document/d/1T0EItL09lk_oL36v6_BgvwRb_9E_FLKVdrOd-Nt8VYo/edit?usp=sharing

Thanks,
Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Cyril Roelandt]

Hello,

On 2019-03-31 13:26, Renee Otten wrote:
> where is your repository for this project? I wanted to take a look and 
> possibly provide some feedback specifically for the Python ports, but have a 
> hard time finding it…

You may find the upt code and a lot of modules at
https://framagit.org/upt . Mojca's backend for MacPorts is available at
https://github.com/mojca/upt-macports .


Regards,
Cyril

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

Sorry but I haven't uploaded the git repo yet as there is no Readme, I will
do it soon and ping you once it done.

Thanks,
Karan Sheth

On Sun, Mar 31, 2019 at 10:56 PM Renee Otten  wrote:

> hi Karan,
>
>
> where is your repository for this project? I wanted to take a look and
> possibly provide some feedback specifically for the Python ports, but have
> a hard time finding it…
>
> Thanks,
> Renee
>
>
> > Date: Sat, 30 Mar 2019 15:46:28 +0530
> > From: KARAN SHETH 
> > To: Cyril Roelandt 
> > Cc: "macports-dev@lists.macports.org"
> >   
> > Subject: Re: GSoC Proposal
> > Message-ID:
> >9wlfhxv...@mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hey Cyril,
> >
> > I have made a merge request as you suggested, please review it if
> anything
> > is wrong.
> >
> > Thanks,
> > Karan Sheth
>
>

-- 

 <https://www.somaiya.edu>        <http://www.somaiya-ayurvihar.org>  
<http://nareshwadi.org>  <http://somaiya.com>  <http://www.helpachild.in>  
<http://nareshwadi.org>

Re: GSoC Proposal

[Renee Otten]

hi Karan, 


where is your repository for this project? I wanted to take a look and possibly 
provide some feedback specifically for the Python ports, but have a hard time 
finding it…

Thanks,
Renee


> Date: Sat, 30 Mar 2019 15:46:28 +0530
> From: KARAN SHETH 
> To: Cyril Roelandt 
> Cc: "macports-dev@lists.macports.org"
>   
> Subject: Re: GSoC Proposal
> Message-ID:
>   
> Content-Type: text/plain; charset="utf-8"
> 
> Hey Cyril,
> 
> I have made a merge request as you suggested, please review it if anything
> is wrong.
> 
> Thanks,
> Karan Sheth

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey Cyril,

I have made a merge request as you suggested, please review it if anything
is wrong.

Thanks,
Karan Sheth

On Sat, Mar 30, 2019 at 4:16 AM Cyril Roelandt 
wrote:

> On 2019-03-30 03:35, KARAN SHETH wrote:
> > I had to update a bit of upt-cpan as it was not parsing description
> properly.
>
> Patches are welcome:) You can connect to
> https://framagit.org/upt/upt-cpan using your Github account, by the way.
>
> Cyril
>
> > Thanks,
> > Karan Sheth
> >
> > --
> >
> >  
> >     
>
> > 
>

-- 

           
      

Re: GSoC Proposal

[Cyril Roelandt]

On 2019-03-30 03:35, KARAN SHETH wrote:
> I had to update a bit of upt-cpan as it was not parsing description properly.

Patches are welcome:) You can connect to
https://framagit.org/upt/upt-cpan using your Github account, by the way.

Cyril

> Thanks,
> Karan Sheth
> 
> -- 
> 
>            
>       
> 

Re: GSoC Proposal

[Cyril Roelandt]

On 2019-03-28 22:56, Mojca Miklavec wrote:
> 
> Support for checksums has been implemented recently:
> 
> https://framagit.org/upt/upt/commit/3f634370bbb80904411ff298d2b79e35c7591d23
> 
> Yes, the file would be downloaded and the checksum would be calculated, but
> this would be done by upt (if some piece of functionality is missing, then
> upt needs to be improved).

If the upstream package archive provides us with the required hashes,
upt just returns that. Otherwise, it downloads the file and computes its
hash, which takes a bit more time.


Cyril

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Mar 29 2019, at 3:26 am, Mojca Miklavec  wrote:
>
> > master_sites git+https://github.com/request/request.git
>
> This line looks a bit suspicious. If the files live on github, we might want 
> to use the github PortGroup anyway, but let's leave that detail for later ...

yeah the link was wrong and I had my doubts, now its fixed in the updated 
script.(more on this below)

> > I still have to add checksum and have a bit of doubt in it - to calculate 
> > checksum it will have to download distfiles from npm and calculate 
> > checksums for that right?
> >
> >
>
>
> Yes.
>
> Support for checksums has been implemented recently:
> https://framagit.org/upt/upt/commit/3f634370bbb80904411ff298d2b79e35c7591d23
>
>
>
> Yes, the file would be downloaded and the checksum would be calculated, but 
> this would be done by upt (if some piece of functionality is missing, then 
> upt needs to be improved).
>
> > Another doubt that I have is how to test the Portfiles generated,because if 
> > I try to directly to install this files then it tries to get distfiles from 
> > the macport server I guess(I have never done this so have no clue on how to 
> > test it)
> >
>
>
> If you try to install the port, MacPorts will first try to fetch the binary 
> package from the server. Of course the binary package will not exist, so it 
> will try to build it locally (which is what you want). While building 
> locally, it will first check the macports mirrors for the source, which will 
> again not be there, so it will (after three tries or so) turn to the original 
> source. You can also run "sudo port -v fetch npm-request --no-mirror" to 
> download directly. But the above URL (master_sites) seems wrong, so it won't 
> work like that.
>
> Here's a version that would fetch the correct file (you can run "sudo port -v 
> extract npm-request" on it), but it won't install just yet.
>
> Probably the easiest way is to do a full git clone of the macports-ports 
> repository and then configure macports to look into it:
> https://guide.macports.org/chunked/development.local-repositories.html
>
>
> Then just add a file, for example
> macports-ports/npm/npm-request/Portfile
> with that contents, run "portindex" inside macports-ports and then keep 
> playing with installation.
>
>
Yeah previously I had created a local repository (without cloning) and it 
worked so far but now for perl branches I guess cloning will be needed.

> > Also for every npm package multiple versions are available so should the 
> > script ask which version to use or by-default newest version.
> >
> >
>
>
> This is a million dollar question. We usually package just the latest version 
> of everything (except for exceptions ... :)
>
> We could package multiple versions, where you have two options:
> - install all at the same time
> - allow installing just one version at a time, others may not be installed
>
> I suspect that the first approach won't work. The second one is often 
> problematic.
>
> With python or perl we ship multiple versions of python / perl, but for each 
> of them either only a single version of any python package, or exceptionally 
> multiple versions, but then they conflict with each other which is highly 
> suboptimal.
>
> In all honesty, if there are too many cases where the latest version of a 
> package doesn't work (another package requires an older version), it's 
> questionable whether we can achieve anything useful with npm at all.
>
Ok so as of now I have kept it to latest version, we can work this out later I 
guess
> > After this I would move on to automating the test, but for that I need help 
> > first with manual testing for npm.
> >
> >
> >
>
>
> Do you mean manually testing whether the port builds at all, or running the 
> unit tests?
>
>
>
>
>
>
>

By manual testing I meant port builds.

So now macports backend for npm and perl is mostly completed, I might have 
missed a few things but that can be solved/added later on.
Updated npm Example:
upt package -f npm -b macports request
Output:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
PortSystem 1.0

name npm-request
version 2.88.0
categories npm
maintainers nomaintainer
description Simplified HTTP request client.
long_description ${description}

platforms darwin
license Apache-2.0
homepage https://github.com/request/request#readme

distname request-2.88.0
distfiles request-2.88.0.tgz
master_sites https://registry.npmjs.org/request/-/request-2.88.0.tgz

checksums md5 d2f4658aa40c2abdca764aa40c6fca07 \
sha256 61af775fd5eb513dfac1569edb60518a65c10a47009abd8d5074ba6b434adaa4 \
rmd160 1d9313e896203695be177c82d63ed200bf0bf20f \
size 58023
Example for perl:
upt package -f cpan -b macports Amazon-S3

Output:
# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
PortSystem 1.0
PortGroup perl5 1.0

perl5.branches 5.26 5.28

Re: GSoC Proposal

[Mojca Miklavec]

On Thu, 28 Mar 2019 at 19:28, KARAN SHETH  wrote:

> Hey,
>
> [image: Sent from Mailspring]
> On Thu, Mar 28, 2019 at 8:48 PM Mojca Miklavec  wrote:
>
>
> We are looking forward. I hope that Cyril will also help with the process
> (my encounter with UPM so far was for the full two hours while hacking on
> it :).
>
>
> I've written a basic npm frontend and macports backend for it so if we do
> something like:
> upt package -f npm -b macports request
>
> this is the output:
>
> # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil;
> c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
> PortSystem  1.0
>
>
> namenpm-request
> version 2.9.3
>
> maintainers nomaintainer
> description Simplified HTTP request client.
> long_description${description}
>
> platforms   darwin
> license Apache-2.0
> homepagehttps://github.com/request/request#readme
>
> master_sitesgit+https://github.com/request/request.git
>

This line looks a bit suspicious. If the files live on github, we might
want to use the github PortGroup anyway, but let's leave that detail for
later ...


> I still have to add checksum and have a bit of doubt in it - to calculate
> checksum it will have to download distfiles from npm and calculate
> checksums for that right?
>

Yes.

Support for checksums has been implemented recently:

https://framagit.org/upt/upt/commit/3f634370bbb80904411ff298d2b79e35c7591d23

Yes, the file would be downloaded and the checksum would be calculated, but
this would be done by upt (if some piece of functionality is missing, then
upt needs to be improved).


> Another doubt that I have is how to test the Portfiles generated,because
> if I try to directly to install this files then it tries to get distfiles
> from the macport server I guess(I have never done this so have no clue on
> how to test it)
>

If you try to install the port, MacPorts will first try to fetch the binary
package from the server. Of course the binary package will not exist, so it
will try to build it locally (which is what you want). While building
locally, it will first check the macports mirrors for the source, which
will again not be there, so it will (after three tries or so) turn to the
original source. You can also run "sudo port -v fetch npm-request
--no-mirror" to download directly. But the above URL (master_sites) seems
wrong, so it won't work like that.

Here's a version that would fetch the correct file (you can run "sudo port
-v extract npm-request" on it), but it won't install just yet.

Probably the easiest way is to do a full git clone of the macports-ports
repository and then configure macports to look into it:
https://guide.macports.org/chunked/development.local-repositories.html

Then just add a file, for example
macports-ports/npm/npm-request/Portfile
with that contents, run "portindex" inside macports-ports and then keep
playing with installation.



> Also for every npm package multiple versions are available so should the
> script ask which version to use or by-default newest version.
>

This is a million dollar question. We usually package just the latest
version of everything (except for exceptions ... :)

We could package multiple versions, where you have two options:
- install all at the same time
- allow installing just one version at a time, others may not be installed

I suspect that the first approach won't work. The second one is often
problematic.

With python or perl we ship multiple versions of python / perl, but for
each of them either only a single version of any python package, or
exceptionally multiple versions, but then they conflict with each other
which is highly suboptimal.

In all honesty, if there are too many cases where the latest version of a
package doesn't work (another package requires an older version), it's
questionable whether we can achieve anything useful with npm at all.

I was thinking that it would make sense to create a portgroup for npm and
> other such frontends.
>

Definitely.


> Here my doubt is by autoupdate it means that if the package is updated
> then the corresponding portfile should be updated, right?
>

Yes. (I don't care if for the start you would manually call some update
routine on each file, but getting that to work is tricky enough in itself.)


> but while updating, the manual edits should be taken care of I guess.
>

Yes.


> After this I would move on to automating the test, but for that I need
> help first with manual testing for npm.
>

Do you mean manually testing whether the port builds at all, or running the
unit tests?


> I am still not fully clear with the exact path of stuff, is this the
> correct flow?
>

I believe you mostly got it right.


> upt will be used to generate portfile from npm,cpan,etc.. including
> distfiles and checksums
>

Distfiles are just source files fetches automatically by macports. Usually
you would need to specify 

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Thu, Mar 28, 2019 at 8:48 PM Mojca Miklavec mailto:mo...@macports.org)> wrote:
>
> We are looking forward. I hope that Cyril will also help with the process (my 
> encounter with UPM so far was for the full two hours while hacking on it :).
I've written a basic npm frontend and macports backend for it so if we do 
something like:
upt package -f npm -b macports request

this is the output:
# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
PortSystem 1.0

name npm-request
version 2.9.3

maintainers nomaintainer
description Simplified HTTP request client.
long_description ${description}

platforms darwin
license Apache-2.0
homepage https://github.com/request/request#readme

master_sites git+https://github.com/request/request.git
I still have to add checksum and have a bit of doubt in it - to calculate 
checksum it will have to download distfiles from npm and calculate checksums 
for that right?
Another doubt that I have is how to test the Portfiles generated,because if I 
try to directly to install this files then it tries to get distfiles from the 
macport server I guess(I have never done this so have no clue on how to test it)
Also for every npm package multiple versions are available so should the script 
ask which version to use or by-default newest version.
> > This is what I have thought to do :
> >
> > - For npm frontend:
> > - get the json for any package from 
> > https://registry.npmjs.org/ > (https://registry.npmjs.org/%3Cpackage-name)>
> > - parse it as per upt-frontend template
> >
> >
> > - For macport backend:
> > - First I will fix the issues with the Pypi code and then add npm to it
> >
> >
> > Given all what I have tried out, I expect everything to go smoothly but if 
> > I have any doubt's, I will get in touch.
>
> OK.
>
> > I've also read about port livecheck but haven't tried it yet.
> >
> > One concern/doubt I have is - does the Portfile generated always have to be 
> > correct as I don't think that's achievable given the complexity sometimes, 
> > but sure upt can be a starting point and with few manual edits Portfile can 
> > be perfected.
>
> Yes, the Portfile should always be syntactically correct, even though it may 
> not always be the final / ultimate version. There are definitely cases where 
> external software is needed and the internal name of macports cannot simply 
> be deduced from pypi or npm index. So I expect that further changes might 
> often be needed.
>
> Note that my version did not yet print the checksums. This functionality has 
> been added to upm after I played with it, so the checksums are definitely 
> still missing.
>
> Now, this brings us to the question of how to keep these ports up to date, 
> which might be even more important than creating those ports in the first 
> place. This is something that needs to be investigated and needs some fresh 
> ideas.
>
> We could in principle have some huge index file with all the data from 
> various packages, and then individual portfiles would only list the name and 
> version of each individual port, while the data (including all dependencies) 
> would be fed from a table somewhere else in the system. Or some other clever 
> method for updating the ports needs to be devoted / invented.
>
> See for example
> https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/crossgcc-1.0.tcl
> which simply lists various versions of gcc compiler, and then individual 
> ports only say which version they want:
> https://github.com/macports/macports-ports/blob/master/cross/i386-elf-gcc/Portfile
>
>
>
>

I was thinking that it would make sense to create a portgroup for npm and other 
such frontends.

> I imagine that it would make sense to write MacPorts backend + some selected 
> frontents like npm, and then:
> find an elegant and efficient way of updating ports while keeping in mind 
> that manual additions to packages might be needed (there are multiple options)
>
> (maybe present some proof-of-concept solution for automatic updates of perl 
> ports)
>
> find a way to recursively generate packages (if you need a package and thirty 
> of its dependencies are still missing in our tree, it should be able generate 
> all of them at once)
>
> automate the testing to some extent (pypi2port offers a command that creates 
> the port, fetches the sources, compiles the port and maybe even runs the 
> tests; I did not recheck it, so maybe I remember some details wrong)
>
> pick some areas in MacPorts with weak support (ruby, cabal, npm, ...) and try 
> to improve multiple packages, bringing them up to date, adding dependencies 
> so that software like pandoc, rails etc. could be built, ...
>
>
> These are just random ideas, picking up your favourites is up to you.
>
>
>
>
>
I will first complete Macports backend for all the frontends as for now only 
npm and pypi is working and that too not fully.
once this is done then I will 

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

On Thu, 28 Mar 2019 at 07:52, KARAN SHETH  wrote:

> Resending the mail cause, previous mail was not sent to mailing list.
> On Wed, Mar 27, 2019 at 2:10 PM Mojca Miklavec  wrote:
>
>> Dear Karan,
>>
>> I'm just answering a small part of your question here, a bit more about
>> upt while CC-ing its author & potential mentor:
>>
>> On Tue, 26 Mar 2019 at 18:15, KARAN SHETH 
>> wrote:
>>
>>>
>>>
 (Another option for a project in Python would be bringing universal
 packaging tool [1] to MacPorts.)

>>>
>>> I looked in to this option and it looks very exciting, this is what I
>>> have understood so far about bringing upt to MacPorts :
>>> - A UPT-Frontend and Backend will have to be created for macports.
>>>
>>
>> No, MacPorts in itself requires writing just a backend to support
>> creating packages for existing frontends (currently there are frontends for
>> pypi, ruby gems, cpan), but you usually need a separate backend for each of
>> the frontends. However, it would be great to maybe create new frontends for
>> npm, cabal, ... (npm would be particularly helpful). And if working on that
>> project, it would make sense to concentrate on at least one frontend (ruby
>> gems, npm, or something else) and try to make some awesome improvement of
>> our packages. Our ruby packages are so outdated that they are literally
>> useless, haskell support is horribly lagging behind as so much manual work
>> is needed (we would like to provide the latest version of pandoc), and npm
>> support is basically non-existent.
>>
>> The work would most likely include doing various improvements to UPT
>> itself. In particular, the biggest problem would be how to keep packages
>> up-to-date after the initial creation. We currently have 1000+ perl
>> packages, plenty of python packages etc. and it's a pain to keep them up to
>> date. When packages change, new dependencies are added, and none of that is
>> currently automated. The current version of UPT support creating the inial
>> version of the package, but there is no mechanism in place yet to keep
>> package updated afterwards (incorporating other changes you might have made
>> to the Portfile itself).
>>
>>
>>> So the Frontend will take the Portfile from macports and convert it
>>> using UPT.
>>>
>>
>> No, the frontend would take a package from pypi / cpan / ruby gems / npm
>> / ... and generate a Portfile.
>>
>>
>>> But my doubt is, what is the UPT-Backend in this case, I know it should
>>> be MacOS but like in the example given on UPT repo its converts json from
>>> PYPI into Guix Pxg Definition.
>>> So same way, what would UPT do in case of Macports?
>>>
>>
>> Generate a Portfile
>>
>>
>>> I may be completely wrong here as I have no prior knowledge about UPT.
>>>
>>
>> You aren't expected to have any prior knowledge about it. It's relatively
>> new, currently one-person project, but we had quite some GSOC projects in
>> the part to do various conversion from X to Portfiles (from pypi, cpan),
>> and supporting such a project would solve a lot more packaging nightmares
>> all at once.
>>
>> This is some example code that was cooked together in cca. two hours:
>> https://github.com/mojca/upt-macports
>>
>> I would need to add some documentation, but once you set it up, the
>> command like
>> upt package -f pypi -b macports upt
>> would create a MacPorts package for the "upt" package (that's the last
>> argument) as fetched from pypi (frontend pypi, backend macports, package
>> upt).
>>
>>
>>> Now seeing that I have two choices - first to continue making the django
>>> app or to make upt, I will first look into UPT after my doubts raised above
>>> have been clarified and then decide which route to take.
>>>
>>
>> The choice is yours, you can also submit two proposals, but it makes most
>> sense to really create one awesome proposal rather than trying to submit
>> three, none of which you had a substantial time to do properly. (Or make
>> one really awesome proposal, and a backup one.) If more than one student
>> applies for the same project, we can only select one, but it really makes
>> most sense to apply for the one that you are most likely to shine in.
>>
>> If you plan to apply for this project, what you might want to look into
>> is approximately the following:
>> - get familiar with cpan2port & pypi2port to generate perl and python
>> packages (install it via macports, figure out how to use it, try to create
>> a package, take a glimpse at the source code)
>> - figure out how to create and / or modify perl / python / ruby / ...
>> packages
>> - get familiar with using "port livecheck", maybe find some outdated
>> package that you are using and try to update it (if your interest is in
>> python, search for ports inside "python")
>> - try to install UPT (maybe via virtualenv & pip) and find out how to
>> generate a package with an existing backend (the macports backend on the
>> link above should also be able to generate super simplified python

GSoC Proposal

[KARAN SHETH via macports-dev]

Resending the mail cause, previous mail was not sent to mailing list.
On Wed, Mar 27, 2019 at 2:10 PM Mojca Miklavec  wrote:

> Dear Karan,
>
> I'm just answering a small part of your question here, a bit more about
> upt while CC-ing its author & potential mentor:
>
> On Tue, 26 Mar 2019 at 18:15, KARAN SHETH  wrote:
>
>>
>>
>>> (Another option for a project in Python would be bringing universal
>>> packaging tool [1] to MacPorts.)
>>>
>>
>> I looked in to this option and it looks very exciting, this is what I
>> have understood so far about bringing upt to MacPorts :
>> - A UPT-Frontend and Backend will have to be created for macports.
>>
>
> No, MacPorts in itself requires writing just a backend to support creating
> packages for existing frontends (currently there are frontends for pypi,
> ruby gems, cpan), but you usually need a separate backend for each of the
> frontends. However, it would be great to maybe create new frontends for
> npm, cabal, ... (npm would be particularly helpful). And if working on that
> project, it would make sense to concentrate on at least one frontend (ruby
> gems, npm, or something else) and try to make some awesome improvement of
> our packages. Our ruby packages are so outdated that they are literally
> useless, haskell support is horribly lagging behind as so much manual work
> is needed (we would like to provide the latest version of pandoc), and npm
> support is basically non-existent.
>
> The work would most likely include doing various improvements to UPT
> itself. In particular, the biggest problem would be how to keep packages
> up-to-date after the initial creation. We currently have 1000+ perl
> packages, plenty of python packages etc. and it's a pain to keep them up to
> date. When packages change, new dependencies are added, and none of that is
> currently automated. The current version of UPT support creating the inial
> version of the package, but there is no mechanism in place yet to keep
> package updated afterwards (incorporating other changes you might have made
> to the Portfile itself).
>
>
>> So the Frontend will take the Portfile from macports and convert it using
>> UPT.
>>
>
> No, the frontend would take a package from pypi / cpan / ruby gems / npm /
> ... and generate a Portfile.
>
>
>> But my doubt is, what is the UPT-Backend in this case, I know it should
>> be MacOS but like in the example given on UPT repo its converts json from
>> PYPI into Guix Pxg Definition.
>> So same way, what would UPT do in case of Macports?
>>
>
> Generate a Portfile
>
>
>> I may be completely wrong here as I have no prior knowledge about UPT.
>>
>
> You aren't expected to have any prior knowledge about it. It's relatively
> new, currently one-person project, but we had quite some GSOC projects in
> the part to do various conversion from X to Portfiles (from pypi, cpan),
> and supporting such a project would solve a lot more packaging nightmares
> all at once.
>
> This is some example code that was cooked together in cca. two hours:
> https://github.com/mojca/upt-macports
>
> I would need to add some documentation, but once you set it up, the
> command like
> upt package -f pypi -b macports upt
> would create a MacPorts package for the "upt" package (that's the last
> argument) as fetched from pypi (frontend pypi, backend macports, package
> upt).
>
>
>> Now seeing that I have two choices - first to continue making the django
>> app or to make upt, I will first look into UPT after my doubts raised above
>> have been clarified and then decide which route to take.
>>
>
> The choice is yours, you can also submit two proposals, but it makes most
> sense to really create one awesome proposal rather than trying to submit
> three, none of which you had a substantial time to do properly. (Or make
> one really awesome proposal, and a backup one.) If more than one student
> applies for the same project, we can only select one, but it really makes
> most sense to apply for the one that you are most likely to shine in.
>
> If you plan to apply for this project, what you might want to look into is
> approximately the following:
> - get familiar with cpan2port & pypi2port to generate perl and python
> packages (install it via macports, figure out how to use it, try to create
> a package, take a glimpse at the source code)
> - figure out how to create and / or modify perl / python / ruby / ...
> packages
> - get familiar with using "port livecheck", maybe find some outdated
> package that you are using and try to update it (if your interest is in
> python, search for ports inside "python")
> - try to install UPT (maybe via virtualenv & pip) and find out how to
> generate a package with an existing backend (the macports backend on the
> link above should also be able to generate super simplified python
> packages, but it's not yet complete)
> - you could check https://trac.macports.org/ticket/48324 /
> https://lists.macports.org/pipermail/macports-dev/2018-March/037531.html 

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

I'm just answering a small part of your question here, a bit more about upt
while CC-ing its author & potential mentor:

On Tue, 26 Mar 2019 at 18:15, KARAN SHETH  wrote:

>
>
>> (Another option for a project in Python would be bringing universal
>> packaging tool [1] to MacPorts.)
>>
>
> I looked in to this option and it looks very exciting, this is what I have
> understood so far about bringing upt to MacPorts :
> - A UPT-Frontend and Backend will have to be created for macports.
>

No, MacPorts in itself requires writing just a backend to support creating
packages for existing frontends (currently there are frontends for pypi,
ruby gems, cpan), but you usually need a separate backend for each of the
frontends. However, it would be great to maybe create new frontends for
npm, cabal, ... (npm would be particularly helpful). And if working on that
project, it would make sense to concentrate on at least one frontend (ruby
gems, npm, or something else) and try to make some awesome improvement of
our packages. Our ruby packages are so outdated that they are literally
useless, haskell support is horribly lagging behind as so much manual work
is needed (we would like to provide the latest version of pandoc), and npm
support is basically non-existent.

The work would most likely include doing various improvements to UPT
itself. In particular, the biggest problem would be how to keep packages
up-to-date after the initial creation. We currently have 1000+ perl
packages, plenty of python packages etc. and it's a pain to keep them up to
date. When packages change, new dependencies are added, and none of that is
currently automated. The current version of UPT support creating the inial
version of the package, but there is no mechanism in place yet to keep
package updated afterwards (incorporating other changes you might have made
to the Portfile itself).


> So the Frontend will take the Portfile from macports and convert it using
> UPT.
>

No, the frontend would take a package from pypi / cpan / ruby gems / npm /
... and generate a Portfile.


> But my doubt is, what is the UPT-Backend in this case, I know it should be
> MacOS but like in the example given on UPT repo its converts json from PYPI
> into Guix Pxg Definition.
> So same way, what would UPT do in case of Macports?
>

Generate a Portfile


> I may be completely wrong here as I have no prior knowledge about UPT.
>

You aren't expected to have any prior knowledge about it. It's relatively
new, currently one-person project, but we had quite some GSOC projects in
the part to do various conversion from X to Portfiles (from pypi, cpan),
and supporting such a project would solve a lot more packaging nightmares
all at once.

This is some example code that was cooked together in cca. two hours:
https://github.com/mojca/upt-macports

I would need to add some documentation, but once you set it up, the command
like
upt package -f pypi -b macports upt
would create a MacPorts package for the "upt" package (that's the last
argument) as fetched from pypi (frontend pypi, backend macports, package
upt).


> Now seeing that I have two choices - first to continue making the django
> app or to make upt, I will first look into UPT after my doubts raised above
> have been clarified and then decide which route to take.
>

The choice is yours, you can also submit two proposals, but it makes most
sense to really create one awesome proposal rather than trying to submit
three, none of which you had a substantial time to do properly. (Or make
one really awesome proposal, and a backup one.) If more than one student
applies for the same project, we can only select one, but it really makes
most sense to apply for the one that you are most likely to shine in.

If you plan to apply for this project, what you might want to look into is
approximately the following:
- get familiar with cpan2port & pypi2port to generate perl and python
packages (install it via macports, figure out how to use it, try to create
a package, take a glimpse at the source code)
- figure out how to create and / or modify perl / python / ruby / ...
packages
- get familiar with using "port livecheck", maybe find some outdated
package that you are using and try to update it (if your interest is in
python, search for ports inside "python")
- try to install UPT (maybe via virtualenv & pip) and find out how to
generate a package with an existing backend (the macports backend on the
link above should also be able to generate super simplified python
packages, but it's not yet complete)
- you could check https://trac.macports.org/ticket/48324 /
https://lists.macports.org/pipermail/macports-dev/2018-March/037531.html just
to get an idea about haskell issues and lack of automation there

I just quickly listed some task, feel free to ask for help at any given
point. For any questions specific to UPT I count on Cyril to help you.
Michael: may I count on you for help with any general port-related
questions?

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

Please, can I get a reply so that I can start working on either of the plan?
Thanks,
Karan Sheth

On Tue, Mar 26, 2019 at 10:45 PM KARAN SHETH 
wrote:

> Hey Mojca,
>
> Thanks for replying.
>
> On Tue, Mar 26, 2019 at 3:00 AM Mojca Miklavec  wrote:
>
>> Dear Karan,
>>
>> Warmly welcome to the MacPorts community!
>>
>> On Mon, 25 Mar 2019 at 20:36, KARAN SHETH via macports-dev wrote:
>>
>>> Hey there,
>>> I would like to start by saying this is not GSoC Proposal as the
>>> subject says but I had no idea what else to call it. This is just to make
>>> contact with you regarding a GSoC Project idea that I would really like to
>>> do.
>>>
>>> My name is Karan Sheth. I have attached a doc link wherein all the
>>> personal info and a small bio (I didn't know how big it was needed, sorry
>>> if it's of improper size) and the rough project goal, the abstract is
>>> written.
>>>
>>
>>>I am extremely sorry for establishing contact soo late and I know
>>> there should be no excuse but I was occupied by college exams and other
>>> team activities wherein we organize workshops to teach others. I think I
>>> would be a perfect fit for this project cause of my experience with
>>> scraping and making apps in Django and Flask.
>>>
>>>I would love to hear back from you regarding where should I start
>>> contributing to the project and how should I go about in making my final
>>> proposal, I know I am late to this but I assure you I will try my best to
>>> do any task very fast and properly.
>>>
>>
>> You are not too late just yet, but the more time you have, the easier it
>> is to make a good proposal.
>>
>> The first step you should make is to carefully read the mailing list
>> archive from both last year (from GSOC application period up to GSOC
>> beginning) and this year. We've provided a lot of information for this
>> project in particular as two students already asked about it, and some
>> started working on the demo projects already.
>>
>
> As suggested I went through the past archives and found three projects :
>
> 1) From last year by Vishnu.
> 2) From this year by Arjun Salyan
> *.*
> 3) From this year by Rajdeep Bharati.
>
>
>> What we want you to do is to prove us that you are capable of carrying
>> out the project through the summer and make an outstanding contribution.
>> For that we would like to both see some of your code, as well as a detailed
>> plan of how to carry out the project.
>>
>> Regarding the code: you can either build some demo from scratch, or
>> improve one of the three existing attempts of proof-of-concept apps
>> (upgrading would be better, but if there are too many problems, feel free
>> to start from scratch). Show us how your work would excel when applied to
>> MacPorts.
>> Another smaller challenge could be to create a Portfile for Buildbot 2.1,
>> to assist us in faster migration, as that would also be needed / helpful
>> for communication between the app and buildbot. Or anything else to aid
>> with a better application later.
>>
>
> I plan on starting out with making a demo app for which I will either
> build upon last year's project or start from scratch and draw inspiration
> for the design from both the projects (If it's fine).
> Once the dynamic app is done, I will try making the Portfile for Buildbot
> 2.1 but I guess I will need a bit of help here.
>
>
>> (Another option for a project in Python would be bringing universal
>> packaging tool [1] to MacPorts.)
>>
>
> I looked in to this option and it looks very exciting, this is what I have
> understood so far about bringing upt to MacPorts :
> - A UPT-Frontend and Backend will have to be created for macports.
> So the Frontend will take the Portfile from macports and convert it using
> UPT.
> But my doubt is, what is the UPT-Backend in this case, I know it should be
> MacOS but like in the example given on UPT repo its converts json from PYPI
> into Guix Pxg Definition.
> So same way, what would UPT do in case of Macports?
> I may be completely wrong here as I have no prior knowledge about UPT.
>
> Mojca
>>
>> [1] https://fosdem.org/2019/schedule/event/package_software_with_upt/
>>
>>
>>> Thanks,
>>> Karan Sheth.
>>>
>>>
>>>
>>> https://docs.google.com/document/d/1gFP3pUIVnto9JlqT03X6JHMFzbbd-wqUuhxIAYbyiZg/edit?usp=sharing
>>>
>>
>> Personal biography is probably ok, but the proposal itself is completely
&

GSoC Proposal

[KARAN SHETH via macports-dev]

Hey Mojca,

Thanks for replying.

On Tue, Mar 26, 2019 at 3:00 AM Mojca Miklavec  wrote:

> Dear Karan,
>
> Warmly welcome to the MacPorts community!
>
> On Mon, 25 Mar 2019 at 20:36, KARAN SHETH via macports-dev wrote:
>
>> Hey there,
>> I would like to start by saying this is not GSoC Proposal as the
>> subject says but I had no idea what else to call it. This is just to make
>> contact with you regarding a GSoC Project idea that I would really like to
>> do.
>>
>> My name is Karan Sheth. I have attached a doc link wherein all the
>> personal info and a small bio (I didn't know how big it was needed, sorry
>> if it's of improper size) and the rough project goal, the abstract is
>> written.
>>
>
>>I am extremely sorry for establishing contact soo late and I know
>> there should be no excuse but I was occupied by college exams and other
>> team activities wherein we organize workshops to teach others. I think I
>> would be a perfect fit for this project cause of my experience with
>> scraping and making apps in Django and Flask.
>>
>>I would love to hear back from you regarding where should I start
>> contributing to the project and how should I go about in making my final
>> proposal, I know I am late to this but I assure you I will try my best to
>> do any task very fast and properly.
>>
>
> You are not too late just yet, but the more time you have, the easier it
> is to make a good proposal.
>
> The first step you should make is to carefully read the mailing list
> archive from both last year (from GSOC application period up to GSOC
> beginning) and this year. We've provided a lot of information for this
> project in particular as two students already asked about it, and some
> started working on the demo projects already.
>

As suggested I went through the past archives and found three projects :

1) From last year by Vishnu.
2) From this year by Arjun Salyan
*.*
3) From this year by Rajdeep Bharati.


> What we want you to do is to prove us that you are capable of carrying out
> the project through the summer and make an outstanding contribution. For
> that we would like to both see some of your code, as well as a detailed
> plan of how to carry out the project.
>
> Regarding the code: you can either build some demo from scratch, or
> improve one of the three existing attempts of proof-of-concept apps
> (upgrading would be better, but if there are too many problems, feel free
> to start from scratch). Show us how your work would excel when applied to
> MacPorts.
> Another smaller challenge could be to create a Portfile for Buildbot 2.1,
> to assist us in faster migration, as that would also be needed / helpful
> for communication between the app and buildbot. Or anything else to aid
> with a better application later.
>

I plan on starting out with making a demo app for which I will either build
upon last year's project or start from scratch and draw inspiration for the
design from both the projects (If it's fine).
Once the dynamic app is done, I will try making the Portfile for Buildbot
2.1 but I guess I will need a bit of help here.


> (Another option for a project in Python would be bringing universal
> packaging tool [1] to MacPorts.)
>

I looked in to this option and it looks very exciting, this is what I have
understood so far about bringing upt to MacPorts :
- A UPT-Frontend and Backend will have to be created for macports.
So the Frontend will take the Portfile from macports and convert it using
UPT.
But my doubt is, what is the UPT-Backend in this case, I know it should be
MacOS but like in the example given on UPT repo its converts json from PYPI
into Guix Pxg Definition.
So same way, what would UPT do in case of Macports?
I may be completely wrong here as I have no prior knowledge about UPT.

Mojca
>
> [1] https://fosdem.org/2019/schedule/event/package_software_with_upt/
>
>
>> Thanks,
>> Karan Sheth.
>>
>>
>>
>> https://docs.google.com/document/d/1gFP3pUIVnto9JlqT03X6JHMFzbbd-wqUuhxIAYbyiZg/edit?usp=sharing
>>
>
> Personal biography is probably ok, but the proposal itself is completely
> missing, so there is not much that I could comment on, other than my plea
> to take a slightly better care of grammar. It doesn't have to be perfect
> English, but at least taking care of spacing (usually there is no space
> before punctuation etc.) and even such trivial stuff as trying to use
> capital "I" instead of the lowercase one when referring to yourself make
> quite a difference. We'll be nitpicking about the code a lot, and this is
> the first step ;)
>
> I will be adding the milestones, Timeline, Stretch Goals, etc. later, once
I am familiar with the 

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

Warmly welcome to the MacPorts community!

On Mon, 25 Mar 2019 at 20:36, KARAN SHETH via macports-dev wrote:

> Hey there,
> I would like to start by saying this is not GSoC Proposal as the
> subject says but I had no idea what else to call it. This is just to make
> contact with you regarding a GSoC Project idea that I would really like to
> do.
>
> My name is Karan Sheth. I have attached a doc link wherein all the
> personal info and a small bio (I didn't know how big it was needed, sorry
> if it's of improper size) and the rough project goal, the abstract is
> written.
>

>I am extremely sorry for establishing contact soo late and I know there
> should be no excuse but I was occupied by college exams and other team
> activities wherein we organize workshops to teach others. I think I would
> be a perfect fit for this project cause of my experience with scraping and
> making apps in Django and Flask.
>
>I would love to hear back from you regarding where should I start
> contributing to the project and how should I go about in making my final
> proposal, I know I am late to this but I assure you I will try my best to
> do any task very fast and properly.
>

You are not too late just yet, but the more time you have, the easier it is
to make a good proposal.

The first step you should make is to carefully read the mailing list
archive from both last year (from GSOC application period up to GSOC
beginning) and this year. We've provided a lot of information for this
project in particular as two students already asked about it, and some
started working on the demo projects already.

What we want you to do is to prove us that you are capable of carrying out
the project through the summer and make an outstanding contribution. For
that we would like to both see some of your code, as well as a detailed
plan of how to carry out the project.

Regarding the code: you can either build some demo from scratch, or improve
one of the three existing attempts of proof-of-concept apps (upgrading
would be better, but if there are too many problems, feel free to start
from scratch). Show us how your work would excel when applied to MacPorts.
Another smaller challenge could be to create a Portfile for Buildbot 2.1,
to assist us in faster migration, as that would also be needed / helpful
for communication between the app and buildbot. Or anything else to aid
with a better application later.

(Another option for a project in Python would be bringing universal
packaging tool [1] to MacPorts.)

Mojca

[1] https://fosdem.org/2019/schedule/event/package_software_with_upt/


> Thanks,
> Karan Sheth.
>
>
>
> https://docs.google.com/document/d/1gFP3pUIVnto9JlqT03X6JHMFzbbd-wqUuhxIAYbyiZg/edit?usp=sharing
>

Personal biography is probably ok, but the proposal itself is completely
missing, so there is not much that I could comment on, other than my plea
to take a slightly better care of grammar. It doesn't have to be perfect
English, but at least taking care of spacing (usually there is no space
before punctuation etc.) and even such trivial stuff as trying to use
capital "I" instead of the lowercase one when referring to yourself make
quite a difference. We'll be nitpicking about the code a lot, and this is
the first step ;)

GSoC Proposal

[KARAN SHETH via macports-dev]

Hey there,
I would like to start by saying this is not GSoC Proposal as the
subject says but I had no idea what else to call it. This is just to make
contact with you regarding a GSoC Project idea that I would really like to
do.

My name is Karan Sheth. I have attached a doc link wherein all the
personal info and a small bio (I didn't know how big it was needed, sorry
if it's of improper size) and the rough project goal, the abstract is
written.

   I am extremely sorry for establishing contact soo late and I know there
should be no excuse but I was occupied by college exams and other team
activities wherein we organize workshops to teach others. I think I would
be a perfect fit for this project cause of my experience with scraping and
making apps in Django and Flask.

   I would love to hear back from you regarding where should I start
contributing to the project and how should I go about in making my final
proposal, I know I am late to this but I assure you I will try my best to
do any task very fast and properly.

Thanks,
Karan Sheth.


https://docs.google.com/document/d/1gFP3pUIVnto9JlqT03X6JHMFzbbd-wqUuhxIAYbyiZg/edit?usp=sharing

-- 

 <https://www.somaiya.edu>        <http://www.somaiya-ayurvihar.org>  
<http://nareshwadi.org>  <http://somaiya.com>  <http://www.helpachild.in>  
<http://nareshwadi.org>

Re: GSoC Proposal

[Clemens Lang]

Hi,

On Thu, Apr 13, 2017 at 12:37:55AM +0200, Mojca Miklavec wrote:
> I was talking to someone from the HB community. Apparently they tried
> very hard to make Travis CI work. One of the major problems they had
> other than the time limit (the builder is relatively slow and jobs get
> killed after 50 minutes, so any builds like Qt would fail of course)
> was the fact that it was impossible to upload the binaries from
> "untrusted pull requests" (not sure what that means, in any case, it
> didn't work for them).

Yeah, that was also the impression I had last time I talked to somebody
from homebrew about their build setup.

> Another option being mentioned was CircleCI (no limitation to upload
> files), but that one has an additional limitation of only 500 free
> minutes per month for OpenSource orgs, so most likely not anywhere
> near enough.

I think in the long run we may not have an other option than providing
the build resources ourselves, but that comes with the responsibility of
containing any effects due to the untrusted input.

> From what I understand there are two different kinds of security
> issues:
> 
> (a) People being able to submit arbitrary code as pull requests
> 
> (b) Most of committers are probably not really inspecting diffs or
> full sources of packages. In theory one can write some originally
> useful opensource tool that we package. Once people start using it,
> some problematic code gets introduced, we happily upgrade the port,
> and both our main builder and any user installing that port would be
> affected.
> 
> We currently don't have a problem with (a) because we don't build pull
> requests, and with Travis we would simply outsource the problem to
> someone else. For (b) we currently don't have any reliable solution,
> but using Travis would not actually help since a maintainer would
> probably eventually merge the code (if it looks reasonable). And then
> the code from official repository would be built on the main builder.

As for (b), we try to run things in sandboxes to limit the attack
surface, but sure, that's an issue we currently have. Ideally, we would
also run 'make install' as non-root user to further reduce the risks.
Realistically speaking, all distributions have the problem of trusting
their upstream code, though.

Our main concern for building PRs are attackers that could try to use
our CI infrastructure to run bots for a botnet or use it to send spam by
sending a pull request with a specially crafted Portfile.


> I'm thinking of an alternative approach that would cover another use
> case (which cannot be solved by Travis CI due to limitations).
> 
> Use cases I have in mind:
> - developer testing a new release of Qt; or a bunch of KDE apps
> - developer trying to build all the 1000+ perl modules once perl5.26
> gets released
> - developer without access to older machines, trying to figure out why
> the build fails on 10.6; having some clue why that could be, but
> trying to avoid doing ten semi-random commits to the main repository
> just to test the hypothesis
> 
> What we *could* do is to set up a few more build slaves on the
> "existing" infrastructure (the build slaves could actually be
> anywhere, people could even volunteer to provide access to their
> machines; but I guess the idea would be to eventually set up a few new
> ones next to the existing slaves). The build master could either run
> in the same instance or separately.

We must take care to not risk the security of the build machines that
build our binary archives. They are a valuable target, so we should
isolate machines that build unreviewed user input, ideally using VMs.

> The idea would be that people with commit access would have to
> manually approve building of a particular pull request (and then that
> pull request would get a green icon of course once the build is
> complete). When a pull request gets updated/modified, the build could
> be tested again, but it would have to be triggered manually again.
> 
> For each manually approved/triggered build the build slave would:
> - clone the repository
> - (?) most likely rebase that on top of master to make sure that the
> latest packages of everything would be used at the time of build
> - determine which ports have been modified
> - maybe run lint as an initial quick test (?)
> - build those ports in a very similar way to what the buildbot does at
> this moment, with a few exceptions:
> 
>   (a) packages from other/existing ports would be fetched from the
> main builders, perhaps even when they are not distributable (we could
> have private copies of binary packages)
>   (b) the resulting packages would be stored between individual builds
> that belong to the same PR, but they would not be uploaded anywhere
> and both sources and binaries (basically anything that has been
> fetched or modified) would be deleted once PR testing is complete

I'm not sure having to review each PR before a build is started is
worthwhile. The whole idea of a 

Re: GSoC Proposal

[Zero King]

On 4/12/17 10:37 PM, Mojca Miklavec wrote:

I was talking to someone from the HB community. Apparently they tried
very hard to make Travis CI work. One of the major problems they had
other than the time limit (the builder is relatively slow and jobs get
killed after 50 minutes, so any builds like Qt would fail of course)
was the fact that it was impossible to upload the binaries from
"untrusted pull requests" (not sure what that means, in any case, it
didn't work for them). They set up their own build infrastructure
almost from scratch. We clearly don't need to upload the result
anywhere, but the limitations will stay in any case. I would find it
valuable if we put our effort in something that can be expanded.


Untrusted means I can't run the bot on Travis because any secrets 
available to PRs would be public.

Travis CI is free and I believe this is an advantage for my proposal.
The bot for PRs is not related to Travis and takes more time in my schedule.


Another option being mentioned was CircleCI (no limitation to upload
files), but that one has an additional limitation of only 500 free
minutes per month for OpenSource orgs, so most likely not anywhere
near enough.


Their macOS builds aren't free. I looked into alternatives but Travis 
seems to be the best.



What we *could* do is to set up a few more build slaves on the
"existing" infrastructure (the build slaves could actually be
anywhere, people could even volunteer to provide access to their
machines; but I guess the idea would be to eventually set up a few new
ones next to the existing slaves). The build master could either run
in the same instance or separately.

The idea would be that people with commit access would have to
manually approve building of a particular pull request (and then that
pull request would get a green icon of course once the build is
complete). When a pull request gets updated/modified, the build could
be tested again, but it would have to be triggered manually again.


I had this idea in mind but chose the one in the proposal because I'd 
like to finish deployment in GSoC and changing the Buildbot 
infrastructure remotely could cause problems. On Travis whatever I do I 
won't break anything hard to recover or leak any secrets but it's not so 
safe on Buildbot. I consider virtualization and snapshots the last 
defence against evil PRs and prefer to have them.



The glue code might need some time to be implemented, but it sounds
like something in the same direction that you already proposed with
that "go" bot. To be honest, I currently don't know yet what exactly
would be needed on that part.


Tentative plan:
0. Verify that webhook requests are indeed from GitHub 
(https://developer.github.com/webhooks/securing/)
1. Add update/submission/enhancement based on smart logic (new Portfile 
means submission) or a new "task list" in the PR template.
2. Determine the maintainer of the ports touched, if the submitter is 
one of maintainers for all touched ports not nomaintainer, set the 
maintainer label. If all touched ports are nomaintainer, set the 
nomaintainer label. If a port touched has a maintainer (not PR 
submitter) and is not openmaintainer, set the waitformaintainer label. 
(All label names are tentative.)
3. Send mail to / mention with @ / request review from / assign PRs to 
related maintainers (single maintainer -> assign, etc.). Limit the 
number of maintainers a PR could notify but manual override via 
commenting can be implemented.

4. Send mail to a mailing list (if a new ML for PRs is helpful)

Database:
1. Read Email to GitHub handle mapping from database (if migration to 
GitHub handle in Portfiles didn't finish in time)

2. Cache port to maintainer mapping (?)


One of the things I really miss is a website with easy access to build
summary on per-port basis:
-https://trac.macports.org/ticket/51995#Statistics
(This might be something that might even be possible to implement as a
special view right on the buildbot.)


I'm not familiar with the front end techs. Maybe next time.


Independent: it would be helpful to hear a better
justification/defence for selecting Go as the language of the "helper
bot".


Go has a great set of standard libraries and I can use 
https://github.com/google/go-github. Go programs are easy to compile and 
deploy.


--
Best regards,
Zero King

Re: GSoC Proposal

[Mojca Miklavec]

Hi,

Just to clarify one thing (also to any other GSOC applicant): this
discussion does not provide any indication about the ranking of the
submitted proposal or whether this or any other proposal might be or
not be accepted. There might be better or worse applications where
less clarification is needed. Independent of whether or not the
proposal would be accepted, I felt that this particular idea needs
some further discussion & clarifications.

Another disclaimer to Zero King: I'm clearly not an authority and my
thoughts might not fully match with the views of other developers, so
please don't take all of my words or ideas for granted. This is
supposed to trigger discussion, not suggestions that one should
blindly follow.

I'm also aware that it makes absolutely no sense to convince someone
to go into directions that don't sound like fun to him, so consider
this just brainstorming for now.

On 2 April 2017 at 14:21, Zero King wrote:
>
> Travis won't keep the VMs up. Every build gets a fresh VM (no need to
> cleanup) and could only run for 50 minutes so if a build takes too long it
> will fail anyway.
>
> You're right about high server load. Since we already have the Buildbot
> infrastructure I'll only do install tests in PRs. Lint tests will be done
> for both commits and PRs. I'll limit the number of ports to test so if a PR
> touches too many ports it'll only do lint tests. We can also use tags like
> [ci-skip] in PR title or label to skip some tests.
>
> Installing from packages.macports.org would avoid rebuilding distributable
> ports after minor edits.
>
> Travis is unlikely to ban us but if that happens we can keep only macOS
> 10.12 or only the lint tests (which can even be done on Linux) and contact
> them to unban us.

I was talking to someone from the HB community. Apparently they tried
very hard to make Travis CI work. One of the major problems they had
other than the time limit (the builder is relatively slow and jobs get
killed after 50 minutes, so any builds like Qt would fail of course)
was the fact that it was impossible to upload the binaries from
"untrusted pull requests" (not sure what that means, in any case, it
didn't work for them). They set up their own build infrastructure
almost from scratch. We clearly don't need to upload the result
anywhere, but the limitations will stay in any case. I would find it
valuable if we put our effort in something that can be expanded.

Another option being mentioned was CircleCI (no limitation to upload
files), but that one has an additional limitation of only 500 free
minutes per month for OpenSource orgs, so most likely not anywhere
near enough.

I was thinking about two aspects:
(a) It would in fact be helpful to have some build jobs running
automatically for PRs
(b) But we also need something to test complex builds. Exactly those
where maintainers can hardly afford to build and test them on their
own machines.


>> The alternative can always be to use our own server for builds, but of
>> course that brings other problems to the table (mostly security) and
>> requires a completely different approach.
>
> Yes, mostly security considerations. We'll have to use VMs and revert to
> snapshot after every build in our own servers. Managing VMs and sending data
> around securely would be painful to setup.

>From what I understand there are two different kinds of security issues:

(a) People being able to submit arbitrary code as pull requests

(b) Most of committers are probably not really inspecting diffs or
full sources of packages. In theory one can write some originally
useful opensource tool that we package. Once people start using it,
some problematic code gets introduced, we happily upgrade the port,
and both our main builder and any user installing that port would be
affected.


We currently don't have a problem with (a) because we don't build pull
requests, and with Travis we would simply outsource the problem to
someone else. For (b) we currently don't have any reliable solution,
but using Travis would not actually help since a maintainer would
probably eventually merge the code (if it looks reasonable). And then
the code from official repository would be built on the main builder.


I'm thinking of an alternative approach that would cover another use
case (which cannot be solved by Travis CI due to limitations).

Use cases I have in mind:
- developer testing a new release of Qt; or a bunch of KDE apps
- developer trying to build all the 1000+ perl modules once perl5.26
gets released
- developer without access to older machines, trying to figure out why
the build fails on 10.6; having some clue why that could be, but
trying to avoid doing ten semi-random commits to the main repository
just to test the hypothesis

What we *could* do is to set up a few more build slaves on the
"existing" infrastructure (the build slaves could actually be
anywhere, people could even volunteer to provide access to their
machines; but I guess the idea would be to 

Re: GSoC Proposal

[Zero King]

On 4/2/17 6:03 AM, Mojca Miklavec wrote:

Thank you. Please include that into the proposal as well. It wasn't
clear to me whether your project would be a full "replacement"
compared to the current functionality of the buildbot or just covering
some small load cycle (like doing very very basic checks). It seems
you aim for the former.

In that case there are a few things that potentially worry me.

1.) They have HomeBrew installed. This is probably a bad thing and a
good thing at the same time. The official server should not have any
traces of HomeBrew to avoid linking against /usr/local/whatever
instead of /opt/local (but it's not a dealbreaker for the testing
infrastructure). Then again, we might spot some problems and try to
avoid them when we find such cases.


I'll uninstall Homebrew from the VM if needed.


2.) We sometimes consume almost 100% of server load on a multicore
server (sometimes for a couple of days). And we sometimes run out disk
space. What happens if Travis bans us (or asks for a price that we
cannot afford to pay) right at the moment when the full solution gets
implemented? I believe we should contact Travis upfront to make sure
that they can meet our expectations. And in any case I would like to
see slightly more details about how the plan intends to keep our
footprint (disk resources and cpu cycles) low. Would you uninstall all
existing packages after each trial and install them again from
packages.macports.org for the next build? Would you keep official
packages installed & inactive between multilple runs and just clean up
whatever the pull request tried to build? (Less important: How would
you avoid rebuilding the whole of Qt after, say, minor edit of
comments in a pull request?)


Travis won't keep the VMs up. Every build gets a fresh VM (no need to 
cleanup) and could only run for 50 minutes so if a build takes too long 
it will fail anyway.


You're right about high server load. Since we already have the Buildbot 
infrastructure I'll only do install tests in PRs. Lint tests will be 
done for both commits and PRs. I'll limit the number of ports to test so 
if a PR touches too many ports it'll only do lint tests. We can also use 
tags like [ci-skip] in PR title or label to skip some tests.


Installing from packages.macports.org would avoid rebuilding 
distributable ports after minor edits.


Travis is unlikely to ban us but if that happens we can keep only macOS 
10.12 or only the lint tests (which can even be done on Linux) and 
contact them to unban us.



The alternative can always be to use our own server for builds, but of
course that brings other problems to the table (mostly security) and
requires a completely different approach.


Yes, mostly security considerations. We'll have to use VMs and revert to 
snapshot after every build in our own servers. Managing VMs and sending 
data around securely would be painful to setup.



And yes, it would be great if mpbb sources could be shared (and
improved for both buildbot & travis at the same time). Not a
requirement, but if one part can benefit from the other, that would be
great.

Mojca


--
Best regards,
Zero King

Re: GSoC Proposal

[Zero King]

macOS VMs on Travis would do `port lint  | tee lint.txt; if 
grep "^Error: " lint.txt ; then ...`


and `port install ...` (`port test ...` if test exists) and if one of 
them failed Travis will report that back to GitHub.


Also Travis will keep logs so lint results will be available there.


On 4/1/17 9:43 PM, Mojca Miklavec wrote:

Can you please elaborate a bit more on
 Travis CI will do lint and install tests on macOS 10.10-12 for PRs
and commits.

What does "install tests" refer to exactly?

Mojca


--
Best regards,
Zero King

Re: GSoC Proposal

[Ken Cunningham]

Have you seen how homebrew does this? I imagine he means something like that:

Every submission has to be submitted to the 10.10 - 10.12 bots first, to see if 
it builds.

Every submission is suggested / required to have at least a minimal test 
`myport --version` to make sure something actually works.

Only after those things are submission PRs inhaled.

Ken



On 2017-04-01, at 2:43 PM, Mojca Miklavec wrote:

> Can you please elaborate a bit more on
>Travis CI will do lint and install tests on macOS 10.10-12 for PRs
> and commits.
> 
> What does "install tests" refer to exactly?
> 
> Mojca

Re: GSoC Proposal

[Mojca Miklavec]

Can you please elaborate a bit more on
Travis CI will do lint and install tests on macOS 10.10-12 for PRs
and commits.

What does "install tests" refer to exactly?

Mojca

Re: GSoC Proposal

[Zero King]

On 3/31/17 6:23 PM, Rainer Müller wrote:
First of all, great to see a proposal coming from you as a project 
member! 

Thanks.

We do not have anything else in Go yet, so this would be new to our
infrastructure. I am a bit hesitant with that, because even after this
GSoC, we will need someone to maintain this service. Personally, I also
know next to nothing about Go.

I'll maintain it :)

Go produce static binaries and should be easy to deploy.

The other details given in the proposal are still quite sparse. I do not
yet see everything this project includes and how you will use the time
for the proposed task. Remember the GSoC program is meant to keep you
working for 12 weeks.

Updated the gist with more details.

I was wondering which should I follow " How to work with us" on GSoC 
website or

https://trac.macports.org/wiki/SummerOfCodeApplicationTemplate.

Could you give us a timeline? Do you have any milestones, especially for
the midterm evaluation? What will be the status at the end of GSoC this
summer? Is your plan to have it fully deployed already?

Updated the gist. Yes, I plan to have it fully deployed.

How will the bot work in detail? I guess listening to the webhooks?
Where will it get its database for the maintainers? Or do you intend to
parse the submitted/modified Portfile?

Sure, it will listen to the webhooks and need a URL for that.

It will get the maintainers from unmodified Portfile for security 
considerations.
However if the migration to GitHub handles didn't make it will have to 
get that from a database.

What will the scripts for Travis CI be written in? Can we reuse code
from the buildbot infrastructure, namely mpbb [1] which is written in
bash shell script? What are steps you will take to implement this
functionality?

YAML and Bash. mpbb code can be reused.

To implement this, first let Travis generate archives for base and then
use the archive and modified mpbb to test the PR.

--
Best regards,
Zero King

Re: GSoC Proposal

[Rainer Müller]

Hello,

On 2017-03-31 17:43, Zero King wrote:
> I'd like to implement a bot in Go dealing with pull requests for
> macports-ports on GitHub and utilize Travis CI to test PRs and commits
> in GSoC 2017.
> 
> More details in my draft:
> https://gist.github.com/l2dy/420533b821570e26dc7374898c3264fb
> 
> Any advice is welcome and since this is a new idea I'm seeking for
> potential mentors.

First of all, great to see a proposal coming from you as a project member!

We do not have anything else in Go yet, so this would be new to our
infrastructure. I am a bit hesitant with that, because even after this
GSoC, we will need someone to maintain this service. Personally, I also
know next to nothing about Go.

The other details given in the proposal are still quite sparse. I do not
yet see everything this project includes and how you will use the time
for the proposed task. Remember the GSoC program is meant to keep you
working for 12 weeks.

Could you give us a timeline? Do you have any milestones, especially for
the midterm evaluation? What will be the status at the end of GSoC this
summer? Is your plan to have it fully deployed already?

How will the bot work in detail? I guess listening to the webhooks?
Where will it get its database for the maintainers? Or do you intend to
parse the submitted/modified Portfile?

What will the scripts for Travis CI be written in? Can we reuse code
from the buildbot infrastructure, namely mpbb [1] which is written in
bash shell script? What are steps you will take to implement this
functionality?

Rainer

[1] https://github.com/macports/mpbb

Re: GSoC Proposal

[Jackson Isaac]

Hi,

On Fri, 31 Mar 2017 at 9:14 PM, Zero King  wrote:

Hi,

I'd like to implement a bot in Go dealing with pull requests for
macports-ports on GitHub and utilize Travis CI to test PRs and commits
in GSoC 2017.

More details in my draft:
https://gist.github.com/l2dy/420533b821570e26dc7374898c3264fb

Any advice is welcome and since this is a new idea I'm seeking for
potential mentors.

--
Best regards,
Zero King


Please upload the draft on gsoc portal so that we can review and add
comments if required.

--
Jackson Isaac