Re: GSoC 2019 [Phase out dependency on Xcode]
Hi Mojca, Clemens, and everyone. Thank you for the feedback and comments, I've incorporated the suggestions and submitted my final proposal. On Tue, Apr 9, 2019 at 3:50 AM Clemens Lang wrote: > I've left a couple of comments in your Google Doc, but overall yours is > a solid proposal. Sorry for only picking up on your thread now, but > we've been somewhat overwhelmed by GSoC proposals this year. I understand that you're being overloaded right now. I've read up on last year's mailing list about GSOC and it looks like you're receiving more than 5x (?) of last year's traffic. > don't have time to review the PR right now, but please feel free to keep > pestering me about it weekly or biweekly until I get to it. Will do. On Mon, Apr 8, 2019 at 6:50 PM Mojca Miklavec wrote: > Amazing new feature, thank you! You're welcome, I'm glad to help. > Cool, thank you very much. > I then realised that it wasn't asciidoctor we were missing as a > package, but docbookrx (which we will need for conversion of our guide > which is currently in docbook). > Sorry for misguidance, but thank you for the update in any case. I see now, I thought that I had misread your email haha. Good luck for everyone involved.
Re: GSoC 2019 [Phase out dependency on Xcode]
Hi Satryaji, On Mon, Apr 08, 2019 at 10:44:43AM +0700, Satryaji Aulia wrote: > Hi Clemens, I'd also like your feedback on my proposal, particularly > about Xcode dependency. What do you think? I've left a couple of comments in your Google Doc, but overall yours is a solid proposal. Sorry for only picking up on your thread now, but we've been somewhat overwhelmed by GSoC proposals this year. I didn't comment on it in the proposal document, but I like your work on the bump tool. Haven't looked at the code, but the screen recording looks like a very helpful tool. We've actually wanted something like this for a long time, but haven't found the time to implement it. I don't have time to review the PR right now, but please feel free to keep pestering me about it weekly or biweekly until I get to it. > Also thanks for the 1 hour tutorial on macports-base. It helped a lot > with understanding the MacPorts system, not just me but other > prospective GSOC students as I read on the mailing list. Any time, glad this MacPorts Meeting recording has been so helpful to people over the years :) -- Clemens
Re: GSoC 2019 [Phase out dependency on Xcode]
On Sun, 7 Apr 2019 at 16:36, Satryaji Aulia wrote: > > Hi Marcus and Mojca. > > I’ve fleshed out my proposal (also my understanding of MacOS compilers) > again. To disallow Xcode compilers, I added my idea to use a compiler > blacklist which implementation already exists. I’ve also worked a lot on my > macports-base PR and am confident that I can comfortably code in Tcl. > > I marked the PR ready for review as it is now already functional. > Demo here: https://asciinema.org/a/1X9VFUxRXncOE1m7PhB9DVKcX Amazing new feature, thank you! > Any feedback on my proposal would be greatly appreciated. Thank you. Please note that I have absolutely no idea how difficult it is to implement these features (I'm not familiar with the base too much, so it makes sense to listen to Marcus or Clemens or someone else from the core team, don't take me as an authority in this area). Here are some of my random ideas that could be either parts of proposal or stretch goals: (a) https://trac.macports.org/ticket/52575 (report xcode version in main.log; should be simple to do) (b) https://trac.macports.org/ticket/15712 (declarative syntax to specify incompatible macOS versions) (c) https://trac.macports.org/ticket/56093 (simplify blacklisting compilers) (d) (There are probably a couple other things that could be done with trace mode.) (e) Probably more of a stretch goal anyway: It could be nice to support "submit pull request" as the next step of "port bump". Maybe from web interface, once we get there :) >> Ruby is still an issue, whether or not that's urgent is a matter of >> taste. Die-hard rubyist would probably turn to the other package >> manager :). We are currently also not packaging asciidoctor, for >> example (which would be nice to have for our documentation / guide). > > Of course, sorry to assume it’s not urgent. I'm not saying it is. I just said that it's hard to say :) > Also, I used port bump on > asciidoctor to test it out and submitted a PR. Cool, thank you very much. I then realised that it wasn't asciidoctor we were missing as a package, but docbookrx (which we will need for conversion of our guide which is currently in docbook). Sorry for misguidance, but thank you for the update in any case. Mojca
Re: GSoC 2019 [Phase out dependency on Xcode]
Hi Clemens, I'd also like your feedback on my proposal, particularly about Xcode dependency. What do you think? Also thanks for the 1 hour tutorial on macports-base. It helped a lot with understanding the MacPorts system, not just me but other prospective GSOC students as I read on the mailing list. On Sun, Apr 7, 2019 at 9:36 PM Satryaji Aulia wrote: > > Hi Marcus and Mojca. > > I’ve fleshed out my proposal (also my understanding of MacOS compilers) > again. To disallow Xcode compilers, I added my idea to use a compiler > blacklist which implementation already exists. I’ve also worked a lot on my > macports-base PR and am confident that I can comfortably code in Tcl. > > I marked the PR ready for review as it is now already functional. > Demo here: https://asciinema.org/a/1X9VFUxRXncOE1m7PhB9DVKcX > > Any feedback on my proposal would be greatly appreciated. Thank you.
Re: GSoC 2019 [Phase out dependency on Xcode]
Hi Marcus and Mojca.I’ve fleshed out my proposal (also my understanding of MacOS compilers)again. To disallow Xcode compilers, I added my idea to use a compilerblacklist which implementation already exists. I’ve also worked a lot on mymacports-base PR and am confident that I can comfortably code in Tcl.I marked the PR ready for review as it is now already functional.Demo here: https://asciinema.org/a/1X9VFUxRXncOE1m7PhB9DVKcXAny feedback on my proposal would be greatly appreciated. Thank you.On 2 Apr 2019, at 03.45, Mojca Miklavec wrote:Ruby is still an issue, whether or not that's urgent is a matter oftaste. Die-hard rubyist would probably turn to the other packagemanager :). We are currently also not packaging asciidoctor, forexample (which would be nice to have for our documentation / guide).Of course, sorry to assume it’s not urgent. Also, I used port bump onasciidoctor to test it out and submitted a PR.Also added the fact that there's already a proposal for that problem area.Project may complement each other, and students are welcome to helpeach other, in particular if one is stronger in a particular field. Ifyou are an eager rubyist, even if the upt project gets selected andthe ruby import gets implemented, the expertise and/or passion toimprove this area would still help.Alright, got it. Thank you for your help.
Re: GSoC 2019 [Phase out dependency on Xcode]
Dear Satryaji, On Mon, 1 Apr 2019 at 21:29, Satryaji Aulia wrote: > > Hi again Mojca and Marcus. > > >> I notice how useful a "port bump" command would be if it existed. > > > > Yes, it would be. > > I've started my implementation, under macports-base/src/port1.0/portbump.tcl. Cool! > Besides that, any feedback about my implementation would be appreciated. I > currently use Tcl’s > reinplace method to update the Portfile [2]. Even if that's still work in progress, may I suggest you to open a pull request against macports base? If the code is not yet ready, mark it as such, but it will be easier to provide feedback line-by-line, and faster to merge it. > > For most of the software we just ship one single version, the latest > > one. Of course there are exceptions, usually in cases when: > > - sufficient backward incompatibility is of concern (ruby on rails is > > an excellent example; not that we have any support worth mentioning > > for that one) > > - dependent software is not yet compatible with the latest version > > - the latest version doesn't work on older systems (and we care enough > > to support the old systems) > > I see. Then the php5 and ruby problems I encountered fall into the exception > then, right? Yes, we currently provide multiple versions for both php and ruby. As far as php is concerned: yes, we do ship old unsupported versions, but we also ship the latest one(s). Try "port info php". Ruby on the other hand is a total mess. While we do ship the latest version(s) of ruby, the packages for ruby are probably so outdated that they are literally useless. > It seems > that these types of cases are not urgent, so I've discarded the sub-project > idea of updating > retired ruby/php5 ports. Ruby is still an issue, whether or not that's urgent is a matter of taste. Die-hard rubyist would probably turn to the other package manager :). We are currently also not packaging asciidoctor, for example (which would be nice to have for our documentation / guide). > Also added the fact that there's already a proposal for that problem area. Project may complement each other, and students are welcome to help each other, in particular if one is stronger in a particular field. If you are an eager rubyist, even if the upt project gets selected and the ruby import gets implemented, the expertise and/or passion to improve this area would still help. Mojca
Re: GSoC 2019 [Phase out dependency on Xcode]
Hi again Mojca and Marcus.
> On 1 Apr 2019, at 17.15, Mojca Miklavec wrote:
>
> I'm not in position to answer this, but you may check some discussion
> on this mailing list (from March / GSOC) regarding trace mode, in case
> it's relevant or provides you some more insight.
I have read more about trace mode and updated my proporal, including an
analysis of porttrace.tcl
and what could be done to modify it. I think the detection feature could be
done over the course
of summer, including implementing the `port bump` action + continually making
Port updates.
Feedback regarding the proposal/technical explanations would be greatly
appreciated.
At first, I was under the impression that Xcode command line tools was a subset
of Xcode.app.
Apparently not, as both have different sets of compilers. Do you think it is
realistic if we just just
implemented the "use_xcode" flag and made the command line tools mandatory?
Added the fact that MacPorts support for systems fully without Xcode seems to
be a far goal.
>> I notice how useful a "port bump" command would be if it existed.
>
> Yes, it would be.
I've started my implementation, under macports-base/src/port1.0/portbump.tcl.
The wiki and the 1-hour tutorial by cal has been a tremendous help in
understanding macports-base.
My current implementation is a "bump" stage which requires the "main" and
"fetch" stages to be
sucessfully run first. portbump.tcl largely reuses code from portchecksums.tcl.
After updating the version of an outdated Portfile, I already have it running
like:
$ sudo /opt/macports-test/bin/port bump
---> Fetching distfiles for bibledit
---> Bumping checksums for bibledit
We will bump these:
Old md5 : e509449e52142757c2c75af124847941
New md5 : c06483349e496501248f5a4dc9193184
Old rmd160 : 02e628f018d075cc72ff5c2bf8fb0989e2dc63cc
New rmd160 : f4c103e24b313744633a20e55365fdd126ac980d
I need help in issuing an interactive prompt using
`macports::ui_options(questions_yesno)` [1].
I don't understand how to make `if {[info exists
macports::ui_options(questions_yesno)]}` default
to true.
Besides that, any feedback about my implementation would be appreciated. I
currently use Tcl’s
reinplace method to update the Portfile [2].
>> I will work on this feature right now. Hopefully I can make a functioning
>> tool by the end of the week. I will add the extended implementation to my
>> proposal.
>
> Please also read the discussion (and code etc.) on this mailing list
> related to "upt" as that's quite a bit related (not the same though).
Will do.
> For most of the software we just ship one single version, the latest
> one. Of course there are exceptions, usually in cases when:
> - sufficient backward incompatibility is of concern (ruby on rails is
> an excellent example; not that we have any support worth mentioning
> for that one)
> - dependent software is not yet compatible with the latest version
> - the latest version doesn't work on older systems (and we care enough
> to support the old systems)
I see. Then the php5 and ruby problems I encountered fall into the exception
then, right? It seems
that these types of cases are not urgent, so I've discarded the sub-project
idea of updating
retired ruby/php5 ports. Also added the fact that there's already a proposal
for that problem area.
Thank you.
[1]
https://github.com/satraul/macports-base/commit/d5e652c0de77e30afa64b6a064ebfcf373f04c5f
[2]
https://github.com/satraul/macports-base/commit/0e9c64a1de42e2f23da9eefa6dbb28b88e90a98f
Re: GSoC 2019 [Phase out dependency on Xcode]
Dear Satryaji, On Sun, 31 Mar 2019 at 21:37, Satryaji Aulia wrote: > > - Xcode detection > > The detailed explanation is helpful and it clarifies a lot. So to me it seems > like the core solution of the problem is simple but the extended > functionality (trace mode) is a bit more complex. I'll research more about > trace mode and get back ASAP. If it seems difficult, I'll put Xcode detection > for stretch goals and assign my schedule with more doable tasks. Would you > advise doing that, or do you think it's mandatory for this project idea? I'm not in position to answer this, but you may check some discussion on this mailing list (from March / GSOC) regarding trace mode, in case it's relevant or provides you some more insight. > - The "port bump" tool > > I did `port livecheck maintainer:nomaintainer` for several minutes and > updated some Ports [1][2], one which I have used before (chromedriver) [3]. Thank you very much. > I notice how useful a "port bump" command would be if it existed. Yes, it would be. > I will work on this feature right now. Hopefully I can make a functioning > tool by the end of the week. I will add the extended implementation to my > proposal. Please also read the discussion (and code etc.) on this mailing list related to "upt" as that's quite a bit related (not the same though). > - Ports which depend on obsolete Ports If a port depends on an obsolete port, it makes sense to figure out if we can make it depend on the latest version (of that particular dependency). > A Port that need updating [4] depend on php5-web, but still depends on php5 > (obsolete, successed by php53). The non-destructive solution would be to > replace php5-web with a new app called php53-web I guess. > > Also, the ruby port is Ruby 1.8.7, which has been long retired since 2013 and > all rb-* gems depend on it. > > It seems that people still use this version and some of the gems could be > updated [5]. > What's the stance about supporting retired versions? For most of the software we just ship one single version, the latest one. Of course there are exceptions, usually in cases when: - sufficient backward incompatibility is of concern (ruby on rails is an excellent example; not that we have any support worth mentioning for that one) - dependent software is not yet compatible with the latest version - the latest version doesn't work on older systems (and we care enough to support the old systems) We have no general guidelines, it's often up to maintainer, often not exactly clear, and often we have different opinions. Some developers are in favour of dropping support for python 3.6 packages, others would keep ancient software that's no longer maintained. Generally we are way more relaxed that the other package manager which threw away tons of packages. On one way that's positive as you might be able to get software that's not even possible to download upstream any longer, on the other hand that means that we still provide a large number of broken ports with zero maintenance, possibly scaring away users who give up with macports after being scared off with unsuccessful attempt to install such a package. I have absolutely no idea about PHP, but our ruby support is close-to-useless. The (latest) ruby port itself is probably OK, but all other packages would need a lot of loving care from a volunteer. If you read the mailing list, there's one proposal to somewhat help automate importing ruby packages into MacPorts, but after the software is written, there's still quite some work to actually prepare packages that matter and test them. Any improvement of ruby situation would be warmly welcome. Mojca
Re: GSoC 2019 [Phase out dependency on Xcode]
Thank you Mojca and Marcus for the response. It's totally okay to risk scaring me off! I'd ultimately like to contribute so I like the honest feedback haha. I'd like to report/ask about several things: - Xcode detection The detailed explanation is helpful and it clarifies a lot. So to me it seems like the core solution of the problem is simple but the extended functionality (trace mode) is a bit more complex. I'll research more about trace mode and get back ASAP. If it seems difficult, I'll put Xcode detection for stretch goals and assign my schedule with more doable tasks. Would you advise doing that, or do you think it's mandatory for this project idea? - The "port bump" tool I did `port livecheck maintainer:nomaintainer` for several minutes and updated some Ports [1][2], one which I have used before (chromedriver) [3]. I notice how useful a "port bump" command would be if it existed. I will work on this feature right now. Hopefully I can make a functioning tool by the end of the week. I will add the extended implementation to my proposal. - Ports which depend on obsolete Ports A Port that need updating [4] depend on php5-web, but still depends on php5 (obsolete, successed by php53). The non-destructive solution would be to replace php5-web with a new app called php53-web I guess. Also, the ruby port is Ruby 1.8.7, which has been long retired since 2013 and all rb-* gems depend on it. It seems that people still use this version and some of the gems could be updated [5]. What's the stance about supporting retired versions? I will get back with an updated proposal soon, which will be better. Thank you for reading. [1] https://github.com/macports/macports-ports/pull/3978 [2] https://github.com/macports/macports-ports/pull/3977 [3] https://github.com/macports/macports-ports/pull/3974 [4] https://trac.macports.org/ticket/58173 [5] https://trac.macports.org/ticket/55974 > On 31 Mar 2019, at 03.55, Marcus Calhoun-Lopez wrote: > > Greetings. > > Forgive me if this is obvious to you, but just to make sure we are on the > same page, I will go into a little more detail than you probably want. > The first thing to understand is that adding support for `use_xcode` (and/or > `use_command_line_tools`) is the *first step*. > This should be fairly easy. > After that, you would modify trace mode to assist maintainers in figuring out > if changing `use_xcode` is *necessary*. > Trace mode is the part of the base code I am least familiar with, but I > believe this is a reasonable goal. > > Currently, the official policy is that MacPorts requires *both* Xcode and the > command line tools need to be installed [1]. > The `port` command issues warnings if either one is not installed [2]. > That being said, several (many?, most?) ports build fine with just the > command line tools. > There are at least a few ports that would build just fine with just Xcode > (e.g. port that build with qmake [3]). > In a Portfile, if `use_xcode` were set to `no`, MacPorts would have to > attempt to build the port with just the command line tools. > Just to be clear, `use_xcode` does not exist yet. > MacPorts does have *some* support for systems without Xcode, but the code is > new, untested, and has not even made it into a released version of MacPorts > [4]. > > After `use_xcode` is implemented, the next step would be to help maintainers > determine if they should change it in a Portfile. > With my somewhat limited understating of trace mode, you might want to start > with porttrace.tcl [5]. > For example, if `use_xcode` is `no`, then some of the directories being added > to the sandbox no longer make sense. > > Here is the flow we have now: > A user installs a port. > If Xcode is not installed, a warning is issued, and maybe the port installs > and maybe it doesn’t. > > Here is the flow we would like: > A user installs a port. > If, in the Portfile, it is explicitly stated that Xcode is not needed, then > Xcode is not used, regardless of whether it is installed or not [6]. > The port builds successfully. > To facilitate the flow we would like, trace mode attempts to determine if > Xcode is being accessed and used, which would be strong indicator that Xcode > is needed. > > If I have not satisfactorily answered your question, please feel free to ask > for clarification. > > -Marcus > > [1] https://www.macports.org/install.php > [2] > https://github.com/macports/macports-base/blob/45e62d7e9e7679a43075d1912c8d4a06c2abc6fe/src/port1.0/portutil.tcl#L3296 > [3] https://doc.qt.io/qt-5/qmake-manual.html > [4] > https://github.com/macports/macports-base/commit/76a804cc360924927fa8e2dfa41c761a19d17f3b#diff-98449d939df165f9b8cc4d1aab90ed2c > [5] > https://github.com/macports/macports-base/blob/master/src/port1.0/porttrace.tcl > [6] https://trac.macports.org/wiki/ReproducibleBuilds
Re: GSoC 2019 [Phase out dependency on Xcode]
Greetings. Forgive me if this is obvious to you, but just to make sure we are on the same page, I will go into a little more detail than you probably want. The first thing to understand is that adding support for `use_xcode` (and/or `use_command_line_tools`) is the *first step*. This should be fairly easy. After that, you would modify trace mode to assist maintainers in figuring out if changing `use_xcode` is *necessary*. Trace mode is the part of the base code I am least familiar with, but I believe this is a reasonable goal. Currently, the official policy is that MacPorts requires *both* Xcode and the command line tools need to be installed [1]. The `port` command issues warnings if either one is not installed [2]. That being said, several (many?, most?) ports build fine with just the command line tools. There are at least a few ports that would build just fine with just Xcode (e.g. port that build with qmake [3]). In a Portfile, if `use_xcode` were set to `no`, MacPorts would have to attempt to build the port with just the command line tools. Just to be clear, `use_xcode` does not exist yet. MacPorts does have *some* support for systems without Xcode, but the code is new, untested, and has not even made it into a released version of MacPorts [4]. After `use_xcode` is implemented, the next step would be to help maintainers determine if they should change it in a Portfile. With my somewhat limited understating of trace mode, you might want to start with porttrace.tcl [5]. For example, if `use_xcode` is `no`, then some of the directories being added to the sandbox no longer make sense. Here is the flow we have now: A user installs a port. If Xcode is not installed, a warning is issued, and maybe the port installs and maybe it doesn’t. Here is the flow we would like: A user installs a port. If, in the Portfile, it is explicitly stated that Xcode is not needed, then Xcode is not used, regardless of whether it is installed or not [6]. The port builds successfully. To facilitate the flow we would like, trace mode attempts to determine if Xcode is being accessed and used, which would be strong indicator that Xcode is needed. If I have not satisfactorily answered your question, please feel free to ask for clarification. -Marcus [1] https://www.macports.org/install.php [2] https://github.com/macports/macports-base/blob/45e62d7e9e7679a43075d1912c8d4a06c2abc6fe/src/port1.0/portutil.tcl#L3296 [3] https://doc.qt.io/qt-5/qmake-manual.html [4] https://github.com/macports/macports-base/commit/76a804cc360924927fa8e2dfa41c761a19d17f3b#diff-98449d939df165f9b8cc4d1aab90ed2c [5] https://github.com/macports/macports-base/blob/master/src/port1.0/porttrace.tcl [6] https://trac.macports.org/wiki/ReproducibleBuilds > On Mar 29, 2019, at 1:35 AM, Satryaji Aulia wrote: > > Hi Marcus, I’m a final-year student from University of Indonesia interested > in contributing to MacPort, and I’m working on my proposal right now. > > I’d like to ask about the Phase out dependency on Xcode project idea on the > Wiki page. > > Just making sure of the flow: user/maintainer installs a Port using trace > mode (-t), if it turns out it the Port needs the full Xcode package, it > outputs: > - An error message if full Xcode is not installed. > - A warning message if use_xcode yes in the Portfile is not set. > Thus, all the contributions are to be made on the macports-base repo. Correct? > > In achieving this, exactly which parts of the codebase need to be modified > and how do you detect that a package needs full Xcode? > > I understand that trace mode reads filenames that are accessed using > DYLD_INSERT_LIBRARIES. Do we compare these filenames to a list of filenames > that are from the full Xcode? > > Looking forward to your reply. > satraul
Re: GSoC 2019 [Phase out dependency on Xcode]
Dear Satryaji, On Fri, 29 Mar 2019 at 09:35, Satryaji Aulia wrote: > > Hi Marcus, I’m a final-year student from University of Indonesia interested > in contributing to MacPort, and I’m working on my proposal right now. > > I’d like to ask about the Phase out dependency on Xcode project idea on the > Wiki page. > > Just making sure of the flow: user/maintainer installs a Port using trace > mode (-t), if it turns out it the Port needs the full Xcode package, it > outputs: > - An error message if full Xcode is not installed. > - A warning message if use_xcode yes in the Portfile is not set. > Thus, all the contributions are to be made on the macports-base repo. Correct? > > In achieving this, exactly which parts of the codebase need to be modified > and how do you detect that a package needs full Xcode? > > I understand that trace mode reads filenames that are accessed using > DYLD_INSERT_LIBRARIES. Do we compare these filenames to a list of filenames > that are from the full Xcode? I would love to be able to respond to your enquiry, but I'm not so familiar with the part of the code that handles this to be able to be of any help as far as the contents go. I hope that Marcus (or Clemens or anyone else from the core group) reply as soon as possible. Some general feedback though: - "Demonstrate ability to MacPorts mentor if needed": this needs to be done *BEFORE* the selection process rather than until May. That basically means until the 9th of April + maybe a short buffer beyond (until we cast the vote and request the desired number of slots to Google). It would be awesome if you could (with help of others, of course) try to create some pull requests to demonstrate your capability of contributing and solving problems, even if the contributions start from very small ones. Proving to us that you could make some awesome contribution during the summer (before we select students) is probably the only way to get accepted. - "Consult with mentors to identify which parts of the codebase need to be modified": this needs to be done before the end of application period (9th of April). You need to understand the problem and have an idea of what needs to be done before you can make a good proposal. - "Open a merge request and review with mentors": this should ideally be done on regular basis (ideally every few days) and by the time of *FIRST* evaluation you should ideally already have some functional improvements merged to the codebase. Planning to have the first code merged only after two months (+ all of the usual delays that you forgot to count in) is probably way too late into the process. Mojca (It wasn't my intention to scare you off, and I hope I didn't :) :) :)
