Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Apr 3, 2018, at 21:27, Andrew Moore wrote:

> With Apple now planning to phase out Server.app, might they be open to 
> supporting MacPorts once again?  Going forward, I would hope they recognize a 
> vested interest in seeing that the open-source community continues to support 
> them…

As you know, Apple divested themselves of their responsibility for MacPorts, by 
hiring me for one year to be the macOS forge administrator and to plan the 
transition away from Apple infrastructure, which was completed in November 
2016. Apple has not contacted us since then to indicate any desire to 
re-involve themselves. Individual Apple employees continue to contribute to 
MacPorts development as their time and interests permit. It is possible that 
Apple or other companies who value MacPorts might want to contribute 
financially, but MacPorts is not currently set up as a legal entity, so we do 
not currently accept financial donations.

Re: Changing default cxx_stdlib to libc++

[Andrew Moore]

With Apple now planning to phase out Server.app, might they be open to 
supporting MacPorts once again?  Going forward, I would hope they recognize a 
vested interest in seeing that the open-source community continues to support 
them…
-AM

> On Mar 15, 2018, at 12:19 AM, Ryan Schmidt  wrote:
> 
> 
> On Mar 14, 2018, at 07:23, db wrote:
>> On 14 Mar 2018, at 01:14, Rainer Müller wrote:
>>> Are you going to sponsor a dedicated Mac server for GitLab CI?
>>> Travis CI is available at no cost and we have no funds to pay for anything.
>> 
>> If MacPorts is short on resources, these could be listed on the website. 
>> Folks can certainly ask around.
> 
> We did have an offer on the list some months ago from someone with spare Mac 
> hardware in a data center that they would let us use. We did not respond to 
> the offer. I'm not sure how comfortable I am accepting unsolicited 
> infrastructure offers from unfamiliar parties.
> 

Re: Changing default cxx_stdlib to libc++

[db]

On 15 Mar 2018, at 05:19, Ryan Schmidt  wrote:
> On Mar 14, 2018, at 07:23, db wrote:
>> On 14 Mar 2018, at 01:14, Rainer Müller wrote:
>>> Are you going to sponsor a dedicated Mac server for GitLab CI?
>>> Travis CI is available at no cost and we have no funds to pay for anything.
>> If MacPorts is short on resources, these could be listed on the website. 
>> Folks can certainly ask around.
> We did have an offer on the list some months ago from someone with spare Mac 
> hardware in a data center that they would let us use. We did not respond to 
> the offer. I'm not sure how comfortable I am accepting unsolicited 
> infrastructure offers from unfamiliar parties.

Not responding to an offer doesn't encourage people to ask around for or 
provide resources. Definitively not me. You might also need to describe how a 
gift horse has to look like, i.e., specify requirements.

Re: CI system for PR builds (was: Re: Changing default cxx_stdlib to libc++)

[db]

On 15 Mar 2018, at 05:13, Ryan Schmidt  wrote:
> Because PRs come from untrusted sources, we have to assume their contents are 
> tainted. So after any PR is finished building, the VM is tainted and we have 
> to throw it away and make a new one from our template for the next PR build.

> On Mar 14, 2018, at 07:25, db wrote:
>> Otherwise, you could make the machines sync to the packages public server 
>> for the distributable, and to a private server for the non-distributable 
>> binaries.
> I can't find an interpretation of that sentence that helps to solve the 
> prepopulation problem.

I didn't know how you handled the templating. Couldn't you just prepopulate the 
cloned VM, take a snapshot, build the PR, restore the snapshot, eventually, 
delete the snapshot, update outdated, then retake it?

Re: Changing default cxx_stdlib to libc++

[db]

On 14 Mar 2018, at 01:14, Rainer Müller  wrote:
> Are you going to sponsor a dedicated Mac server for GitLab CI?
> Travis CI is available at no cost and we have no funds to pay for anything.

If MacPorts is short on resources, these could be listed on the website. Folks 
can certainly ask around.

Re: CI system for PR builds (was: Re: Changing default cxx_stdlib to libc++)

[Ryan Schmidt]

On Mar 13, 2018, at 19:14, Rainer Müller wrote:
> On 2018-03-14 00:42, db wrote:
>> On 14 Mar 2018, at 00:22, Mojca Miklavec wrote:
>>> Because someone would need to write the code for an alternative CI
>> 
>> Wouldn't self-hosted GitLab CI be good enough?
> 
> Are you going to sponsor a dedicated Mac server for GitLab CI?
> Travis CI is available at no cost and we have no funds to pay for anything.

I was not aware of the existence of GitLab CI, but I haven't done a survey of 
CI systems, mainly because we already selected one many years ago: Buildbot. 
The problem, to my mind, was not that of selecting which CI system to use, but 
the fact that we want to have a virtual machine template that the CI system 
should clone, boot up, run a job in, and then delete, or even multiple 
templates, one for each OS version we want to test. I see now that GitLab CI 
has support for doing that with both VirtualBox and Parallels.

https://docs.gitlab.com/runner/executors/README.html

That's good, and I didn't know that it had that feature. The Xserves running 
the Buildbot might have room for a couple additional virtual machines, 
especially after the duplicate 10.6/10.7/10.8 VMs are removed after we switch 
those users from libstdc++ to libc++. But out Buildbot VMs run on VMware ESXi, 
so if I were to add builds for PRs to the system, I would of course want to 
continue using VMware. It might be possible to add support for VMware to GitLab 
CI, but it is probably just as possible to add it to our Buildbot 
configuration, and personally I would rather do that and continue to work with 
the CI system I'm already familiar with than learn a whole new one, and have to 
remain proficient with both. Adding this to our Buildbot config doesn't seem 
outside the realm of possibility, but beyond being aware that VMware ESXi has a 
command line interface by which VMs can be cloned, started, stopped, and 
deleted, I haven't looked into it.

We also have the problem of what exactly to put on the VM templates, and how to 
keep them updated. Certainly, a VM template would need to contain an 
installation of macOS, and Xcode, and MacPorts, and Java, and the VMware tools, 
and the necessary CI software such as Buildbot worker. But one of the reasons 
why our Buildbot setup is as fast as it is is that each Buildbot worker keeps 
previously-built ports installed (but inactive). So when someone commits a 
change to the demeter port, for example, the Buildbot does not have to first 
build all of demeter's 430 recursive dependencies; it just has to activate 
them. If we instead clone a VM template that doesn't have any ports installed 
yet, it will take a lot longer to first install all those dependencies. Best 
case, all the dependencies are distributable, were already built on the 
post-commit Buildbot workers and uploaded to the packages server, and so they 
just need to be downloaded and extracted. Worst case, they aren't distributable 
and have to be built from source.

An idea that's been suggested before, which sounds good to me but which has not 
been implemented yet, is that in addition to uploading distributable archives 
to the public packages server like we already do, we should upload any 
non-distributable archives to a private server that only Travis or this new 
hypothetical PR build system can access.

We could pre-populate the VM templates with lots of installed but inactive 
ports, but in addition to making the templates and their clone(s) take up more 
disk space, which is at a premium, the installed ports would quickly become 
outdated as port updates are committed. We would need a mechanism for updating 
the ports installed on the templates fairly frequently. I like the private 
packages server idea better.

Even if we don't want to keep any ports installed on the VM templates, we still 
need to update them from time to time, for example to update the OS, Xcode, 
Java, etc. With the post-commit Buildbot workers, I do that manually when I 
notice an update is needed and the Buildbot is otherwise idle, and that has 
been sufficient so far.

Re: Changing default cxx_stdlib to libc++

[Rainer Müller]

On 2018-03-14 00:42, db wrote:
> On 14 Mar 2018, at 00:22, Mojca Miklavec  wrote:
>> Because someone would need to write the code for an alternative CI
> 
> Wouldn't self-hosted GitLab CI be good enough?

Are you going to sponsor a dedicated Mac server for GitLab CI?
Travis CI is available at no cost and we have no funds to pay for anything.

Rainer

Re: Changing default cxx_stdlib to libc++

[db]

On 14 Mar 2018, at 00:22, Mojca Miklavec  wrote:
> Because someone would need to write the code for an alternative CI

Wouldn't self-hosted GitLab CI be good enough?
I get the impression that testing across the board is not a priority.

Re: Changing default cxx_stdlib to libc++

[Mojca Miklavec]

On 14 March 2018 at 00:12, db  wrote:
> On 13 Mar 2018, at 17:10, Mojca Miklavec  
> wrote:
>> The main problem is that Travis is *not* out main build system, we use
>> a different system for that. Travis has lots of limitations:
>
> Got it. Then why use it in the first place instead of an alternative CI?

Because someone would need to write the code for an alternative CI and
either have enough hardware for builds available or find one that
doesn't have limitations of Travis, none of which is trivial.

Mojca

Re: Changing default cxx_stdlib to libc++

[db]

On 13 Mar 2018, at 17:10, Mojca Miklavec  wrote:
> The main problem is that Travis is *not* out main build system, we use
> a different system for that. Travis has lots of limitations:

Got it. Then why use it in the first place instead of an alternative CI?

Re: Changing default cxx_stdlib to libc++

[db]

On 13 Mar 2018, at 15:22, Ryan Schmidt  wrote:
> On Mar 13, 2018, at 08:47, db wrote:
>> On 12 Mar 2018, at 21:35, Ryan Schmidt wrote:
>>> On Mar 12, 2018, at 13:56, db wrote:
 If not, shouldn't it, prior to accepting an updated port definition?
>>> The buildbot is only engaged after a commit is in the repository master 
>>> branch.
>>> We have a separate system, using Travis CI, for doing test builds of 
>>> updates that have been submitted as pull requests but have not yet been 
>>> merged into the repository master branch. But most maintainers that are 
>>> committers don't submit pull requests for their own software, they just 
>>> commit directly to master, so their changes would not go through Travis.
>> Then why contribute an update to a port circumventing the test build system? 
>> What's the sense in that?
> You're the first person, that I'm aware of, to suggest that we should 
> restrict commits to master and require all contributors, including those who 
> have already been vetted and given commit access, to make their contributions 
> through pull requests. I'm sure there are other projects that have made such 
> policies, but we haven't so far. Instead, we allow direct commits to master, 
> and we encourage others to review those commits on the macports-dev mailing 
> list or on GitHub.
> 
> We've only been on GitHub since late 2016. For the 10 years prior to that, we 
> used a Subversion repository, and prior to that it was CVS. The only way to 
> update those repositories was for a someone with commit access to commit to 
> them. Many of us who were MacPorts committers during that time are used to 
> this way of working and might find the additional roadblocks to our 
> contributions that pull requests represent to be intrusive. In my case, I 
> feel like I might finally be becoming proficient at submitting pull requests 
> at least, but it has been a long and frustrating learning process for me, and 
> one which is far from over; I'm still making git mistakes daily.
> 
> We've only had our Travis CI integration set up since mid-2017. I am not 
> certain what its reliability is now. Often, when I've looked at a Travis 
> failure log, it has turned out to be a problem specific to Travis, and not a 
> reason to reject a PR. So I am not paying a great amount of attention to what 
> Travis is doing at this time. It's nice that we have it, it's nice that our 
> other developers like to use it to check their work before publishing it. But 
> developers are already expected to check their work and verify the port 
> installs on their own machines before committing or submitting a PR or patch; 
> if they do that, they shouldn't see much on Travis that they didn't already 
> know about.

Thanks for the background. I don't know how much inconvenience would that mean 
for developers, I do know as a user though. I suppose that problems would be 
properly caught in the pipeline and almost never make it to the user. Skipping 
it because of being inconvenienced, that doesn't make any sense whatsoever to 
me.

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 13, 2018, at 08:45, db wrote:
> On 12 Mar 2018, at 21:56, Ryan Schmidt wrote:
>> I'm not aware of any plan to notify users of the cxx_stdlib change, other 
>> than the same way that users are notified of any other port update being 
>> available: by the user running "sudo port selfupdate" and then examining the 
>> output of "port outdated".
> 
> I was considering only subscribing to macports-announce. Would this change 
> make it in there?

Yes, we announce new versions of MacPorts on macports-announce.

>>> Besides rebuilding from source from time to time, I only do port sync.
>> You'll of course have to run "sudo port selfupdate" (or run the installer 
>> downloaded from our web site), not just "sudo port sync", to receive the new 
>> version of MacPorts.
> 
> I compile both base and ports from source, so there's that.

Ok. When the new version of MacPorts is out, you would need to update to it, 
either using "sudo port selfupdate", or using the installer from our web site, 
or by updating your copy of the base source and recompiling it.

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 13, 2018, at 08:47, db wrote:
> On 12 Mar 2018, at 21:35, Ryan Schmidt wrote:
>> On Mar 12, 2018, at 13:56, db wrote:
>>> If not, shouldn't it, prior to accepting an updated port definition?
>> The buildbot is only engaged after a commit is in the repository master 
>> branch.
>> We have a separate system, using Travis CI, for doing test builds of updates 
>> that have been submitted as pull requests but have not yet been merged into 
>> the repository master branch. But most maintainers that are committers don't 
>> submit pull requests for their own software, they just commit directly to 
>> master, so their changes would not go through Travis.
> 
> Then why contribute an update to a port circumventing the test build system? 
> What's the sense in that?

You're the first person, that I'm aware of, to suggest that we should restrict 
commits to master and require all contributors, including those who have 
already been vetted and given commit access, to make their contributions 
through pull requests. I'm sure there are other projects that have made such 
policies, but we haven't so far. Instead, we allow direct commits to master, 
and we encourage others to review those commits on the macports-dev mailing 
list or on GitHub.

We've only been on GitHub since late 2016. For the 10 years prior to that, we 
used a Subversion repository, and prior to that it was CVS. The only way to 
update those repositories was for a someone with commit access to commit to 
them. Many of us who were MacPorts committers during that time are used to this 
way of working and might find the additional roadblocks to our contributions 
that pull requests represent to be intrusive. In my case, I feel like I might 
finally be becoming proficient at submitting pull requests at least, but it has 
been a long and frustrating learning process for me, and one which is far from 
over; I'm still making git mistakes daily.

We've only had our Travis CI integration set up since mid-2017. I am not 
certain what its reliability is now. Often, when I've looked at a Travis 
failure log, it has turned out to be a problem specific to Travis, and not a 
reason to reject a PR. So I am not paying a great amount of attention to what 
Travis is doing at this time. It's nice that we have it, it's nice that our 
other developers like to use it to check their work before publishing it. But 
developers are already expected to check their work and verify the port 
installs on their own machines before committing or submitting a PR or patch; 
if they do that, they shouldn't see much on Travis that they didn't already 
know about.

Re: Changing default cxx_stdlib to libc++

[db]

On 12 Mar 2018, at 21:35, Ryan Schmidt  wrote:
> On Mar 12, 2018, at 13:56, db wrote:
>> If not, shouldn't it, prior to accepting an updated port definition?
> The buildbot is only engaged after a commit is in the repository master 
> branch.
> We have a separate system, using Travis CI, for doing test builds of updates 
> that have been submitted as pull requests but have not yet been merged into 
> the repository master branch. But most maintainers that are committers don't 
> submit pull requests for their own software, they just commit directly to 
> master, so their changes would not go through Travis.

Then why contribute an update to a port circumventing the test build system? 
What's the sense in that?

Re: Changing default cxx_stdlib to libc++

[db]

On 12 Mar 2018, at 21:56, Ryan Schmidt  wrote:
> I'm not aware of any plan to notify users of the cxx_stdlib change, other 
> than the same way that users are notified of any other port update being 
> available: by the user running "sudo port selfupdate" and then examining the 
> output of "port outdated".

I was considering only subscribing to macports-announce. Would this change make 
it in there?

>> Besides rebuilding from source from time to time, I only do port sync.
> You'll of course have to run "sudo port selfupdate" (or run the installer 
> downloaded from our web site), not just "sudo port sync", to receive the new 
> version of MacPorts.

I compile both base and ports from source, so there's that.

Re: Changing default cxx_stdlib to libc++

[Kenneth F. Cunningham]

On 2018-03-12, at 5:21 PM, Michael wrote:


> Is there a reason for a universal binary on PPC anymore?
> 
> Anyone running a PPC binary today will either be on a PPC machine (10.5), or 
> an Intel machine with Rosetta (10.6).
> They either want a 10.5 PPC binary, or a 10.6 Intel binary.
> 
> So is there any reason for a PPC universal build today?
> 

I use the PPC universal binaries to build things on 10.5 Intel that require 
tools that only exist on 10.5 Intel (working newer versions of clang, for 
example, like clang-5.0), and then move them manually over to the 10.5 PPC 
machine and install them there from the binary.

But it's fair to say that's a niche, and the worldwide crowd who wants this 
behaviour might have an n < 5 (me, Fred, the guy who made the patches for 
libsdl2, Ricky Zhang, and maybe Michael Dickens with his new PPC setups he 
mentioned). 

Anyway, it's just one more reason to keep 10.5 and less on libstdc++, which we 
had already decided to do.

Ken

Re: Changing default cxx_stdlib to libc++

[Clemens Lang]

Hi,

On Mon, Mar 12, 2018 at 04:45:39PM -0500, Ryan Schmidt wrote:
> You'll see each PR title has a red "X" or a green "√" next to it.
> That's the indicator of whether Travis built the PR successfully or
> not. You can click them for more information.
> 
> A failed Travis CI build does not necessarily mean there is anything
> wrong with the PR. There are many reasons why a build might fail on
> Travis but succeed on our buildbot and on user systems. For example,
> Travis imposes a time limit for jobs; some port builds exceed that
> time limit.

To add to this, we will soon deploy a change so that our bot will
comment and explain the status of the Travis build, e.g. it will tell
you when the build timed out. It will also post a link to the build log
if a build failed.

Ideally, you do not have to know how the Travis integration works. The
bot will post the relevant information for you.

-- 
Clemens

Re: Changing default cxx_stdlib to libc++

[Michael]

On 2018-03-12, at 4:15 PM, Kenneth F. Cunningham 
 wrote:

> 
> On 2018-03-12, at 4:26 PM, Kenneth F. Cunningham wrote:
>> 
>> 
>> It _could_ be done in 10.5 Intel, but that makes not much sense. Nobody 
>> should be running 10.5 intel anyway, and if we made the libc++ switch in 
>> 10.5 intel we'd be endlessly trying to figure out if we're on 10.5 PPC 
>> (libstdc++) or 10.5 intel (libc++) and -- ugh. 
> 
> Also, I guess universal binaries Intel/PPC on 10.5 would be in a terrible 
> mess if we had Intel on libc++ and PPC on libstdc++.

Is there a reason for a universal binary on PPC anymore?

Anyone running a PPC binary today will either be on a PPC machine (10.5), or an 
Intel machine with Rosetta (10.6).
They either want a 10.5 PPC binary, or a 10.6 Intel binary.

So is there any reason for a PPC universal build today?

---
Entertaining minecraft videos
http://YouTube.com/keybounce

Re: Changing default cxx_stdlib to libc++

[Kenneth F . Cunningham]

On 2018-03-12, at 4:26 PM, Kenneth F. Cunningham wrote:
> 
> 
> It _could_ be done in 10.5 Intel, but that makes not much sense. Nobody 
> should be running 10.5 intel anyway, and if we made the libc++ switch in 10.5 
> intel we'd be endlessly trying to figure out if we're on 10.5 PPC (libstdc++) 
> or 10.5 intel (libc++) and -- ugh. 

Also, I guess universal binaries Intel/PPC on 10.5 would be in a terrible mess 
if we had Intel on libc++ and PPC on libstdc++.

K

Re: Changing default cxx_stdlib to libc++

[Kenneth F. Cunningham]

On 2018-03-12, at 3:42 PM, Mojca Miklavec wrote:

> On 12 March 2018 at 22:32, Michael wrote:
>> On 2018-03-08, at 9:32 AM, Ken Cunningham wrote:
>> 
>>> Thank God.
>>> 
>>> 10.6.8, agree.
>> 
>> So PPC machines will not get this upgrade?
> 
> PPC and libc++ don't particularly like each other.

True. It is quite possible to make libc++ with PPC slices on Intel 10.5, and 
copy that, and a few other bits, over to a PPC machine. It works, for the most 
part.

It is also quite possible to build clang-3.8 for PPC, and it links against this 
libc++ quite effectively.

But there is a problem with handling c++ exceptions with this setup. And no 
thread_local support. And … we don't even know the rest yet.



OTOH, gcc6 builds almost everything, and has no problems with exceptions or 
thread_local storage.  

So gcc6 is presently the path of least resistance for PPC, at least until I 
learn enough about compiler debugging to figure out how to make the problem 
with exceptions work with clang on PPC, or until someone else with the skills 
already finds an interest in this dying hobby platform.

ppc64 has some enthusiasm, as it is current in the linux world. That helps us a 
bit, but not much in this area.


> 
> I don't know if the recenct change also takes gcc's macports-libstdc++
> into account.

Everything about macports-libstdc++ will continue to work exactly as it does 
now, except they won't get the pre made binaries any more, the libc++ people 
will.

And tickets against that setup will most likely not get fixed, I would imagine. 
There are differences in the headers, especially in math and a few other gutty 
parts, that can be a PITA to work through.

WIth libc++, no such issues, as the builds are identical to all the newer 
systems, and it's all been worked out already (in most cases …. ).



> 
> I'm not absolutely sure how the change is supposed to be "deployed".
> The functionality to rev-upgrade should be present everywhere, but
> changing the default to libc++ should not be done on 10.5 and earlier.
> 

It _could_ be done in 10.5 Intel, but that makes not much sense. Nobody should 
be running 10.5 intel anyway, and if we made the libc++ switch in 10.5 intel 
we'd be endlessly trying to figure out if we're on 10.5 PPC (libstdc++) or 10.5 
intel (libc++) and -- ugh. 

So as I see it, the 10.6 cutoff is the right one.

K

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 12, 2018, at 16:42, Michael wrote:

> On 2018-03-12, at 2:33 PM, Ryan Schmidt wrote:
> 
>> 
>> On Mar 12, 2018, at 16:31, Michael wrote:
>>> 
>>> How about documenting the procedure to submit changes by poll requests,
>> 
>> There's lots of documentation about how to submit a pull request, such as:
>> 
>> https://help.github.com/articles/creating-a-pull-request/
>> 
>> We also have:
>> 
>> https://trac.macports.org/wiki/WorkingWithGit#pr
>> 
>>> and then verify the build bot.
>> 
>> What do you mean?
>> 
>>> (It may exist, but I have never seen it.)
> 
> 
> I just took a look at all of these:
> 
> https://help.github.com/articles/creating-a-pull-request/
> https://help.github.com/articles/merging-a-pull-request/
> https://trac.macports.org/wiki/WorkingWithGit#a3.Publishchanges
> 
> None of them mention the buildbot, the Tarvis system, or how to test to see 
> if it actually compiled on the buildbot machine.

If you look at our list of pull requests:

https://github.com/macports/macports-ports/pulls

You'll see each PR title has a red "X" or a green "√" next to it. That's the 
indicator of whether Travis built the PR successfully or not. You can click 
them for more information.

A failed Travis CI build does not necessarily mean there is anything wrong with 
the PR. There are many reasons why a build might fail on Travis but succeed on 
our buildbot and on user systems. For example, Travis imposes a time limit for 
jobs; some port builds exceed that time limit.

Re: Changing default cxx_stdlib to libc++

[Mojca Miklavec]

On 12 March 2018 at 22:32, Michael wrote:
> On 2018-03-08, at 9:32 AM, Ken Cunningham wrote:
>
>> Thank God.
>>
>> 10.6.8, agree.
>
> So PPC machines will not get this upgrade?

PPC and libc++ don't particularly like each other. But we could and
should switch to making gcc 6 or 7 the default compiler there in a
similar way.

I don't know if the recenct change also takes gcc's macports-libstdc++
into account.

I'm not absolutely sure how the change is supposed to be "deployed".
The functionality to rev-upgrade should be present everywhere, but
changing the default to libc++ should not be done on 10.5 and earlier.

Mojca

Re: Changing default cxx_stdlib to libc++

[Michael]

On 2018-03-12, at 2:33 PM, Ryan Schmidt  wrote:

> 
> On Mar 12, 2018, at 16:31, Michael wrote:
>> 
>> How about documenting the procedure to submit changes by poll requests,
> 
> There's lots of documentation about how to submit a pull request, such as:
> 
> https://help.github.com/articles/creating-a-pull-request/
> 
> We also have:
> 
> https://trac.macports.org/wiki/WorkingWithGit#pr
> 
>> and then verify the build bot.
> 
> What do you mean?
> 
>> (It may exist, but I have never seen it.)


I just took a look at all of these:

https://help.github.com/articles/creating-a-pull-request/
https://help.github.com/articles/merging-a-pull-request/
https://trac.macports.org/wiki/WorkingWithGit#a3.Publishchanges

None of them mention the buildbot, the Tarvis system, or how to test to see if 
it actually compiled on the buildbot machine.

---
Entertaining minecraft videos
http://YouTube.com/keybounce

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 12, 2018, at 16:31, Michael wrote:

> On 2018-03-12, at 1:35 PM, Ryan Schmidt wrote:
>> 
>> We have a separate system, using Travis CI, for doing test builds of updates 
>> that have been submitted as pull requests but have not yet been merged into 
>> the repository master branch. But most maintainers that are committers don't 
>> submit pull requests for their own software, they just commit directly to 
>> master, so their changes would not go through Travis.
> 
> How about documenting the procedure to submit changes by poll requests,

There's lots of documentation about how to submit a pull request, such as:

https://help.github.com/articles/creating-a-pull-request/

We also have:

https://trac.macports.org/wiki/WorkingWithGit#pr

> and then verify the build bot.

What do you mean?

> (It may exist, but I have never seen it.)

Re: Changing default cxx_stdlib to libc++

[Michael]

On 2018-03-08, at 9:32 AM, Ken Cunningham  
wrote:

> Thank God. 
> 
> 10.6.8, agree. 

So PPC machines will not get this upgrade?
---
Entertaining minecraft videos
http://YouTube.com/keybounce

Re: Changing default cxx_stdlib to libc++

[Michael]

On 2018-03-12, at 1:35 PM, Ryan Schmidt  wrote:
> 
> We have a separate system, using Travis CI, for doing test builds of updates 
> that have been submitted as pull requests but have not yet been merged into 
> the repository master branch. But most maintainers that are committers don't 
> submit pull requests for their own software, they just commit directly to 
> master, so their changes would not go through Travis.

How about documenting the procedure to submit changes by poll requests, and 
then verify the build bot.
(It may exist, but I have never seen it.)

---
Entertaining minecraft videos
http://YouTube.com/keybounce

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 10, 2018, at 05:38, db wrote:

> How will I know about the lib change and the availability of binaries?

The MacPorts program doesn't have a way to notify you of the availability of 
binaries in advance. You try to install a port, and MacPorts checks for a 
binary then, and if it's available, it downloads it, otherwise it builds from 
source.

I'm not aware of any plan to notify users of the cxx_stdlib change, other than 
the same way that users are notified of any other port update being available: 
by the user running "sudo port selfupdate" and then examining the output of 
"port outdated".

The cxx_stdlib change would be in a new version of MacPorts, and we would of 
course announce the availability of the new version on our web site and on the 
mailing list and probably explain the relevant changes then.

> Besides rebuilding from source from time to time, I only do port sync.

You'll of course have to run "sudo port selfupdate" (or run the installer 
downloaded from our web site), not just "sudo port sync", to receive the new 
version of MacPorts.

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 8, 2018, at 18:27, Ryan Schmidt wrote:

> On Mar 8, 2018, at 15:26, Joshua Root wrote:
> 
>> On 2018-3-9 07:07 , Ryan Schmidt wrote:
>> 
>>> On Mar 8, 2018, at 11:24, Joshua Root wrote:
>>> 
 Code to check C++ stdlib linkage in rev-upgrade is now in master. That
 takes care of the main obstacle to being able to change the default stdlib.
>>> 
>>> Currently, we publish archives on packages.macports.org for 10.8 and 
>>> earlier that are built for libstdc++. Currently, if we were to create 
>>> archives for libc++ on those systems, the archive names would be the same. 
>>> Is your plan to leave it that way, or to change the archive names for 
>>> libc++ to distinguish them from the old libstdc++ archives?
>> 
>> I was not planning to rename the archives. It would work if we left all
>> the existing archives, albeit inefficiently when an archive using the
>> old stdlib is downloaded and immediately gets rebuilt.
> 
> Ok, rev-upgrade would detect a mismatch between the user's selected 
> cxx_stdlib and the one in the archive they received. That does answer one 
> concern of mine.

Well, no it doesn't. You are proposing to "stealth update" our current 
libstdc++ archives and replace them with libc++ archives. This would horribly 
break the MacPorts installations of anyone running any version of MacPorts 
that's current today, which doesn't have a rev-upgrade that knows about 
cxx_stdlib nor does it know what cxx_stdlib or delete_la_files setting an 
archive site uses. So we need to be sure that all users have upgraded to the 
future version of MacPorts that have those capabilities before replacing the 
archives. How do we know when all users have done that?

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 9, 2018, at 08:18, Joshua Root wrote:

>> What subset of the instructions on the LibcxxOnOlderSystems page would they 
>> have to follow?
> 
> None of them, hopefully.
> 
> We may have to create some bootstrap versions of toolchain ports in
> order to achieve this, so that they can be automatically installed as
> dependencies with no manual intervention. I'm hoping someone more
> familiar with building a C++11 toolchain on these old OS versions will
> chime in.

Looking at the instructions, it looks like Lion and Mountain Lion will be no 
problem, but there are massive bootstrap instructions for Snow Leopard which 
would need some major port changes to automate an upgrade to.

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 12, 2018, at 13:56, db wrote:
> On 11 Mar 2018, at 15:47, Ken Cunningham wrote:
>>> On Mar 11, 2018, at 06:06, db wrote:
>>> On 11 Mar 2018, at 02:47, Kenneth F. Cunningham wrote:
> On 2018-03-10, at 12:23 PM, db wrote:
> Except for building from source for minor versions and revbumps, 
> especially large binaries, and for ports that have open defects. Oh well…
 Well, everything for 10.6.8 users on MacPorts is about to get supremely 
 better. Hard to be too upset about that!
>>> I don't doubt that! What I don't understand is, why would port be allowed 
>>> to build a port from source on a certain system that already failed on the 
>>> buildbot, and what's worse, be left with a port in a non-working state, 
>>> while downloading binaries would work, because it couldn't donwload 
>>> something that's failed to build.
>> You can sent your buildfromsource value to "never" I believe  would that 
>> do what you're asking?
> 
> No. Recently, someone posted about a port dependent on qscintilla-qt4 left in 
> a non-working state because this dependency fails to build. Why is a current 
> version of software made available in a port definition whereby it cannot be 
> built by the buildbot yet in the first place?

Maybe because it worked for the developer on their system before they committed 
it, and they were not aware that it would fail in other situations. Maybe the 
problem only affected one subport, or one variant, or one version of macOS, and 
the developer did not test that specifically beforehand. Who knows. Why does 
any software have bugs? What we can do is try to fix the bugs when we become 
aware of them, as Michael did to fix this issue yesterday.


> I noticed that qscintilla-qt4 is not distributed at 
> http://distfiles.macports.org/, although the license should allow it, I guess.

Yes it is: https://distfiles.macports.org/qscintilla/

As far as we know, we are allowed to distribute the distfiles of all but three 
of our ports.

Distributing binary archives is much more likely to be affected by license 
restrictions, as it is for qscintilla-qt4, for example:

"qscintilla-qt4" is not distributable because its license "gpl" conflicts with 
license "OpenSSL" of dependency "openssl"


> But even if it weren't, does the buildbot build software that's not 
> distributable just for the sake of testing?

Of course it does. You're welcome to observe what the buildbot does: 
https://build.macports.org/waterfall


> If not, shouldn't it, prior to accepting an updated port definition?

The buildbot is only engaged after a commit is in the repository master branch.

We have a separate system, using Travis CI, for doing test builds of updates 
that have been submitted as pull requests but have not yet been merged into the 
repository master branch. But most maintainers that are committers don't submit 
pull requests for their own software, they just commit directly to master, so 
their changes would not go through Travis.

Re: Changing default cxx_stdlib to libc++

[Rainer Müller]

On 2018-03-09 15:21, Joshua Root wrote:
> On 2018-3-9 20:38 , db wrote:
>> On 9 Mar 2018, at 01:27, Ryan Schmidt  wrote:
>>> When would we run this script to delete libstdc++ archives? Presumably, 
>>> after all legacy users have MacPorts 2.5 and have switched to libc++.
>>
>> How about those that build MP from source?
> 
> There are no additional problems arising from that, whether you mean
> building base from source or building all ports from source. In fact
> there's one less problem if you're not downloading any binary archives,
> as you can't get ones that are built for the wrong cxx_stdlib.

I think we should use cxx_stdlib as one additional criteria to respect
when selecting an archive site here:

https://github.com/macports/macports-ports/blob/master/_resources/port1.0/fetch/archive_sites.tcl

https://github.com/macports/macports-base/blob/995dde8476c48580db4f6eedfde09e90dc5e8c99/src/package1.0/portarchivefetch.tcl#L66

Then the archive site would no longer be used if anyone changes the
default in macports.conf.

Rainer

Re: Changing default cxx_stdlib to libc++

[Ken Cunningham]

> On Mar 11, 2018, at 06:06, db  wrote:
> 
>> On 11 Mar 2018, at 02:47, "Kenneth F. Cunningham" 
>>  wrote:
>>> On 2018-03-10, at 12:23 PM, db wrote:
>>> Except for building from source for minor versions and revbumps, especially 
>>> large binaries, and for ports that have open defects. Oh well…
>> Well, everything for 10.6.8 users on MacPorts is about to get supremely 
>> better. Hard to be too upset about that!
> 
> I don't doubt that! What I don't understand is, why would port be allowed to 
> build a port from source on a certain system that already failed on the 
> buildbot, and what's worse, be left with a port in a non-working state, while 
> downloading binaries would work, because it couldn't donwload something 
> that's failed to build.

You can sent your buildfromsource value to "never" I believe  would that do 
what you're asking?

K

Re: Changing default cxx_stdlib to libc++

[db]

On 11 Mar 2018, at 02:47, "Kenneth F. Cunningham" 
 wrote:
> On 2018-03-10, at 12:23 PM, db wrote:
>> Except for building from source for minor versions and revbumps, especially 
>> large binaries, and for ports that have open defects. Oh well…
> Well, everything for 10.6.8 users on MacPorts is about to get supremely 
> better. Hard to be too upset about that!

I don't doubt that! What I don't understand is, why would port be allowed to 
build a port from source on a certain system that already failed on the 
buildbot, and what's worse, be left with a port in a non-working state, while 
downloading binaries would work, because it couldn't donwload something that's 
failed to build.

Re: Changing default cxx_stdlib to libc++

[Kenneth F. Cunningham]

On 2018-03-10, at 12:23 PM, db wrote:

> On 10 Mar 2018, at 18:06, "Kenneth F. Cunningham" 
>  wrote:
>> I think I"m likely not going to change anything on my LibcxxOnOlderSystems 
>> 10.6.8 system for a while, db. It will continue to work just as it does now, 
>> and I don't mind building from source -- kinda got used to it.
> 
> Except for building from source for minor versions and revbumps, especially 
> large binaries, and for ports that have open defects. Oh well…

Well, everything for 10.6.8 users on MacPorts is about to get supremely better. 
Hard to be too upset about that!

Ken

Re: Changing default cxx_stdlib to libc++

[db]

On 10 Mar 2018, at 18:06, "Kenneth F. Cunningham" 
 wrote:
> I think I"m likely not going to change anything on my LibcxxOnOlderSystems 
> 10.6.8 system for a while, db. It will continue to work just as it does now, 
> and I don't mind building from source -- kinda got used to it.

Except for building from source for minor versions and revbumps, especially 
large binaries, and for ports that have open defects. Oh well…

Re: Changing default cxx_stdlib to libc++

[Kenneth F. Cunningham]

On 2018-03-10, at 9:06 AM, Kenneth F. Cunningham wrote:


> I'll probably keep building against a current lib curl installed via 
> /opt/bootstrap as well, just 'cause it works.

I meant to say I'll keep building MacPorts against a current libcurl...

> Best,
> 
> Ken

Re: Changing default cxx_stdlib to libc++

[Kenneth F. Cunningham]

On 2018-03-10, at 3:38 AM, db wrote:


> How will I know about the lib change and the availability of binaries? 
> Besides rebuilding from source from time to time, I only do port sync.

I think I"m likely not going to change anything on my LibcxxOnOlderSystems 
10.6.8 system for a while, db. It will continue to work just as it does now, 
and I don't mind building from source -- kinda got used to it. So there will be 
no difference at all. I'll probably keep building against a current lib curl 
installed via /opt/bootstrap as well, just 'cause it works.

However, at some point in the next few months, when most of the dust has 
settled and most or all of the available binaries are all properly built 
against libc++, I'll change the single line in macports.conf from 
"buildfromsource always" to "buildfromsource if needed" , and get some prebuilt 
binaries. This will be nice for the real big ones, like octave and llvm and 
clang and qt4 etc.

The real benefit to older systems will be supporting them. If it all goes as 
planned, there should be considerably less work to do to fix odd errors that 
show up on older compilers. I may be able to roll out the webkit2gtk fixes I 
have that work on libc++ but are just too much of a PITA to sort out against 
gcc7's libstdc++.

All ports will build with a version of clang similar to a 10.13 system's clang, 
and against libc++. 

The only hiccups should be deficiencies in the SDK, libc, and some Xcode things 
-- libc we can fix most of, the rest is  -- well -- every system has to die 
sometime.

I would like to see clang set up on these systems to default to adding 
-stdlib=libc++ if it's not otherwise specified on the build line, like newer 
systems do -- but even if  that isn't allowable, there should be a marked 
reduction in fixing ancient headers / outdated compiler capabilities, etc

Best,

Ken

Re: Changing default cxx_stdlib to libc++

[db]

On 10 Mar 2018, at 00:09, Joshua Root  wrote:
> On 2018-3-10 08:43 , db wrote:
>> 
>> If I build base and ports from source, will buildfromsource be changed 
>> unilaterally by MP or would it need user intervention? I also have certain 
>> versions of gcc and llvm locally to avoid rebuilding after minor versions 
>> and revbumps.
> We're not going to modify your macports.conf.

How will I know about the lib change and the availability of binaries? Besides 
rebuilding from source from time to time, I only do port sync.

Re: Changing default cxx_stdlib to libc++

[Joshua Root]

On 2018-3-10 08:43 , db wrote:
> On 9 Mar 2018, at 15:21, Joshua Root  wrote:
>> On 2018-3-9 20:38 , db wrote:
>>> On 9 Mar 2018, at 01:27, Ryan Schmidt  wrote:
 When would we run this script to delete libstdc++ archives? Presumably, 
 after all legacy users have MacPorts 2.5 and have switched to libc++.
>>> How about those that build MP from source?
>> There are no additional problems arising from that
> 
> If I build base and ports from source, will buildfromsource be changed 
> unilaterally by MP or would it need user intervention? I also have certain 
> versions of gcc and llvm locally to avoid rebuilding after minor versions and 
> revbumps.

We're not going to modify your macports.conf.

- Josh

Re: Changing default cxx_stdlib to libc++

[db]

On 9 Mar 2018, at 15:21, Joshua Root  wrote:
> On 2018-3-9 20:38 , db wrote:
>> On 9 Mar 2018, at 01:27, Ryan Schmidt  wrote:
>>> When would we run this script to delete libstdc++ archives? Presumably, 
>>> after all legacy users have MacPorts 2.5 and have switched to libc++.
>> How about those that build MP from source?
> There are no additional problems arising from that

If I build base and ports from source, will buildfromsource be changed 
unilaterally by MP or would it need user intervention? I also have certain 
versions of gcc and llvm locally to avoid rebuilding after minor versions and 
revbumps.

Re: Changing default cxx_stdlib to libc++

[Ken Cunningham]

> On Mar 9, 2018, at 06:18, Joshua Root  wrote:
> 
> We may have to create some bootstrap versions of toolchain ports in
> order to achieve this, so that they can be automatically installed as
> dependencies with no manual intervention.

Maybe we could just force clang 3.4 to build against libstdc++. -- perhaps 
dependant on whether libcxx is installed or not 

Ken

Re: Changing default cxx_stdlib to libc++

[Joshua Root]

On 2018-3-9 20:38 , db wrote:
> On 9 Mar 2018, at 01:27, Ryan Schmidt  wrote:
>> When would we run this script to delete libstdc++ archives? Presumably, 
>> after all legacy users have MacPorts 2.5 and have switched to libc++.
> 
> How about those that build MP from source?

There are no additional problems arising from that, whether you mean
building base from source or building all ports from source. In fact
there's one less problem if you're not downloading any binary archives,
as you can't get ones that are built for the wrong cxx_stdlib.

- Josh

Re: Changing default cxx_stdlib to libc++

[Joshua Root]

On 2018-3-9 11:27 , Ryan Schmidt wrote:
> 
> When would we run this script to delete libstdc++ archives? Presumably, after 
> all legacy users have MacPorts 2.5 and have switched to libc++. When is that? 
> Certainly no sooner than two weeks after release since that's how long it 
> takes MacPorts to prompt a user to selfupdate. Do all users follow that 
> prompt right away? The prompt goes away if you sync rather than selfupdate, 
> right? Due to the length of time between MacPorts releases, users may be in 
> the habit of running sync rather than selfupdate, since sync is faster. It 
> may be some time before all users actually upgrade to 2.5.

Sure. We could release a version that records and checks the stdlib and
save the actual changeover for a later version. Don't want to wait too
long though.

I imagine we'd run the script once right before we release the base
version that changes the cxx_stdlib default value, then do the release,
turn off the legacy builders, turn on the new ones, and run the script
again just to get any ports that got built in between.

> How will users upgrade from libstdc++ to libc++?

The default value of cxx_stdlib will simply change with a base release.
We'll want to announce the changeover well in advance of that release,
and have prominent notices shown (to tell users who followed
LibcxxOnOlderSystems that they can return to 'buildfromsource ifneeded'
among other things).

> What subset of the instructions on the LibcxxOnOlderSystems page would they 
> have to follow?

None of them, hopefully.

We may have to create some bootstrap versions of toolchain ports in
order to achieve this, so that they can be automatically installed as
dependencies with no manual intervention. I'm hoping someone more
familiar with building a C++11 toolchain on these old OS versions will
chime in.

> What if they don't do that and stay on libstdc++?
They would have to specifically set cxx_stdlib in macports.conf in order
to do that. If they do, it will work fine. They may want to set
'buildfromsource always' to prevent useless downloading of C++ ports.

> This strategy means we won't have any libc++ archives available for legacy 
> users when they switch over. It will take us time to build them. If we 
> changed the archive name and/or directory they're in, we could build libc++ 
> archives in advance and have them ready for users when they switch.

Well, to be fair, users who followed LibcxxOnOlderSystems get no
binaries right now (and users on libstdc++ get no C++11 ports, binary or
otherwise). We could turn on the non-legacy builders early but not
deploy the archives, or deploy them somewhere else and then move them?

- Josh

Re: Changing default cxx_stdlib to libc++

[db]

On 9 Mar 2018, at 01:27, Ryan Schmidt  wrote:
> When would we run this script to delete libstdc++ archives? Presumably, after 
> all legacy users have MacPorts 2.5 and have switched to libc++.

How about those that build MP from source?

Re: Changing default cxx_stdlib to libc++

[Ken Cunningham]

On 2018-03-08, at 4:27 PM, Ryan Schmidt wrote:


> How will users upgrade from libstdc++ to libc++? What subset of the 
> instructions on the LibcxxOnOlderSystems page would they have to follow? What 
> if they don't do that and stay on libstdc++?

We do have to ponder the bootstrap toolchain process carefully. There is a 
clang-3.4 build that is against libstdc++ as a stepping stone, and some ld64 
bootstrapping. 

Probably need to walk it through on a VM to make sure it all jives. I can help 
with that, as I'm sure others can.

Ken

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 8, 2018, at 15:26, Joshua Root wrote:

> On 2018-3-9 07:07 , Ryan Schmidt wrote:
> 
>> On Mar 8, 2018, at 11:24, Joshua Root wrote:
>> 
>>> Code to check C++ stdlib linkage in rev-upgrade is now in master. That
>>> takes care of the main obstacle to being able to change the default stdlib.
>> 
>> Currently, we publish archives on packages.macports.org for 10.8 and earlier 
>> that are built for libstdc++. Currently, if we were to create archives for 
>> libc++ on those systems, the archive names would be the same. Is your plan 
>> to leave it that way, or to change the archive names for libc++ to 
>> distinguish them from the old libstdc++ archives?
> 
> I was not planning to rename the archives. It would work if we left all
> the existing archives, albeit inefficiently when an archive using the
> old stdlib is downloaded and immediately gets rebuilt.

Ok, rev-upgrade would detect a mismatch between the user's selected cxx_stdlib 
and the one in the archive they received. That does answer one concern of mine.

> I figured we
> could remove all the archives that use C++ (quick hacked together
> example script attached - probably want to clean up old archives before
> running it...).
> 
> - Josh
> 

When would we run this script to delete libstdc++ archives? Presumably, after 
all legacy users have MacPorts 2.5 and have switched to libc++. When is that? 
Certainly no sooner than two weeks after release since that's how long it takes 
MacPorts to prompt a user to selfupdate. Do all users follow that prompt right 
away? The prompt goes away if you sync rather than selfupdate, right? Due to 
the length of time between MacPorts releases, users may be in the habit of 
running sync rather than selfupdate, since sync is faster. It may be some time 
before all users actually upgrade to 2.5.

How will users upgrade from libstdc++ to libc++? What subset of the 
instructions on the LibcxxOnOlderSystems page would they have to follow? What 
if they don't do that and stay on libstdc++?

This strategy means we won't have any libc++ archives available for legacy 
users when they switch over. It will take us time to build them. If we changed 
the archive name and/or directory they're in, we could build libc++ archives in 
advance and have them ready for users when they switch.

Re: Changing default cxx_stdlib to libc++

[Joshua Root]

On 2018-3-9 07:07 , Ryan Schmidt wrote:
> 
> On Mar 8, 2018, at 11:24, Joshua Root wrote:
> 
>> Code to check C++ stdlib linkage in rev-upgrade is now in master. That
>> takes care of the main obstacle to being able to change the default stdlib.
> 
> Currently, we publish archives on packages.macports.org for 10.8 and earlier 
> that are built for libstdc++. Currently, if we were to create archives for 
> libc++ on those systems, the archive names would be the same. Is your plan to 
> leave it that way, or to change the archive names for libc++ to distinguish 
> them from the old libstdc++ archives?

I was not planning to rename the archives. It would work if we left all
the existing archives, albeit inefficiently when an archive using the
old stdlib is downloaded and immediately gets rebuilt. I figured we
could remove all the archives that use C++ (quick hacked together
example script attached - probably want to clean up old archives before
running it...).

- Josh


cxx_archives.sh
Description: Bourne shell script

Re: Changing default cxx_stdlib to libc++

[Ryan Schmidt]

On Mar 8, 2018, at 11:24, Joshua Root wrote:

> Code to check C++ stdlib linkage in rev-upgrade is now in master. That
> takes care of the main obstacle to being able to change the default stdlib.

Currently, we publish archives on packages.macports.org for 10.8 and earlier 
that are built for libstdc++. Currently, if we were to create archives for 
libc++ on those systems, the archive names would be the same. Is your plan to 
leave it that way, or to change the archive names for libc++ to distinguish 
them from the old libstdc++ archives?

Re: Changing default cxx_stdlib to libc++

[Ken Cunningham]

Thank God. 

10.6.8, agree. 

I use clang 3.9 as a good balance between capability and compatability, but 
might as well sort out how to use clang 5 as that is more consistent w older 
systems, and I nearly have thread_local working there on 10.6

Ken

Sent from my iPhone

> On Mar 8, 2018, at 9:24 AM, Joshua Root  wrote:
> 
> Code to check C++ stdlib linkage in rev-upgrade is now in master. That
> takes care of the main obstacle to being able to change the default stdlib.
> 
> That leaves a couple of questions:
> 
> Which OS versions can we make the change on? At first glance it looks
> like probably 10.6 through 10.8. 10.5 appears to have a much more
> involved bootstrap process.
> 
> Do we need to make changes to the compiler selection logic? The
> LibcxxOnOlderSystems instructions override the selection in
> macports.conf. Will there be any bootstrap issues for users after the
> switch?
> 
> - Josh