Re: libgcc11 on macOS monterey

[Enrico Maria Crisostomo]

Hi Brian,

My system just finished building it and I had no issues:

libgcc11   @11.2.0_1

Have you updated your port installation and your ports tree? There have
been quite a lot of updates in the last few hours (even though I'm not
aware of anything that would break this particular port).

Cheers,
Enrico

On Thu, Oct 28, 2021 at 7:23 PM Brian Weitzner 
wrote:

> Hi all,
>
> I discovered that libgcc11 fails to build on the just-released macOS
> Monterey, with the error being that the Darwin version is unsupported. I
> opened a ticket on trac  but
> there doesn’t appear to be a maintainer for the port. Since this is an
> important tool I’m wondering if there is anyone with experience who would
> be willing to update it or work with me to update it so we can get to more
> complete Monterey compatibility sooner. Thanks!
>
> --
> Brian Weitzner
>
>

Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)

[Enrico Maria Crisostomo]

Hi,

Thanks Ryan.

My answer is very similar to Ben’s:

  *   I’d be happy to provide you exclusive access to the resources (encrypted 
VMs, your own users, network and machine are UPS-protected, firewalled, etc.)
  *   I completely agree with you about the safety concerns: those should not 
be relaxed.
  *   I volunteered because I thought they were needed: I love MacPorts, and I 
want it to thrive.

Bye,
Enrico


From: Ben Greenfield 
Date: Friday, 21 May 2021 at 13:26
To: Ryan Schmidt 
Cc: Andrew Janke , Enrico Maria Crisostomo 
, MacPorts Developers 

Subject: Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)
Hey All,

Thanks for the direction Ryan.

> On May 21, 2021, at 12:46 AM, Ryan Schmidt  wrote:
>
> On May 19, 2021, at 12:38, Andrew Janke wrote:
>
>> I have a small stack of Mac Minis I got to use as a buildbot farm for 
>> Octave.app; I might be able to have them pull double duty for MacPorts 
>> depending on your change volume.
>
>
> On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote:
>
>> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite 
>> often idle.
>> I'm not privy with how our build system work, but if we could get to a point 
>> where agents can be added, stopped, throttled, trusted members of our 
>> community could volunteer the computational power they have at their 
>> disposal without fully dedicating a machine.
>> In my specific case: I'm happy to offer VMs on that machine to volunteer 
>> computational resources.
>
>
> On May 20, 2021, at 08:20, Ben Greenfield wrote:
>
>> I can definitely donate the facilities if not the talent.
>>
>> I have a symmetrical fiber connection and a static ip. I also have battery 
>> backup.
>> I’m in the final weeks of making the building legal and I haven’t configured 
>> the final network set-up for the building. I was going to set-up a vlan on 
>> my hp procurve switch.
>> I’m still shopping for a router to run OPNsense I think.
>>
>> I have been a mac sysadmin long time.
>
>
> There seem to be a lot of people suddenly volunteering hardware for our build 
> system. First, thank you; I didn't know we had people interested in that.
>
> Our build system has never been designed to accommodate external hardware. It 
> has always been designed as a centralized system controlled by one 
> administrator. When it was first set up in 2011-12 it was under the control 
> of our Apple administrator at macOS forge. I became the macOS forge 
> administrator temporarily in late 2015, and MacPorts left macOS forge in late 
> 2016 as that service shut down, and I recreated the buildbot system on my own 
> hardware and have run it since then.
>
> We now have one external Apple Silicon build machine hosted at another data 
> center, but it's still under my exclusive control so that I can keep 
> everything working together.
>

I would be happy to provide the same service. I don’t need a log-in and I can 
probably provide out of band power reset. The system could be on it’s own vlan.


> There are currently many situations where the build system gets into a state 
> that requires manual intervention. Because I control all the machines, I'm 
> able to make those fixes and get things back up and running quickly.
>
> We currently have all the builders we need: one for each OS version / arch 
> combination. The system was never designed to have more than that. If for 
> example we added a second macOS 11 / x86_64 builder, there could be confusion 
> and problems if the two machines have different OS / Xcode / command line 
> tools / java versions installed.
>
> There are security issues to consider. The binaries produced by our buildbot 
> workers are signed on the master with our private key. This is our "seal of 
> approval" that says we believe these binaries to be good and safe. Users 
> trust that. If we start allowing other people to run build machines, then we 
> have the problem that we do not know for certain whether those other build 
> machines are free of malware or other problems. We would be signing binaries 
> for distribution to users without being certain of their safety or 
> correctness. I'm not very comfortable with that.

Yes, that safety should be maintained.

>
> Why is this discussion happening? Why do people think we need more hardware? 
> If we need more or faster CPUs or more memory, I can make those changes to 
> the hardware I already manage.

I volunteered because it sounded like resources might be needed:).

Let me know if the free-hosting is needed.

Ben

>

Re: Framing the MacPorts discussion

[Enrico Maria Crisostomo]

Hi all,

I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is
quite often idle.
I'm not privy with how our build system work, but if we could get to a
point where agents can be added, stopped, throttled, trusted members of our
community could volunteer the computational power they have at their
disposal without fully dedicating a machine.
In my specific case: I'm happy to offer VMs on that machine to volunteer
computational resources.

On Wed, May 19, 2021 at 7:38 PM Andrew Janke  wrote:

> I have a small stack of Mac Minis I got to use as a buildbot farm for
> Octave.app; I might be able to have them pull double duty for MacPorts
> depending on your change volume.
>
> Cheers,
> Andrew
>
> On 5/19/21 1:16 PM, Mark Anderson wrote:
>
> Yeah - we are certainly short staffed everywhere - I try to add more and
> more of my time to the project but aside from my ports, I'm still in
> learning mode digging through all the asciidoc and tcl and everything. I'm
> trying to build some tools to help me, but again, more time.
>
> Once we move to the latest build bot, we might want to see if we can get
> other volunteers to host some hardware, but the problem is, we're going to
> have to hit up ebay or something and the infrastructure will be tougher.
>
> I'm honestly not sure how we can manage to staff up more at all - I mean
> this is a FOSS problem all over for non-company sponsored projects.
>
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Tue, May 18, 2021 at 2:35 AM Ryan Schmidt 
> wrote:
>
>> MacPorts is short-staffed in all areas, not just infrastructure.
>>
>> Our Buildbot system works. It produces all the binaries we are able to.
>>
>> Our buildbot system was already substantially redesigned when we took it
>> over from Apple in 2016 and will be substantially redesigned again as we
>> upgrade to the latest version of the buildbot software.
>>
>> We already have a small infrastructure team who are interested in working
>> on improving the buildbot system and our other infrastructure, and do so.
>>
>> We already use GitHub Actions for CI. We cannot use it to replace
>> buildbot because it only offers recent OS versions and because it does not
>> offer persistent machines.
>>
>> I personally am not comfortable adding other build machines to our
>> buildbot system that I do not control. When I control the machines, I know
>> what is installed on them and that they are set up correctly. Having build
>> machines located outside of my local network also poses additional
>> challenges, as I've learned by having our Apple Silicon build machine
>> outside of my network, challenges which I would prefer to minimize, not
>> increase.
>>
>> We currently use one build machine per OS version / arch, and have the
>> hardware needed to do that. Adding more hardware such that we have more
>> than one build machine per OS version / arch is not something our buildbot
>> system was ever designed to accommodate, and would introduce problems.
>>
>> Using Linux and commodity hardware is not applicable because it the macOS
>> EULA only permits running macOS on Apple hardware, as we currently do.
>
>
>

python portgroup best practices for apps (and how to migrate a port from module to app)

[Enrico Maria Crisostomo]

Hi all,

Yesterday a brief discussion started in macports-ports PR #6496
 where we were
arguing about python apps best practices. Basically, a port update
submission was made to add variants to a port (httpie) to support
additional python versions. The PR author notices:

There are ports like httpie which tracks latest python version, others like
> pgcli which use variants and finally ports like py-awscli which use
> subports. None of them are usable as libraries.


I had this very doubt a few days ago, when I updated the docker-compose
port. The python portgroup sources (python-1.0.tcl) state:

# Usage:
# name should be of the form py-foo for modules
# subports pyXY-foo are declared for each XY in python.versions

# for apps (i.e. not named py-foo), no subports will be defined
# only the python.default_version will be used
# you can change that in variants if you want

I'm inclined to say, then, that the PR author approach is correct, in which
case I would also be compelled to migrate other ports I maintain (as the
aforementioned py-awscli). Also, this would have the side effect of
gracefully transition users to the latest python version supported by a
port.

What's your opinion about this?

Also, should I migrate py-awscli to awscli, what would the recommended
course of action be to guarantee that the experience of current users of
py-awscli wouldn't be negatively affected?

Thanks,
Enrico

Re: pypi livecheck not working any longer

[Enrico Maria Crisostomo]

> On 17 Apr 2018, at 01:08, Ryan Schmidt <ryandes...@macports.org> wrote:
> 
> 
> On Apr 16, 2018, at 15:53, Enrico Maria Crisostomo wrote:
> 
>> I was running livecheck a few minutes ago on a bunch of ports of mine and I 
>> started getting the same error on all the pythons ones:
>> 
>>   Error: cannot check if [portname] was updated (regex didn't match)
> 
>> I also noticed the pypi.org web site has been completely renewed.
>> 
>> If this problem is confirmed, we should update the regex.
> 
> https://github.com/macports/macports-ports/commit/58bd00863442dfe8228b5003d5213b00ebded2c7
> 
>> And I'm thinking that parsing this document with a regular expression is 
>> going to be less robust than it was: the version number is now just a quoted 
>> field name.  I wonder whether we could and should properly parse the JSON 
>> document here.
> 
> Doesn't appear to be needed.
> 

Thank you very much Ryan.

pypi livecheck not working any longer

[Enrico Maria Crisostomo]

Hi all,

I was running livecheck a few minutes ago on a bunch of ports of mine and I 
started getting the same error on all the pythons ones:

Error: cannot check if [portname] was updated (regex didn't match)

All the ports use pypi so, after a quick search I think livecheck is using by 
default this regex (see pypi.tcl):

livecheck.regex {"version": "(.+)",}

on this url:

if {!$has_homepage || ${livecheck.url} eq ${homepage}} {
livecheck.url \
https://pypi.python.org/pypi/${livecheck.name}/json
}

I manually fetched the livecheck URL and I'm getting a different JSON document 
structure:

"releases": {
  "0.0": [],
  "0.1.0": [
{
  "comment_text": "",
  "digests": {
"md5": "9c4575f5602de11b9bc8a580922974da",
"sha256": 
"04b1fafdcfb9eefa9fe03d4c69c71e6fee6e02a39cf3df92bd1f58bc7eb6a3d2"
  },
  "downloads": -1,
...
}
],
"0.1.1": [
  {
...

I also noticed the pypi.org web site has been completely renewed.

If this problem is confirmed, we should update the regex.  And I'm thinking 
that parsing this document with a regular expression is going to be less robust 
than it was: the version number is now just a quoted field name.  I wonder 
whether we could and should properly parse the JSON document here.

Cheers,
-- 
Enrico

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

> On 13 Apr 2018, at 23:55, Joshua Root <j...@macports.org> wrote:
> 
> On 2018-4-14 07:30 , Enrico Maria Crisostomo wrote:
>> I think I found the culprit:
>> 
>>% strings /opt/macports-2.4.3-dirty/libexec/macports/lib/libtcl8.5.dylib 
>> | grep 2\\.4\\.
>>/opt/macports-2.4.2/libexec/macports/lib
>>/opt/macports-2.4.2/libexec/macports/bin
>>/opt/macports-2.4.2/libexec/macports/lib/tcl8.5
>>/opt/macports-2.4.2/libexec/macports/include
>>/opt/macports-2.4.2/libexec/macports/man
>>/opt/macports-2.4.3-dirty/libexec/macports/lib/tcl8.5
>>/opt/macports-2.4.3-dirty/libexec/macports/lib
>> 
>> while the library built from the clean repo is right:
>> 
>>% strings /opt/macports-2.4.3-clean/libexec/macports/lib/libtcl8.5.dylib 
>> | grep 2\\.4\\.
>>/opt/macports-2.4.3-clean/libexec/macports/lib
>>/opt/macports-2.4.3-clean/libexec/macports/bin
>>/opt/macports-2.4.3-clean/libexec/macports/lib/tcl8.5
>>/opt/macports-2.4.3-clean/libexec/macports/include
>>/opt/macports-2.4.3-clean/libexec/macports/man
>>/opt/macports-2.4.3-clean/libexec/macports/lib/tcl8.5
>>/opt/macports-2.4.3-clean/libexec/macports/lib
>> 
>> Hence, the macports installation that was built from the dirty tree is 
>> referring to the installation directory of the previous build.  That also 
>> explains why removing that directory makes port crash, as I said in a 
>> previous mail.
> 
> OK, so the issue is not with switching tags without cleaning at all, but
> configuring for a different prefix without cleaning. The bug is in one
> of Tcl's Makefiles, which compiles tclPkgConfig.c with several -D flags
> using values that are directly substituted into the Makefile by
> autoconf. There should probably be a dependency declared on the Makefile
> itself.
> 
> - Josh

Thanks Josh.

You're right, and thanks to your pointing out the file involved I've just added 
a dependency from tclPkgConfig.o to the Makefile itself and the problem 
disappears.

Here's a PR:

https://github.com/macports/macports-base/pull/79

I'm not merging until somebody has a look at it.

Cheers,
-- 
Enrico

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

I think I found the culprit:

% strings /opt/macports-2.4.3-dirty/libexec/macports/lib/libtcl8.5.dylib | 
grep 2\\.4\\.
/opt/macports-2.4.2/libexec/macports/lib
/opt/macports-2.4.2/libexec/macports/bin
/opt/macports-2.4.2/libexec/macports/lib/tcl8.5
/opt/macports-2.4.2/libexec/macports/include
/opt/macports-2.4.2/libexec/macports/man
/opt/macports-2.4.3-dirty/libexec/macports/lib/tcl8.5
/opt/macports-2.4.3-dirty/libexec/macports/lib

while the library built from the clean repo is right:

% strings /opt/macports-2.4.3-clean/libexec/macports/lib/libtcl8.5.dylib | 
grep 2\\.4\\.
/opt/macports-2.4.3-clean/libexec/macports/lib
/opt/macports-2.4.3-clean/libexec/macports/bin
/opt/macports-2.4.3-clean/libexec/macports/lib/tcl8.5
/opt/macports-2.4.3-clean/libexec/macports/include
/opt/macports-2.4.3-clean/libexec/macports/man
/opt/macports-2.4.3-clean/libexec/macports/lib/tcl8.5
/opt/macports-2.4.3-clean/libexec/macports/lib

Hence, the macports installation that was built from the dirty tree is 
referring to the installation directory of the previous build.  That also 
explains why removing that directory makes port crash, as I said in a previous 
mail.

Cheers,
-- 
Enrico

> On 13 Apr 2018, at 22:28, Enrico Maria Crisostomo 
> <enrico.m.crisost...@gmail.com> wrote:
> 
> Ryan,
> 
> Unless there's something I'm not understanding, in which case I apologise for 
> the noise, I don't think my shell is caching anything.  I went so far as to 
> change my user's .zshrc file and try this four possibilities _with a reboot 
> between each test_:
> 
>  * /opt/local: installed from packages: works fine
>  * /opt/macports-2.4.2: installed from git, built clean: works fine and 
> reports 2.4.2
>  * /opt/macports-2.4.3-dirty: installed from git after switching tag and 
> _not_ cleaning the tree: still reports 2.4.2.
>  * /opt/macports-2.4.3-clean: installed from git after cleaning (git clean 
> -xfd): works fine and reports 2.4.3.
> 
> How could the shell be caching anything?  And why it would be caching for the 
> /opt/macports-2.4.3-dirty tree and not for the /opt/macports-2.4.3-clean tree?
> 
> Cheers,
> -- 
> Enrico
> 
>> On 12 Apr 2018, at 18:44, Ryan Schmidt <ryandes...@macports.org> wrote:
>> 
>> 
>> On Apr 12, 2018, at 03:39, Enrico Maria Crisostomo wrote:
>> 
>>> On 12 Apr 2018, at 02:30, Ryan Schmidt wrote:
>>> 
>>>> On Apr 11, 2018, at 08:48, Enrico Maria Crisostomo wrote:
>>>> 
>>>>> Well, I replicated it:
>>>>> 
>>>>> * Clean the repo (e.g.: git clean -xfd)
>>>>> * git checkout v2.4.2
>>>>> * build and install 2.4.2:
>>>>> 
>>>>>  $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>>>>>  $ MP_PREFIX=/opt/macports-2.4.2
>>>>>  $ ./configure --prefix=$MP_PREFIX 
>>>>> --with-applications-dir=$MP_PREFIX/Applications
>>>>>  $ make
>>>>>  $ sudo make install
>>>>> 
>>>>> * open new terminal
>>>>> * git checkout v2.4.3
>>>>> * configure 2.4.3:
>>>>> 
>>>>>  $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>>>>>  $ MP_PREFIX=/opt/macports-2.4.3
>>>>>  $ ./configure --prefix=$MP_PREFIX 
>>>>> --with-applications-dir=$MP_PREFIX/Applications
>>>>> 
>>>>> * At this point src/macports1.0/macports_autoconf.tcl correctly contains 
>>>>> `variable macports_version "2.4.3"`
>>>>> * build and install 2.4.3:
>>>>> 
>>>>>  $ make
>>>>>  $ sudo make install
>>>>> 
>>>>> * At this point 
>>>>> /opt/macports-2.4.3/libexec/macports/lib/macports1.0/macports_autoconf.tcl
>>>>>  correctly contains `variable macports_version "2.4.3"`.
>>>>> * But port reports 2.4.2:
>>>>> 
>>>>>  % echo path
>>>>>  
>>>>> /opt/macports-2.4.3/bin:/opt/macports-2.4.3/sbin:/bin:/usr/bin:/usr/ucb:/usr/local/bin
>>>>>  % type port
>>>>>  port is /opt/macports-2.4.3/bin/port
>>>>>  % port version
>>>>>  Version: 2.4.2
>>>> 
>>>> Just to be absolutely certain which `port' binary you're running:
>>>> 
>>>> What happens if you run:
>>>> 
>>>> /opt/macports-2.4.3/bin/port version
>>> 
>>> Hi Ryan,
>>> 
>>> This happens:
>>> 
>>>  % /opt/macports-2.4.3-clean/bin/port version
>>>  Version: 2.4.3
>> 
>> Then I would say that MacPorts is working correctly. I would guess that the 
>> problem occurred because of your shell's cached lookup of the port command 
>> in the previous /opt/macports-2.4.2 location.
> 

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Ryan,

Unless there's something I'm not understanding, in which case I apologise for 
the noise, I don't think my shell is caching anything.  I went so far as to 
change my user's .zshrc file and try this four possibilities _with a reboot 
between each test_:

  * /opt/local: installed from packages: works fine
  * /opt/macports-2.4.2: installed from git, built clean: works fine and 
reports 2.4.2
  * /opt/macports-2.4.3-dirty: installed from git after switching tag and _not_ 
cleaning the tree: still reports 2.4.2.
  * /opt/macports-2.4.3-clean: installed from git after cleaning (git clean 
-xfd): works fine and reports 2.4.3.

How could the shell be caching anything?  And why it would be caching for the 
/opt/macports-2.4.3-dirty tree and not for the /opt/macports-2.4.3-clean tree?

Cheers,
-- 
Enrico

> On 12 Apr 2018, at 18:44, Ryan Schmidt <ryandes...@macports.org> wrote:
> 
> 
> On Apr 12, 2018, at 03:39, Enrico Maria Crisostomo wrote:
> 
>> On 12 Apr 2018, at 02:30, Ryan Schmidt wrote:
>> 
>>> On Apr 11, 2018, at 08:48, Enrico Maria Crisostomo wrote:
>>> 
>>>> Well, I replicated it:
>>>> 
>>>> * Clean the repo (e.g.: git clean -xfd)
>>>> * git checkout v2.4.2
>>>> * build and install 2.4.2:
>>>> 
>>>>   $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>>>>   $ MP_PREFIX=/opt/macports-2.4.2
>>>>   $ ./configure --prefix=$MP_PREFIX 
>>>> --with-applications-dir=$MP_PREFIX/Applications
>>>>   $ make
>>>>   $ sudo make install
>>>> 
>>>> * open new terminal
>>>> * git checkout v2.4.3
>>>> * configure 2.4.3:
>>>> 
>>>>   $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>>>>   $ MP_PREFIX=/opt/macports-2.4.3
>>>>   $ ./configure --prefix=$MP_PREFIX 
>>>> --with-applications-dir=$MP_PREFIX/Applications
>>>> 
>>>> * At this point src/macports1.0/macports_autoconf.tcl correctly contains 
>>>> `variable macports_version "2.4.3"`
>>>> * build and install 2.4.3:
>>>> 
>>>>   $ make
>>>>   $ sudo make install
>>>> 
>>>> * At this point 
>>>> /opt/macports-2.4.3/libexec/macports/lib/macports1.0/macports_autoconf.tcl 
>>>> correctly contains `variable macports_version "2.4.3"`.
>>>> * But port reports 2.4.2:
>>>> 
>>>>   % echo path
>>>>   
>>>> /opt/macports-2.4.3/bin:/opt/macports-2.4.3/sbin:/bin:/usr/bin:/usr/ucb:/usr/local/bin
>>>>   % type port
>>>>   port is /opt/macports-2.4.3/bin/port
>>>>   % port version
>>>>   Version: 2.4.2
>>> 
>>> Just to be absolutely certain which `port' binary you're running:
>>> 
>>> What happens if you run:
>>> 
>>> /opt/macports-2.4.3/bin/port version
>> 
>> Hi Ryan,
>> 
>> This happens:
>> 
>>   % /opt/macports-2.4.3-clean/bin/port version
>>   Version: 2.4.3
> 
> Then I would say that MacPorts is working correctly. I would guess that the 
> problem occurred because of your shell's cached lookup of the port command in 
> the previous /opt/macports-2.4.2 location.

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

> On 12 Apr 2018, at 02:30, Ryan Schmidt <ryandes...@macports.org> wrote:
> 
> 
> On Apr 11, 2018, at 08:48, Enrico Maria Crisostomo wrote:
> 
>> Well, I replicated it:
>> 
>> * Clean the repo (e.g.: git clean -xfd)
>> * git checkout v2.4.2
>> * build and install 2.4.2:
>> 
>> $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>> $ MP_PREFIX=/opt/macports-2.4.2
>> $ ./configure --prefix=$MP_PREFIX 
>> --with-applications-dir=$MP_PREFIX/Applications
>> $ make
>> $ sudo make install
>> 
>> * open new terminal
>> * git checkout v2.4.3
>> * configure 2.4.3:
>> 
>> $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>> $ MP_PREFIX=/opt/macports-2.4.3
>> $ ./configure --prefix=$MP_PREFIX 
>> --with-applications-dir=$MP_PREFIX/Applications
>> 
>> * At this point src/macports1.0/macports_autoconf.tcl correctly contains 
>> `variable macports_version "2.4.3"`
>> * build and install 2.4.3:
>> 
>> $ make
>> $ sudo make install
>> 
>> * At this point 
>> /opt/macports-2.4.3/libexec/macports/lib/macports1.0/macports_autoconf.tcl 
>> correctly contains `variable macports_version "2.4.3"`.
>> * But port reports 2.4.2:
>> 
>> % echo path
>> 
>> /opt/macports-2.4.3/bin:/opt/macports-2.4.3/sbin:/bin:/usr/bin:/usr/ucb:/usr/local/bin
>> % type port
>> port is /opt/macports-2.4.3/bin/port
>> % port version
>> Version: 2.4.2
> 
> Just to be absolutely certain which `port' binary you're running:
> 
> What happens if you run:
> 
> /opt/macports-2.4.3/bin/port version

Hi Ryan,

This happens:

% /opt/macports-2.4.3-clean/bin/port version
Version: 2.4.3

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Not if I'm installing a MacPorts instance from source _side-by-side_ with 
another one installed in /opt/local (e.g.: from the packages).  At least that's 
my understanding of the documentation.

> On 11 Apr 2018, at 15:52, G A <artist.impression...@gmail.com> wrote:
> 
> Your path should have /opt/local/bin first.
> 
> On Wed, Apr 11, 2018 at 06:48 Enrico Maria Crisostomo 
> <enrico.m.crisost...@gmail.com> wrote:
> Well, I replicated it:
> 
>   * Clean the repo (e.g.: git clean -xfd)
>   * git checkout v2.4.2
>   * build and install 2.4.2:
> 
>   $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>   $ MP_PREFIX=/opt/macports-2.4.2
>   $ ./configure --prefix=$MP_PREFIX 
> --with-applications-dir=$MP_PREFIX/Applications
>   $ make
>   $ sudo make install
> 
>   * open new terminal
>   * git checkout v2.4.3
>   * configure 2.4.3:
> 
>   $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
>   $ MP_PREFIX=/opt/macports-2.4.3
>   $ ./configure --prefix=$MP_PREFIX 
> --with-applications-dir=$MP_PREFIX/Applications
> 
>   * At this point src/macports1.0/macports_autoconf.tcl correctly contains 
> `variable macports_version "2.4.3"`
>   * build and install 2.4.3:
> 
>   $ make
>   $ sudo make install
> 
>   * At this point 
> /opt/macports-2.4.3/libexec/macports/lib/macports1.0/macports_autoconf.tcl 
> correctly contains `variable macports_version "2.4.3"`.
>   * But port reports 2.4.2:
> 
>   % echo path
>   
> /opt/macports-2.4.3/bin:/opt/macports-2.4.3/sbin:/bin:/usr/bin:/usr/ucb:/usr/local/bin
>   % type port
>   port is /opt/macports-2.4.3/bin/port
>   % port version
>   Version: 2.4.2
> 
> If I clean the repo between the builds, the problem does not happen.
> 
> Cheers,
> --
> Enrico
> 
> > On 11 Apr 2018, at 15:32, Joshua Root <j...@macports.org> wrote:
> >
> > On 2018-4-11 23:27 , Enrico Maria Crisostomo wrote:
> >> Thanks Rainer,
> >>
> >> I can't check it but as I said in my previous mail I think I had forgotten 
> >> to run `make distclean` when I previously built `v2.4.2`.  And now I see 
> >> that the `macports_autoconf.tcl` file you cite is an Autoconf config file. 
> >>  Why this happen now makes sense.
> >
> > As Rainer said, running configure is supposed to update that file (and
> > in my experience it does).
> >
> > - Josh
> 

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Well, I replicated it:

  * Clean the repo (e.g.: git clean -xfd)
  * git checkout v2.4.2
  * build and install 2.4.2:

  $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
  $ MP_PREFIX=/opt/macports-2.4.2
  $ ./configure --prefix=$MP_PREFIX 
--with-applications-dir=$MP_PREFIX/Applications
  $ make
  $ sudo make install

  * open new terminal
  * git checkout v2.4.3
  * configure 2.4.3:

  $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
  $ MP_PREFIX=/opt/macports-2.4.3
  $ ./configure --prefix=$MP_PREFIX 
--with-applications-dir=$MP_PREFIX/Applications

  * At this point src/macports1.0/macports_autoconf.tcl correctly contains 
`variable macports_version "2.4.3"`
  * build and install 2.4.3:

  $ make
  $ sudo make install

  * At this point 
/opt/macports-2.4.3/libexec/macports/lib/macports1.0/macports_autoconf.tcl 
correctly contains `variable macports_version "2.4.3"`.
  * But port reports 2.4.2:

  % echo path
  
/opt/macports-2.4.3/bin:/opt/macports-2.4.3/sbin:/bin:/usr/bin:/usr/ucb:/usr/local/bin
  % type port
  port is /opt/macports-2.4.3/bin/port
  % port version
  Version: 2.4.2

If I clean the repo between the builds, the problem does not happen.

Cheers,
-- 
Enrico

> On 11 Apr 2018, at 15:32, Joshua Root <j...@macports.org> wrote:
> 
> On 2018-4-11 23:27 , Enrico Maria Crisostomo wrote:
>> Thanks Rainer,
>> 
>> I can't check it but as I said in my previous mail I think I had forgotten 
>> to run `make distclean` when I previously built `v2.4.2`.  And now I see 
>> that the `macports_autoconf.tcl` file you cite is an Autoconf config file.  
>> Why this happen now makes sense.
> 
> As Rainer said, running configure is supposed to update that file (and
> in my experience it does).
> 
> - Josh

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Ok, now I'm sufficiently curious as to dig deeper and replicate what I think 
the sequence of events were.

> On 11 Apr 2018, at 15:32, Joshua Root <j...@macports.org> wrote:
> 
> On 2018-4-11 23:27 , Enrico Maria Crisostomo wrote:
>> Thanks Rainer,
>> 
>> I can't check it but as I said in my previous mail I think I had forgotten 
>> to run `make distclean` when I previously built `v2.4.2`.  And now I see 
>> that the `macports_autoconf.tcl` file you cite is an Autoconf config file.  
>> Why this happen now makes sense.
> 
> As Rainer said, running configure is supposed to update that file (and
> in my experience it does).
> 
> - Josh

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Thanks Rainer,

I can't check it but as I said in my previous mail I think I had forgotten to 
run `make distclean` when I previously built `v2.4.2`.  And now I see that the 
`macports_autoconf.tcl` file you cite is an Autoconf config file.  Why this 
happen now makes sense.

Cheers,
-- 
Enrico

> On 11 Apr 2018, at 14:22, Rainer Müller <rai...@macports.org> wrote:
> 
> On 2018-04-11 10:52, Enrico Maria Crisostomo wrote:
>> I've just created a new installation of macports-base from tag v2.4.3 
>> following the instructions in the documentation (basically git checkout 
>> v2.4.3, ./configure ..., make and make install) and I've just noticed that 
>> `port` reports 2.4.2:
> 
> Hm, I cannot reproduce this problem. The configure script is supposed to
> update the relevant file with the version number.
> 
> Check "macports_version" in src/macports1.0/macports_autoconf.tcl in
> your working tree.
> 
> Rainer

Re: port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Hi Joshua,

Thanks.  The port script was the correct one.  I think I re-configured and 
built v2.4.3 over a previous v2.4.2 build but I can't confirm.  I cleaned the 
HEAD and rebuilt and the result is as expected.

I noticed in the documentation that "2.2.3 git install" instructs to make 
distclean after the build, while "2.2.4 Install multiple MacPorts Copies" does 
not.  Perhaps amending the documentation may help others not to stumble on this 
one, even if the blame is to be attributed to the user in this case.  I'll send 
a PR for that one.

Cheers,
-- 
Enrico

> On 11 Apr 2018, at 14:17, Joshua Root <j...@macports.org> wrote:
> 
> On 2018-4-11 18:52 , Enrico Maria Crisostomo wrote:
>> Hi,
>> 
>> I've just created a new installation of macports-base from tag v2.4.3 
>> following the instructions in the documentation (basically git checkout 
>> v2.4.3, ./configure ..., make and make install) and I've just noticed that 
>> `port` reports 2.4.2:
>> 
>>% which port
>>/opt/macports-2.4.3/bin/port
>>% port version
>>Version: 2.4.2
>> 
>> I built from:
>> 
>>% git rev-parse HEAD
>>a393413460aae5ac9749b994681381087089cdb5
>> 
>> which appears to be the correct v2.4.3:
>> 
>>* commit a393413460aae5ac9749b994681381087089cdb5 (HEAD, tag: v2.4.3, 
>> upstream/release-2.4)
>>| Author: Joshua Root <j...@macports.org>
>>| Date:   Wed Apr 11 10:23:40 2018 +1000
>>|
>>| Bump branch version to 2.4.3
>> 
>> Am I missing something obvious or it this a typo?
> 
> I'm not sure how that happened for you. Browsing the tag on GitHub, it
> says 2.4.3 in the appropriate places.
> 
> <https://github.com/macports/macports-base/blob/v2.4.3/config/macports_version>
> <https://github.com/macports/macports-base/blob/v2.4.3/configure#L2790>
> 
> Are you sure your shell didn't cache the location of port? Check with
> 'type port'. (Using 'which' tells you where it would be found if you
> cleared the cache.)
> 
> - Josh

port version reports 2.4.2 when built from tag v2.4.3

[Enrico Maria Crisostomo]

Hi,

I've just created a new installation of macports-base from tag v2.4.3 following 
the instructions in the documentation (basically git checkout v2.4.3, 
./configure ..., make and make install) and I've just noticed that `port` 
reports 2.4.2:

% which port
/opt/macports-2.4.3/bin/port
% port version
Version: 2.4.2

I built from:

% git rev-parse HEAD
a393413460aae5ac9749b994681381087089cdb5

which appears to be the correct v2.4.3:

* commit a393413460aae5ac9749b994681381087089cdb5 (HEAD, tag: v2.4.3, 
upstream/release-2.4)
| Author: Joshua Root 
| Date:   Wed Apr 11 10:23:40 2018 +1000
|
| Bump branch version to 2.4.3

Am I missing something obvious or it this a typo?

Cheers,
-- Enrico

Re: Include file search path problem building py-grpcio with setuptools

[Enrico Maria Crisostomo]

> On 2 Apr 2018, at 19:50, Joshua Root <j...@macports.org> wrote:
> 
> On 2018-4-3 02:12 , Enrico Maria Crisostomo wrote:
>> How is the compiler include file search path set up in this case?  The 
>> /opt/local/include/openssl path should come from MacPorts, but I don't know 
>> where it's set and how I can override it.
> 
> It's not: src/boringssl/err_data.c says #include . So
> apparently it's putting -I/opt/local/include before its internal
> boringssl include path.
> 
> MacPorts normally passes -I${prefix}/include in CPPFLAGS, but that's not
> done for setup.py builds. Perhaps it comes from python, though the only
> place -I flags occur in sysconfig are the CPPFLAGS and PY_CFLAGS
> variables, which I don't think are normally used by distutils. But I
> could be wrong. I guess that leaves cython or some other dependency?
> 
> - Josh

Thank you very much Joshua.

BTW, I just discovered Google Mail was sending your mails into the spam folder. 
 I hope it's fixed now.

Thank you for the overview.  Yes, I also think it's cython.  I'm still 
inspecting the build files though: I'll bump this thread.

Include file search path problem building py-grpcio with setuptools

[Enrico Maria Crisostomo]

Hi,

When building py-grpcio I'm getting this error:

:notice:build --->  Building py27-grpcio
:debug:build Executing proc-pre-org.macports.build-build-0
:debug:build Executing org.macports.build (py27-grpcio)
:debug:build Environment:
:debug:build CC='/usr/bin/clang'
:debug:build CC_PRINT_OPTIONS='YES'
:debug:build 
CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-grpcio/py27-grpcio/work/.CC_PRINT_OPTIONS'
:debug:build CFLAGS='-arch x86_64'
:debug:build CPATH='/opt/local/include'
:debug:build CXX='/usr/bin/clang++'
:debug:build CXXFLAGS='-arch x86_64'
:debug:build F90FLAGS='-m64'
:debug:build FCFLAGS='-m64'
:debug:build FFLAGS='-m64'
:debug:build LDFLAGS='-arch x86_64'
:debug:build LIBRARY_PATH='/opt/local/lib'
:debug:build MACOSX_DEPLOYMENT_TARGET='10.13'
:debug:build OBJC='/usr/bin/clang'
:debug:build OBJCFLAGS='-arch x86_64'
:info:build Executing:  cd 
"/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-grpcio/py27-grpcio/work/grpcio-1.10.0"
 && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 
setup.py --no-user-cfg build
:debug:build system:  cd 
"/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-grpcio/py27-grpcio/work/grpcio-1.10.0"
 && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 
setup.py --no-user-cfg build
... snip ...
:info:build running build_ext
:info:build In file included from src/boringssl/err_data.c:18:
:info:build In file included from /opt/local/include/openssl/err.h:115:
:info:build /opt/local/include/openssl/e_os2.h:276:11: error: 'OPENSSL_EXPORT' 
macro redefined [-Werror,-Wmacro-redefined]
:info:build #  define OPENSSL_EXPORT extern
:info:build   ^
:info:build third_party/boringssl/include/openssl/base.h:178:9: note: previous 
definition is here
:info:build #define OPENSSL_EXPORT
:info:build ^ 

Since grpcio includes boringssl sources (an openssl fork), I think the compiler 
include files search path is too wide and should not contain 
/opt/local/include/openssl.  Examining setup.py I haven't been able to tell why 
are openssl header files pulled in as well.  For reference, this is where 
include paths are set:

CORE_INCLUDE = ('include', '.',)
BORINGSSL_INCLUDE = (os.path.join('third_party', 'boringssl', 'include'),)
ZLIB_INCLUDE = (os.path.join('third_party', 'zlib'),)
CARES_INCLUDE = (
os.path.join('third_party', 'cares'),
os.path.join('third_party', 'cares', 'cares'),)
if 'darwin' in sys.platform:
  CARES_INCLUDE += (os.path.join('third_party', 'cares', 'config_darwin'),)
if 'freebsd' in sys.platform:
  CARES_INCLUDE += (os.path.join('third_party', 'cares', 'config_freebsd'),)
if 'linux' in sys.platform:
  CARES_INCLUDE += (os.path.join('third_party', 'cares', 'config_linux'),)
if 'openbsd' in sys.platform:
  CARES_INCLUDE += (os.path.join('third_party', 'cares', 'config_openbsd'),)

I'm not sure about how I should proceed.  How is the compiler include file 
search path set up in this case?  The /opt/local/include/openssl path should 
come from MacPorts, but I don't know where it's set and how I can override it.

Thank in advance,
-- 
Enrico

Re: Python ports: should we use wheel files?

[Enrico Maria Crisostomo]

> On 2 Apr 2018, at 08:35, Joshua Root  wrote:
> 
> Good summary from Rainer, which covers most of what I was going to say.
> 
> On 2018-4-1 12:53 , Rainer Müller wrote:
>> I see two reasons why python modules need natively compiled binary files:
>> 
>> a) for performance
>> b) they link against another library
>> 
>> In case of b), using pre-compiled binary files is out of the question,
>> because we want to link to our own libraries and not against libraries
>> in /usr/lib.
> 
> I'll just add that some extensions use libraries that do not ship with
> the OS at all, which makes binary wheels impossible unless you either
> statically link or just assume the user has the libs installed in some
> preselected place (which will probably be /usr/local/lib).
> 
> The direction among packagers in general seems to be towards encouraging
> upstream authors to make it easy (or at least possible) for others to
> reproduce their build process. See .
> 
> - Josh

Thanks everybody for sharing your point of view.  I was not aware of all the 
subtleties and your insights have been very helpful.

Rainer: 

> In case of b), using pre-compiled binary files is out of the question,
> because we want to link to our own libraries and not against libraries
> in /usr/lib.

Thanks for you detailed answer.  Linked libraries are indeed an issue.

> Can we rely on Portfile authors to check what is linked?
> Are they aware which libraries this python module links to before
> submitting the Portfile?

I wondered whether a Portfile author should check and I recognise it's a 
brittle process at best.

> It would probably always be safe to use wheels for platform "any", but
> there is also no advantage over setup.py for them.

I agree.

> We would also lose +universal variants for python ports, but I did not
> check if we actually have such ports.

Thanks for pointing this out, I wasn't fully aware.

> Whether we can
> safely ship pre-compiled Java code is still not clear to me.

I had quite a few surprises in the past while I was maintaining 
Logstash/ElasticSearch for FreeBSD or a running Jenkins on Solaris.  These 
project (used to?) include native libraries which sometimes broke on certain OS 
versions.  As far as I can tell, though, if it's pure Java byte code meant to 
run on a compliant runtime, then I would say it's safe.

Ryan:

> For what macOS versions and architectures are they compiled? I assume the 
> list is smaller than what MacPorts currently supports.

The list is actually rather small:

  * tensorflow-1.6.0-...-macosx_10_11_x86_64.whl

And for py-grpcio pretty much the same:

  * grpcio-1.10.0-...-macosx_10_11_x86_64.whl

tensorflow-1.6.0-cp27-cp27m-macosx_10_11_x86_64.whl

> An additional question is: can you fully trust the binary code of all
> the hundreds of python packages? (It's also true that you should not
> always trust the sources until you checked them, but binaries are
> easier to infect without noticing.)

Thanks Mojca, that's a very good point.

Joshua:

> The direction among packagers in general seems to be towards encouraging
> upstream authors to make it easy (or at least possible) for others to
> reproduce their build process. See .
 
Thanks Joshua.  Yes, that a great initiative.



I think you've given me enough reasons to try and build everything from source.

Thanks,
-- 
Enrico

 

Python ports: should we use wheel files?

[Enrico Maria Crisostomo]

Hi all,

While updating the py-tensorflow port to 1.6.0 I started to think about this 
question: we shortly discussed about it on the corresponding PR 
(https://github.com/macports/macports-ports/pull/1499) and decided to bring it 
up in the mailing list.

Safe harbour statement: I'm a Python noob.

The definition of a wheel file is the following:

> Wheel (in this context) is a project that adds the bdist_wheel command to 
> distutils/setuptools. This produces a cross platform binary packaging format 
> (called “wheels” or “wheel files” and defined in PEP 427) that allows Python 
> libraries, even those including binary extensions, to be installed on a 
> system without needing to be built locally.  In the case in point: one 
> TensorFlow dependency has a native component that needs to be built locally.

Now, the problem.  macOS is a platform where TensorFlow is _built_ and _tested_ 
by upstream.  Some of its dependencies (e.g.: grpcio) are packaged and 
distributed as wheel files.  Hence, what upstream supports is the state of a 
system after the dependencies are installed from wheel files.

Given the above assumption, my impression is that we have an opportunity to 
leverage wheel files instead of rebuild each dependency from source.  The 
advantages in my opiion are:

  * The _state_ of the system after the installation of the wheel files shall 
be _exactly_ se seen, tested and validated by upstream.

  * We're not duplicating effort already put in by upstream.

  * We may not be introducing subtle problems.

I'm well aware that the philosophy of MacPorts (and BSD ports in general) is to 
rebuild every package from source.  But I'm wondering, for the sake of 
argument, whether a parallel can be drawn between (Python, wheel file, PyPi) 
and (Java, *AR files, Maven Central).  After all we already ship Java byte-code 
without recompiling it, and sometimes Java projects ship precompiled native 
libraries.

Closing remarks: my expertise with Python is very limited, so I apologise if my 
reasoning may be flaky.  But in the last few months I couldn't but observe that 
the general guideline given by most Python projects I'm using is (a) installing 
Python packages (using pip and/or wheel files) or (b) even build a virtual env 
(again, using a package-driven process).

I like the fact that MacPorts helps us centralise package installations and 
lets us extends our Python installation using ports.  And from a Portfile 
author I really appreciate how easy it is to build a port from a PyPi package 
that uses setuptools to get installed.  But as far as wheel files are 
concerned, especially wheel files that contain native compiled code, isn't it 
worthwhile to at least leverage all the compiling, testing, packaging and 
distribution work that has already been performed by upstream when macOS is a 
supported platform (as in this case)?

Thanks for sharing your point of view.

Cheers,
-- 
Enrico

Re: python package installed with setuptools: fixing permission of the PKG-INFO file

[Enrico Maria Crisostomo]

> On 23 Mar 2018, at 11:37, Mojca Miklavec <mo...@macports.org> wrote:
> 
> On 23 March 2018 at 11:35, Enrico Maria Crisostomo wrote:
>> On 23 Mar 2018, at 11:06, Mojca Miklavec wrote:
>>> 
>>> I would say this in an upstream issue that should be reported and
>>> fixed. (That said, we could probably also have some code to
>>> automatically fix these kind of problems after extracting the files.)
>> 
>> Thanks Mojca.  I'll see whether I can report it upstream, then.  Yes, if 
>> this file should have an expected permission mask perhaps it would be 
>> desirable to fix it and emit a warning.
>> 
>>> For the moment by far the easiest thing to do would be to use
>>>   system "chmod +r ${worksrcpath}/wherever-the-file-is"
>>> probably in post-extract phase.
>> 
>> Thanks, I'll try that then.  The python version appear multiple times in the 
>> path: I'll try with a plain replacement, hoping the path structure is not 
>> python version-dependent.
>> 
>>> You can do it in post-destroot, but why not fix it immediately? (I
>>> guess that other files in the tarball might need fixing as well.)
>> 
>> Good point.  I'll do that in post-extract.  I didn't know whether the setup 
>> code was making some assumptions (some obvious to me) and set it as such.  
>> From your answer I infer that's not the case, in which case the post-extract 
>> phase is probably a better place to do it.
> 
> During post-extract there shouldn't be any strange python version in the path.

Doh!  It makes sense.  Great then, it seems post-extract is the way to go for 
this case.

> 
> Mojca

Re: python package installed with setuptools: fixing permission of the PKG-INFO file

[Enrico Maria Crisostomo]

On 23 Mar 2018, at 11:06, Mojca Miklavec <mo...@macports.org> wrote:
> 
> On 23 March 2018 at 10:28, Enrico Maria Crisostomo
> <enrico.m.crisost...@gmail.com> wrote:
>> Hi all,
>> 
>> I'm working on fixing a `py-tensorflow` port issue 
>> (https://trac.macports.org/ticket/55972) and I've found the following 
>> problem when building `py-protobuf3`:
>> 
>>--->  Building py36-protobuf3
>>Error: Failed to build py36-protobuf3: command execution failed
>> 
>> The log shows the following:
>> 
>>:info:build Executing:  cd 
>> "/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-protobuf3/py36-protobuf3/work/protobuf-3.5.1/python"
>>  && 
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 
>> setup.py --no-user-cfg build
>>:debug:build system:  cd 
>> "/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-protobuf3/py36-protobuf3/work/protobuf-3.5.1/python"
>>  && 
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 
>> setup.py --no-user-cfg build
>>:info:build Traceback (most recent call last):
>>:info:build   File "setup.py", line 12, in 
>>:info:build from setuptools import setup, Extension, find_packages
>>[...snip...]
>>:info:build PermissionError: [Errno 13] Permission denied: 
>> '/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO'
>> 
>> The `py-absl` port is a new port that I've added while fixing this issue.  
>> The `PKG-INFO` file referred to in the log message has the following 
>> permissions:
>> 
>>-rw-r-  1 root  wheel  889 Mar 23 10:04 
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO
>> 
>> I've verified that adding o+r to the permissions fixes the build issue:
>> 
>>chmod o+r 
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO
>> 
>> So far so good.  The question now is: what's the best practice to fix _that_ 
>> file permissions in the `Portfile`?  The `Portfile` is trivial, uses the 
>> `python 1.0` port group and the installation is taken care of by 
>> `setuptools`.  Plus, I'm kind of a Python noob.  Is this an installation 
>> issue which should be fixed in any one of {upstream, setuptools, python port 
>> group, elsewhere}?  Or should I just fix the permissions in the `Portfile` 
>> by running `chmod` manually?  In the latter case, I've tried to guess what 
>> variables to use to refer to that path on the `post-destroot` phase but I 
>> couldn't find the answer.
> 
> I would say this in an upstream issue that should be reported and
> fixed. (That said, we could probably also have some code to
> automatically fix these kind of problems after extracting the files.)

Thanks Mojca.  I'll see whether I can report it upstream, then.  Yes, if this 
file should have an expected permission mask perhaps it would be desirable to 
fix it and emit a warning.

> 
> For the moment by far the easiest thing to do would be to use
>system "chmod +r ${worksrcpath}/wherever-the-file-is"
> probably in post-extract phase.

Thanks, I'll try that then.  The python version appear multiple times in the 
path: I'll try with a plain replacement, hoping the path structure is not 
python version-dependent.

> 
> You can do it in post-destroot, but why not fix it immediately? (I
> guess that other files in the tarball might need fixing as well.)
> 
> Mojca

Good point.  I'll do that in post-extract.  I didn't know whether the setup 
code was making some assumptions (some obvious to me) and set it as such.  From 
your answer I infer that's not the case, in which case the post-extract phase 
is probably a better place to do it.

Cheers,
-- 
Enrico

python package installed with setuptools: fixing permission of the PKG-INFO file

[Enrico Maria Crisostomo]

Hi all,

I'm working on fixing a `py-tensorflow` port issue 
(https://trac.macports.org/ticket/55972) and I've found the following problem 
when building `py-protobuf3`:

--->  Building py36-protobuf3
Error: Failed to build py36-protobuf3: command execution failed

The log shows the following:

:info:build Executing:  cd 
"/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-protobuf3/py36-protobuf3/work/protobuf-3.5.1/python"
 && /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 
setup.py --no-user-cfg build
:debug:build system:  cd 
"/opt/local/var/macports/build/_Users_enrico_repos_github_macports-ports_python_py-protobuf3/py36-protobuf3/work/protobuf-3.5.1/python"
 && /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 
setup.py --no-user-cfg build
:info:build Traceback (most recent call last):
:info:build   File "setup.py", line 12, in 
:info:build from setuptools import setup, Extension, find_packages
[...snip...]
:info:build PermissionError: [Errno 13] Permission denied: 
'/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO'

The `py-absl` port is a new port that I've added while fixing this issue.  The 
`PKG-INFO` file referred to in the log message has the following permissions:

-rw-r-  1 root  wheel  889 Mar 23 10:04 
/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO

I've verified that adding o+r to the permissions fixes the build issue:

chmod o+r 
/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/absl_py-0.1.11-py3.6.egg-info/PKG-INFO

So far so good.  The question now is: what's the best practice to fix _that_ 
file permissions in the `Portfile`?  The `Portfile` is trivial, uses the 
`python 1.0` port group and the installation is taken care of by `setuptools`.  
Plus, I'm kind of a Python noob.  Is this an installation issue which should be 
fixed in any one of {upstream, setuptools, python port group, elsewhere}?  Or 
should I just fix the permissions in the `Portfile` by running `chmod` 
manually?  In the latter case, I've tried to guess what variables to use to 
refer to that path on the `post-destroot` phase but I couldn't find the answer.

Thanks for you guidance,
-- 
Enrico

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

> On 12 Mar 2018, at 08:28, Ryan Schmidt <ryandes...@macports.org> wrote:
> 
> 
> On Mar 9, 2018, at 13:22, Enrico Maria Crisostomo wrote:
> 
>> I pushed to GitHub a skeleton of the idea:
>> 
>>   https://github.com/emcrisostomo/macports-utils
>> 
>> I've moved what I'm using to a new script to see what the end result looked 
>> like. If you want to try it, just grab the release tarball here (if you 
>> don't have the Autotools installed):
>> 
>>   ./configure && make install
>> 
>> otherwise just clone the repo, bootstrap it and use it:
>> 
>>   ./autogen.sh && ./configure && make install
>> 
>> An example:
>> 
>>   $ port-gen --url 
>> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
>> 
>> outputs:
>> 
>>   # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
>> c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
>> 
>>   PortSystem  1.0
>>   PortGroup   github 1.0
>> 
>>   github.setupemcrisostomo semver-utils 1.1.3
>>   github.tarball_from releases
>> 
>>   categories  changeme
>>   platforms   darwin
>>   license GPL-3
>> 
>>   maintainers obfuscated-maintainer-mail \
>>   openmaintainer
>>   description Add a short port description here.
>>   long_descriptionAdd a long port description here.
>> 
>>   homepagehttps://github.com/emcrisostomo/semver-utils
>> 
>>   checksums   md5e65be62dc9e25af8aa467aa99cde1e00 \
>>   rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
>>   sha256 
>> 888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
>>   size   514429
>> 
>>   livecheck.url   ${github.homepage}/releases/latest
> 
> The github portgroup takes care of setting the homepage and livecheck for you.
> 
> 

Thanks Ryan: that's why the expected output must be reviewed and/or designed by 
the experts.

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

> On 11 Mar 2018, at 16:40, Rainer Müller <rai...@macports.org> wrote:
> 
> On 2018-03-09 20:22, Enrico Maria Crisostomo wrote:
>> I pushed to GitHub a skeleton of the idea:
>> 
>>https://github.com/emcrisostomo/macports-utils
>> 
>> I've moved what I'm using to a new script to see what the end result looked 
>> like. If you want to try it, just grab the release tarball here (if you 
>> don't have the Autotools installed):
>> 
>>./configure && make install
>> 
>> otherwise just clone the repo, bootstrap it and use it:
>> 
>>./autogen.sh && ./configure && make install
>> 
>> An example:
>> 
>>$ port-gen --url 
>> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
> 
> Nice work! Looks already very polished for a prototype.

Thanks Rainer.

> 
> I think it would be even nicer to just specify the URL to the project
> (or to the tag) and it will find the latest distfile to download itself.

Yes.

> 
> However, when we are going to add more templates, lots of options will
> be repeated in each of them. If we are later going to change any of
> these, we have to do this in all templates. I am not sure how much of a
> problem that will be.
> 
> Also, such a template approach will never allow to combine different
> features, like using github port group for fetching, but the python port
> group for building the port.
> 

The prototype uses just one monolithic template, but I'd expect the following 
to be true:

  * Templates may not need to be one per possible output, they could be finer 
grained and get concatenated.

  * Dynamic content could be generated and interleaved with template output, if 
necessary.

I think I share your feelings about templates, and I have little idea about how 
complex the output can/should be.


> Rainer

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

Ken,

I pushed to GitHub a skeleton of the idea:

https://github.com/emcrisostomo/macports-utils

I've moved what I'm using to a new script to see what the end result looked 
like. If you want to try it, just grab the release tarball here (if you don't 
have the Autotools installed):

./configure && make install

otherwise just clone the repo, bootstrap it and use it:

./autogen.sh && ./configure && make install

An example:

$ port-gen --url 
https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz

outputs:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem  1.0
PortGroup   github 1.0

github.setupemcrisostomo semver-utils 1.1.3
github.tarball_from releases

categories  changeme
platforms   darwin
license GPL-3

maintainers obfuscated-maintainer-mail \
openmaintainer
description Add a short port description here.
long_descriptionAdd a long port description here.

homepagehttps://github.com/emcrisostomo/semver-utils

checksums   md5e65be62dc9e25af8aa467aa99cde1e00 \
rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
sha256 
888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
size   514429

livecheck.url   ${github.homepage}/releases/latest


The template is intuitive:

$ cat /usr/local/share/macports-utils/templates/github.in
# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem  1.0
PortGroup   github 1.0

github.setup${GITHUB_AUTHOR} ${PORT_NAME} ${PORT_VERSION}
github.tarball_from releases
   
categories  changeme
platforms   darwin
license GPL-3

maintainers obfuscated-maintainer-mail 
openmaintainer
description Add a short port description here.
long_descriptionAdd a long port description here.

homepage${GITHUB_REPO_URL}

checksums   md5${CHECKSUM_MD5} 
rmd160 ${CHECKSUM_RMD160} 
sha256 ${CHECKSUM_SHA256} 
size   ${CHECKSUM_SIZE}

livecheck.url   \${github.homepage}/releases/latest

> On 7 Mar 2018, at 10:23, Enrico Maria Crisostomo 
> <enrico.m.crisost...@gmail.com> wrote:
> 
> Hi Ken,
> 
> I think it's a great idea. I'm maintaining a lot of software both on my 
> personal machines and at work using MacPorts and I've setup some local 
> Makefile targets to do some of this work. Perhaps I could contribute some 
> work on this area, but first I'd like to know what is it that we'd like to 
> have.
> 
> What is currently working for me is a very simple, template-based approach. 
> Since 90% of the time I'm dealing with ports using the github portgroup, I'm 
> basically calculating the relevant variables (port name, version, checksums, 
> etc.) and fill in a template. The template is a file where shell expansion is 
> performed (i.e.: variables in the form ${VAR}) are expanded.) I think this 
> approach could be both very useful and simple to maintain if we had a list of 
> pre-defined templates. It doesn't scale well with the number of different 
> templates to support and their complexity. If we needed templates to have 
> logic in it (flow control, expressions, etc.),then my guts say we should move 
> to a suitable templating engine (or a different kind of solution altogether).
> 
> I've had a look at portfile-gen and it's a nice source of inspiration. Yet, 
> I'm not convinced that outputting text directly from the program flow is 
> either easy to read or easily maintainable, especially if complexity goes up.
> 
> portfile-gen, if I understand correctly, supports the following groups:
> 
>  * perl5
>  * php 
>  * python 
>  * ruby
>  * github
> 
> If we extracted a template for each one of those groups (or more templates in 
> case a group may have multiple templates), then the effort required to come 
> up with a working prototype would be implementing the functions required to 
> populate the environment (i.e.: downloading a file if needed, calculating 
> checksums, examining it). Perhaps we would have to split a Portfile in 
> multiple segments, and generate each segment from a specific template (e.g.: 
> if a port requires GNU Autotools).
> 
> What are your gut feelings about a template-based solution? Do you think it 
> would be able to manage the level of Portfi

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

Thanks Ryan for the explanation.

> On 7 Mar 2018, at 19:51, Ryan Schmidt <ryandes...@macports.org> wrote:
> 
> 
> On Mar 7, 2018, at 07:32, Rainer Müller wrote:
> 
>> On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
>>> This is a little bit offtopic IMHO, anyway: I found surprising too that
>>> the github portgroup was not documented, given the high percentage of
>>> software that I build out of github repositories, and that's why I
>>> recently contributed the documentation of the github portgroup:
>>> 
>>> https://github.com/macports/macports-guide/pull/12
>>> 
>>> It has been merged some days ago. Yet, I don't see it online yet.
>> 
>> That is because we still have not any automatic deployment on
>> build.macports.org for any of our websites. I am irregularly running
>> these jobs from my local buildbot setup on my personal MacBook every few
>> weeks, which is the only reason they are not completely outdated.
>> 
>> I triggered another run of these jobs, so these changes should be
>> available shortly online.
> 
> We now have one of the new "jobs" running on the buildbot, so hopefully 
> adding the others will be easy.
> 

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

This is a little bit offtopic IMHO, anyway: I found surprising too that the 
github portgroup was not documented, given the high percentage of software that 
I build out of github repositories, and that's why I recently contributed the 
documentation of the github portgroup:

https://github.com/macports/macports-guide/pull/12 


It has been merged some days ago. Yet, I don't see it online yet.

> On 7 Mar 2018, at 11:29, db  wrote:
> 
> On 7 Mar 2018, at 01:53, Rainer Müller  wrote:
>> On 2018-03-06 23:00, db wrote:
>>> [...] an *overview* of how to write a portfile is much needed.
>> Isn't this what this chapter in the guide is supposed to provide?
>> https://guide.macports.org/#development
> 
> Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably 
> trying to write a portfile for something hosted on GitHub without knowing 
> about the relative portgroup and its documentation being buried somewhere 
> under the prefix in the tcl file itself.

Re: Installing a Python wheel (whl) file using a port - Tensorflow

[Enrico Maria Crisostomo]

Ah ok, I saw this. If I understand it correctly it's a subset of it. I've
also tried to build it from source but I'm not sure at all I would solve
the primary issue: according to the official documentation, the output of
the build is the whl file:

https://www.tensorflow.org/install/install_sources

So, even if we update the port to build from source, we end up exactly with
the same issue: how to install the whl file.

Or perhaps my Python-foo is so weak I'm not seeing something obvious.

Thanks.

On 18 December 2017 at 23:10:01, Cunningham Ken (
ken.cunningham.web...@gmail.com) wrote:

https://github.com/Homebrew/homebrew-core/blob/master/Formula/libtensorflow.rb



On 2017-12-18, at 2:09 PM, Enrico Maria Crisostomo wrote:

Thanks Ken.

Actually I checked brew and they haven't got a formula for TensorFlow (at
least at the moment).

On 18 December 2017 at 22:57:42, Cunningham Ken (
ken.cunningham.web...@gmail.com) wrote:

FYI, I recall homebrew has tensorflow, so you might get some hints looking
at their formula.

Marius was taking this on a few months ago as well, so might have some
thoughts.

Ken


On 2017-12-18, at 1:52 PM, Enrico Maria Crisostomo wrote:

Hi,

I’m trying to create a port for TensorFlow and I have already accumulated
quite a number of doubts in just a couple of hours I've been working on it.

First of all, I realised (late) that TensorFlow (and some of its
dependencies) is built as a wheel package (a .whl file), and as such it
gets uploaded to the Python Package Index: ​
https://pypi.python.org/pypi/tensorflow/1.4.1.  whl files are meant to be
installed with pip, so my current port file does the following:

  if {${name} ne ${subport}} {

build {
}

destroot.cmdpip-${python.branch}
destroot.pre_args
destroot.args   \
install \
--no-cache-dir \
--no-dependencies \
--target ${destroot}${python.pkgd} \
${worksrcpath}/${distname}${extract.suffix}
destroot.post_args

livecheck.type  none
  }

You can see the PR here:
https://github.com/macports/macports-ports/pull/1131

The port apparently work, but I'd like to submit this port for review for
the following reasons:

  * I'm not sure we should install a whl file this way, invoking `pip`.

  * I had to redefine `master_sites` and `checksums` for each version of
Phyton.

  * I had to redefine the `extract` properties to skip extraction and just
copy the downloaded file into `${worksrcpath}`:

  extract.suffix  .whl
  extract.cmd cp
  extract.pre_args
  extract.post_args   ${worksrcpath}
  extract.mkdir   yes

  * Is there a better idiom to refer to the currently-installed `pip`?

  destroot.cmdpip-${python.branch}

  * Is there a better idiom to refer to the Python site packages directory
in the staging area?

  ${destroot}${python.pkgd}

Finally, I'm wondering whether this is a good idea at all.  I've grep-ed
the ports and I saw no other whl file installed this way.

If you can have a look at the PR I will appreciate any feedback and insight.

Cheers,
-- 
Enrico

Re: Installing a Python wheel (whl) file using a port - Tensorflow

[Enrico Maria Crisostomo]

Thanks Ken.

Actually I checked brew and they haven't got a formula for TensorFlow (at
least at the moment).

On 18 December 2017 at 22:57:42, Cunningham Ken (
ken.cunningham.web...@gmail.com) wrote:

FYI, I recall homebrew has tensorflow, so you might get some hints looking
at their formula.

Marius was taking this on a few months ago as well, so might have some
thoughts.

Ken


On 2017-12-18, at 1:52 PM, Enrico Maria Crisostomo wrote:

Hi,

I’m trying to create a port for TensorFlow and I have already accumulated
quite a number of doubts in just a couple of hours I've been working on it.

First of all, I realised (late) that TensorFlow (and some of its
dependencies) is built as a wheel package (a .whl file), and as such it
gets uploaded to the Python Package Index: ​
https://pypi.python.org/pypi/tensorflow/1.4.1.  whl files are meant to be
installed with pip, so my current port file does the following:

  if {${name} ne ${subport}} {

build {
}

destroot.cmdpip-${python.branch}
destroot.pre_args
destroot.args   \
install \
--no-cache-dir \
--no-dependencies \
--target ${destroot}${python.pkgd} \
${worksrcpath}/${distname}${extract.suffix}
destroot.post_args

livecheck.type  none
  }

You can see the PR here:
https://github.com/macports/macports-ports/pull/1131

The port apparently work, but I'd like to submit this port for review for
the following reasons:

  * I'm not sure we should install a whl file this way, invoking `pip`.

  * I had to redefine `master_sites` and `checksums` for each version of
Phyton.

  * I had to redefine the `extract` properties to skip extraction and just
copy the downloaded file into `${worksrcpath}`:

  extract.suffix  .whl
  extract.cmd cp
  extract.pre_args
  extract.post_args   ${worksrcpath}
  extract.mkdir   yes

  * Is there a better idiom to refer to the currently-installed `pip`?

  destroot.cmdpip-${python.branch}

  * Is there a better idiom to refer to the Python site packages directory
in the staging area?

  ${destroot}${python.pkgd}

Finally, I'm wondering whether this is a good idea at all.  I've grep-ed
the ports and I saw no other whl file installed this way.

If you can have a look at the PR I will appreciate any feedback and insight.

Cheers,
-- 
Enrico

Installing a Python wheel (whl) file using a port

[Enrico Maria Crisostomo]

Hi,

I’m trying to create a port for TensorFlow and I have already accumulated
quite a number of doubts in just a couple of hours I've been working on it.

First of all, I realised (late) that TensorFlow (and some of its
dependencies) is built as a wheel package (a .whl file), and as such it
gets uploaded to the Python Package Index: ​
https://pypi.python.org/pypi/tensorflow/1.4.1.  whl files are meant to be
installed with pip, so my current port file does the following:

  if {${name} ne ${subport}} {

build {
}

destroot.cmdpip-${python.branch}
destroot.pre_args
destroot.args   \
install \
--no-cache-dir \
--no-dependencies \
--target ${destroot}${python.pkgd} \
${worksrcpath}/${distname}${extract.suffix}
destroot.post_args

livecheck.type  none
  }

You can see the PR here:
https://github.com/macports/macports-ports/pull/1131

The port apparently work, but I'd like to submit this port for review for
the following reasons:

  * I'm not sure we should install a whl file this way, invoking `pip`.

  * I had to redefine `master_sites` and `checksums` for each version of
Phyton.

  * I had to redefine the `extract` properties to skip extraction and just
copy the downloaded file into `${worksrcpath}`:

  extract.suffix  .whl
  extract.cmd cp
  extract.pre_args
  extract.post_args   ${worksrcpath}
  extract.mkdir   yes

  * Is there a better idiom to refer to the currently-installed `pip`?

  destroot.cmdpip-${python.branch}

  * Is there a better idiom to refer to the Python site packages directory
in the staging area?

  ${destroot}${python.pkgd}

Finally, I'm wondering whether this is a good idea at all.  I've grep-ed
the ports and I saw no other whl file installed this way.

If you can have a look at the PR I will appreciate any feedback and insight.

Cheers,
-- 
Enrico