Re: How can I make unbound's accepting of incoming network connections in application firewall in Catalina 'stick'?

[Gerben Wierda via macports-users]

Sorry, no go.

I found a solution on superuser with 0 votes ;-). 
https://superuser.com/a/940696/582447

It turns out that I just had to turn the firewall off and on again. It might 
have been a necessary last step to make it regenerate something, but after that 
step, a reboot (or just port unload/load cycle) will just allow unbound to 
startup and accept incoming connections without further panels

Note, that on my active production server I run Murus (PF configurator) and 
Vallum (configurator for the application-level firewall), which, though 
somewhat hard to work with sometimes, work well in configuring this.

Gerben Wierda (LinkedIn )
R Enterprise Architecture  (main site)
Book: Chess and the Art of Enterprise Architecture 
Book: Mastering ArchiMate 

> On 15 Mar 2021, at 02:17, Steven Smith  wrote:
> 
> Just turn off your firewall! 
> 
> Seriously, the macOS firewall is an Application firewall. If that suits your 
> risk profile, you can control it through the command line:
> 
> /usr/libexec/ApplicationFirewall/socketfilterfw -h
> 
> Port- and packet-based filtering is handled by pfctl, and that’s a lot more 
> flexible than the macOS application firewall.
> 
>> On Mar 14, 2021, at 20:55, Gerben Wierda via macports-users 
>>  wrote:
>> 
>> I am running an extensive MacPorts (with postfix, dovecot, nginx, minion, 
>> etc.) on my macOS Server, which is still running macOS Mojave.
>> 
>> On one of the other Macs, running macOS Catalina, I run a backup unbound 
>> caching nameserver. This also offers me a way to do some minimal testing of 
>> the MacPorts setup on a more recent version of macOS (as a preparation for 
>> upgrading the Mojave system when Apple stops supporting it)
>> 
>> The unbound on macOS Catalina runs fine, except for one thing. After a 
>> reboot, unbound will not accept incoming connections until I have logged in 
>> an answer the application firewalls’ question:
>> 
>> Do you want the application “unbound” to accept incoming network connections?
>> Clicking Deny may limit the application’s behaviour. This setting can be 
>> changed in the Firewall pane of Security & Privacy preferences.
>> 
>> I can answer yes, check the entry in the application firewall (set to yes, 
>> accept, even before I allow it through the panel). But even if it is set to 
>> accept incoming connections, after a reboot I need to log in and answer 
>> again via the GUI before it accepts. Setting this in the Application 
>> firewall doesn’t ’stick’ for some reason.
>> 
>> This is not acceptable behaviour if I ever upgrade my Mojave Server, as that 
>> one must be able to do unsupervised reboots/running without any login.
>> 
>> Is there something special in Catalina I must do? Or is this expected 
>> behaviour?
>> 
>> Thanks,
>> 
>> Gerben Wierda (LinkedIn )
>> R Enterprise Architecture  (main site)
>> Book: Chess and the Art of Enterprise Architecture 
>> 
>> Book: Mastering ArchiMate 
>> 

Re: How can I make unbound's accepting of incoming network connections in application firewall in Catalina 'stick'?

[Steven Smith]

Just turn off your firewall! 

Seriously, the macOS firewall is an Application firewall. If that suits your 
risk profile, you can control it through the command line:

/usr/libexec/ApplicationFirewall/socketfilterfw -h

Port- and packet-based filtering is handled by pfctl, and that’s a lot more 
flexible than the macOS application firewall.

> On Mar 14, 2021, at 20:55, Gerben Wierda via macports-users 
>  wrote:
> I am running an extensive MacPorts (with postfix, dovecot, nginx, minion, 
> etc.) on my macOS Server, which is still running macOS Mojave.
> 
> On one of the other Macs, running macOS Catalina, I run a backup unbound 
> caching nameserver. This also offers me a way to do some minimal testing of 
> the MacPorts setup on a more recent version of macOS (as a preparation for 
> upgrading the Mojave system when Apple stops supporting it)
> 
> The unbound on macOS Catalina runs fine, except for one thing. After a 
> reboot, unbound will not accept incoming connections until I have logged in 
> an answer the application firewalls’ question:
> 
> Do you want the application “unbound” to accept incoming network connections?
> Clicking Deny may limit the application’s behaviour. This setting can be 
> changed in the Firewall pane of Security & Privacy preferences.
> 
> I can answer yes, check the entry in the application firewall (set to yes, 
> accept, even before I allow it through the panel). But even if it is set to 
> accept incoming connections, after a reboot I need to log in and answer again 
> via the GUI before it accepts. Setting this in the Application firewall 
> doesn’t ’stick’ for some reason.
> 
> This is not acceptable behaviour if I ever upgrade my Mojave Server, as that 
> one must be able to do unsupervised reboots/running without any login.
> 
> Is there something special in Catalina I must do? Or is this expected 
> behaviour?
> 
> Thanks,
> 
> Gerben Wierda (LinkedIn)
> R Enterprise Architecture (main site)
> Book: Chess and the Art of Enterprise Architecture
> Book: Mastering ArchiMate


smime.p7s
Description: S/MIME cryptographic signature

Re: How can I make unbound's accepting of incoming network connections in application firewall in Catalina 'stick'?

[Gerben Wierda via macports-users]

It seems to be a code signing issue for /opt/local/sbin/unbound, but I haven’t 
found out how to get rid of it and MacPorts doesn’t handle it itself (i.e. 
forces the app to end into a good state or warn why it can’t do it).

Gerben Wierda (LinkedIn )
R Enterprise Architecture  (main site)
Book: Chess and the Art of Enterprise Architecture 
Book: Mastering ArchiMate 

> On 15 Mar 2021, at 01:55, Gerben Wierda via macports-users 
>  wrote:
> 
> I am running an extensive MacPorts (with postfix, dovecot, nginx, minion, 
> etc.) on my macOS Server, which is still running macOS Mojave.
> 
> On one of the other Macs, running macOS Catalina, I run a backup unbound 
> caching nameserver. This also offers me a way to do some minimal testing of 
> the MacPorts setup on a more recent version of macOS (as a preparation for 
> upgrading the Mojave system when Apple stops supporting it)
> 
> The unbound on macOS Catalina runs fine, except for one thing. After a 
> reboot, unbound will not accept incoming connections until I have logged in 
> an answer the application firewalls’ question:
> 
> Do you want the application “unbound” to accept incoming network connections?
> Clicking Deny may limit the application’s behaviour. This setting can be 
> changed in the Firewall pane of Security & Privacy preferences.
> 
> I can answer yes, check the entry in the application firewall (set to yes, 
> accept, even before I allow it through the panel). But even if it is set to 
> accept incoming connections, after a reboot I need to log in and answer again 
> via the GUI before it accepts. Setting this in the Application firewall 
> doesn’t ’stick’ for some reason.
> 
> This is not acceptable behaviour if I ever upgrade my Mojave Server, as that 
> one must be able to do unsupervised reboots/running without any login.
> 
> Is there something special in Catalina I must do? Or is this expected 
> behaviour?
> 
> Thanks,
> 
> Gerben Wierda (LinkedIn )
> R Enterprise Architecture  (main site)
> Book: Chess and the Art of Enterprise Architecture 
> 
> Book: Mastering ArchiMate 
> 

How can I make unbound's accepting of incoming network connections in application firewall in Catalina 'stick'?

[Gerben Wierda via macports-users]

I am running an extensive MacPorts (with postfix, dovecot, nginx, minion, etc.) 
on my macOS Server, which is still running macOS Mojave.

On one of the other Macs, running macOS Catalina, I run a backup unbound 
caching nameserver. This also offers me a way to do some minimal testing of the 
MacPorts setup on a more recent version of macOS (as a preparation for 
upgrading the Mojave system when Apple stops supporting it)

The unbound on macOS Catalina runs fine, except for one thing. After a reboot, 
unbound will not accept incoming connections until I have logged in an answer 
the application firewalls’ question:

Do you want the application “unbound” to accept incoming network connections?
Clicking Deny may limit the application’s behaviour. This setting can be 
changed in the Firewall pane of Security & Privacy preferences.

I can answer yes, check the entry in the application firewall (set to yes, 
accept, even before I allow it through the panel). But even if it is set to 
accept incoming connections, after a reboot I need to log in and answer again 
via the GUI before it accepts. Setting this in the Application firewall doesn’t 
’stick’ for some reason.

This is not acceptable behaviour if I ever upgrade my Mojave Server, as that 
one must be able to do unsupervised reboots/running without any login.

Is there something special in Catalina I must do? Or is this expected behaviour?

Thanks,

Gerben Wierda (LinkedIn )
R Enterprise Architecture  (main site)
Book: Chess and the Art of Enterprise Architecture 
Book: Mastering ArchiMate 

Re: Using RAM instead of disk for build servers (was: Re: Build servers offline due to failed SSD)

[Balthasar Indermuehle]

Hi Ryan,

thanks for your detailed response. I hadn't thought of some of the build
intricacies you mention. Let alone the upcoming silicon change and phasing
out of x86. Sounds like your approach is a good balance for longevity,
performance, and cost.

Cheers

Balthasar

On Mon, 15 Mar 2021 at 09:38, Ryan Schmidt  wrote:

> On Mar 14, 2021, at 06:11, Balthasar Indermuehle wrote:
>
> > I used to run mac servers in what now can only be described as the days
> of yore... when a 32GB RAM bank cost a lot more than a (spinning) disk -
> and those were expensive then too. SSDs were not here yet. I haven't
> checked pricing lately, but I'd think you could put 256GB of RAM into a
> server for probably about the same as a 1TB SSD, and that would offer
> plenty of build space when used as a RAM drive. And that space will not
> degrade over time (unlike the SSD). In terms of longevity, for a machine
> with such a singularly targeted use case, I'd seriously consider taking the
> expense now, and have a server that lives for another decade.
>
> Some pricing details:
>
> OWC sells 96 GB Xserve RAM for $270 and 48 GB for $160. 96 + 96 + 48 would
> be 240 GB for $700.
>
> Meanwhile, the 500 GB SSDs I've been putting in cost about $65 each. I've
> already put in three and still need one more to get rid of the hard drives
> in the fourth server, though I may get a 1 TB SSD for $120 to have some
> extra room.
>
> Note that the way our build system (using our mpbb build script) works is
> that all (or most) ports that exist are kept installed but deactivated.
> When a build request comes in, we first activate all of that port's
> dependencies, then we build and install and activate that port, then we
> deactivate all ports. "Activate" means extract the tbz2 file to disk and
> note it in the registry. "Deactivate" means delete the extracted files from
> disk and note it in the registry. So even if we move all port building to a
> RAM disk, which will undeniably be faster and reduce wear on the disk, it
> will not eliminate it completely, not by a long shot. Some ports have
> hundreds of dependencies. Activating and deactivating ports is a
> considerable portion of what our build machines spend their day doing.
>
> If we wanted to move that from SSD to RAM disk as well, that would mean
> putting MacPorts itself onto the RAM disk. We wouldn't have room on the RAM
> disk to keep all ports installed, so it would mean not keeping any ports
> installed, and instead installing them on demand and then uninstalling them
> (and maybe we would need to budget even more RAM for the RAM disk to
> accommodate both space needed for MacPorts and for the dependencies and for
> the build). That means downloading and verifying each port's tbz2 file from
> the packages web server for every port build. Even though we do have a
> local packages server, so that traffic would not have to go over the
> Internet, the server uses a hard disk based RAID which is not the fastest
> in the world, so this would cause additional delays, not to mention
> additional wear and tear on the RAID's disks.
>
> As far as longevity, the previous set of 3 500 GB SSDs I bought for these
> servers in 2016 lasted 4-5 years. They were rated for 150 TBW (terabytes
> written) and actually endured around 450 TBW by the time they failed, or 3
> times as long as they were expected to last. The new SSDs are rated for 300
> TBW, and if they also last 3 times longer than that, then they might last
> 8-10 years, by which time we might have completely abandoned Intel-based
> Macs and be totally switched over to Apple Silicon hardware and will have
> no use for the Xserves anymore.
>
>
>

Re: Using RAM instead of disk for build servers (was: Re: Build servers offline due to failed SSD)

[Ryan Schmidt]

On Mar 14, 2021, at 06:11, Balthasar Indermuehle wrote:

> I used to run mac servers in what now can only be described as the days of 
> yore... when a 32GB RAM bank cost a lot more than a (spinning) disk - and 
> those were expensive then too. SSDs were not here yet. I haven't checked 
> pricing lately, but I'd think you could put 256GB of RAM into a server for 
> probably about the same as a 1TB SSD, and that would offer plenty of build 
> space when used as a RAM drive. And that space will not degrade over time 
> (unlike the SSD). In terms of longevity, for a machine with such a singularly 
> targeted use case, I'd seriously consider taking the expense now, and have a 
> server that lives for another decade.

Some pricing details:

OWC sells 96 GB Xserve RAM for $270 and 48 GB for $160. 96 + 96 + 48 would be 
240 GB for $700.

Meanwhile, the 500 GB SSDs I've been putting in cost about $65 each. I've 
already put in three and still need one more to get rid of the hard drives in 
the fourth server, though I may get a 1 TB SSD for $120 to have some extra room.

Note that the way our build system (using our mpbb build script) works is that 
all (or most) ports that exist are kept installed but deactivated. When a build 
request comes in, we first activate all of that port's dependencies, then we 
build and install and activate that port, then we deactivate all ports. 
"Activate" means extract the tbz2 file to disk and note it in the registry. 
"Deactivate" means delete the extracted files from disk and note it in the 
registry. So even if we move all port building to a RAM disk, which will 
undeniably be faster and reduce wear on the disk, it will not eliminate it 
completely, not by a long shot. Some ports have hundreds of dependencies. 
Activating and deactivating ports is a considerable portion of what our build 
machines spend their day doing.

If we wanted to move that from SSD to RAM disk as well, that would mean putting 
MacPorts itself onto the RAM disk. We wouldn't have room on the RAM disk to 
keep all ports installed, so it would mean not keeping any ports installed, and 
instead installing them on demand and then uninstalling them (and maybe we 
would need to budget even more RAM for the RAM disk to accommodate both space 
needed for MacPorts and for the dependencies and for the build). That means 
downloading and verifying each port's tbz2 file from the packages web server 
for every port build. Even though we do have a local packages server, so that 
traffic would not have to go over the Internet, the server uses a hard disk 
based RAID which is not the fastest in the world, so this would cause 
additional delays, not to mention additional wear and tear on the RAID's disks.

As far as longevity, the previous set of 3 500 GB SSDs I bought for these 
servers in 2016 lasted 4-5 years. They were rated for 150 TBW (terabytes 
written) and actually endured around 450 TBW by the time they failed, or 3 
times as long as they were expected to last. The new SSDs are rated for 300 
TBW, and if they also last 3 times longer than that, then they might last 8-10 
years, by which time we might have completely abandoned Intel-based Macs and be 
totally switched over to Apple Silicon hardware and will have no use for the 
Xserves anymore.

Re: Build servers offline due to failed SSD

[Balthasar Indermuehle]

I used to run mac servers in what now can only be described as the days of
yore... when a 32GB RAM bank cost a lot more than a (spinning) disk - and
those were expensive then too. SSDs were not here yet. I haven't checked
pricing lately, but I'd think you could put 256GB of RAM into a server for
probably about the same as a 1TB SSD, and that would offer plenty of build
space when used as a RAM drive. And that space will not degrade over time
(unlike the SSD). In terms of longevity, for a machine with such a
singularly targeted use case, I'd seriously consider taking the expense
now, and have a server that lives for another decade.


Dr Balthasar Indermühle
Inside Systems Pty Ltd
5007/101 Bathurst Street
Sydney NSW 2000, Australia
t: +61 (0)405 988 500



On Sun, 14 Mar 2021 at 21:04, Ryan Schmidt  wrote:

> There was some additional downtime in the last few days but the
> buildmaster now has a permanent home on a new SSD and is faster than ever.
> Builds that could not be scheduled during recent downtime have been
> rescheduled and are in progress.
>
>
> On Mar 14, 2021, at 04:02, Vincent Habchi wrote:
>
> > Wouldn’t it make sense to use some sort of RAM caching to speed up
> builds instead of SSD? What’s the point of using a permanent storage device
> for something that is bound to be erased in a very short time?
>
> RAM would be faster than SSD but also a lot more expensive. Certainly I
> know or can figure out how to create a RAM disk, and certainly we could
> tell MacPorts to store the build directory there. But if we ran out of
> space on the RAM disk during a build, the build would fail. Some builds
> need a lot of disk space -- I've seen ports that use 20GB of disk space to
> build. Instead of buying 20GB or more of additional RAM per VM, I've chosen
> to buy 90GB of SSD per VM.
>
> If you're suggesting that we should just set aside 1-2GB of RAM for build
> files and use the SSD if we need more space than that, then I don't know
> how to set that up.
>
> Note that macOS already caches disk files in RAM if there is any free RAM.
>
>

Re: Build servers offline due to failed SSD

[Ryan Schmidt]

There was some additional downtime in the last few days but the buildmaster now 
has a permanent home on a new SSD and is faster than ever. Builds that could 
not be scheduled during recent downtime have been rescheduled and are in 
progress.


On Mar 14, 2021, at 04:02, Vincent Habchi wrote:

> Wouldn’t it make sense to use some sort of RAM caching to speed up builds 
> instead of SSD? What’s the point of using a permanent storage device for 
> something that is bound to be erased in a very short time?

RAM would be faster than SSD but also a lot more expensive. Certainly I know or 
can figure out how to create a RAM disk, and certainly we could tell MacPorts 
to store the build directory there. But if we ran out of space on the RAM disk 
during a build, the build would fail. Some builds need a lot of disk space -- 
I've seen ports that use 20GB of disk space to build. Instead of buying 20GB or 
more of additional RAM per VM, I've chosen to buy 90GB of SSD per VM.

If you're suggesting that we should just set aside 1-2GB of RAM for build files 
and use the SSD if we need more space than that, then I don't know how to set 
that up.

Note that macOS already caches disk files in RAM if there is any free RAM.

Re: Build servers offline due to failed SSD

[Vincent Habchi]

Hi,

Wouldn’t it make sense to use some sort of RAM caching to speed up builds 
instead of SSD? What’s the point of using a permanent storage device for 
something that is bound to be erased in a very short time?

Maybe I’m way off base, though.

V.