Bug#1059649: NTP does not keep accurate time on bookworm

[Rob Janssen]

Craig,

I'm sorry but I know absolutely nothing about that "systemd socket" mechanism.  
I am an old hat
that grew up with init and hates systemd.
Maybe I have inadvertently broken something, I only know what I have written 
before and that it does
not work for me without forcing the service to run as root instead of 
Debian-snmp.
What exactly is going on now is unclear to me.

Rob

On 2024-02-02 06:13, Craig Small wrote:
> On Fri, 2 Feb 2024 at 08:54, Rob Janssen  wrote:
>
> I am using systemd.
>
> Where are you seeing this error? The systemd socket is the thing that opens 
> up the socket, so shouldn't matter what the snmptrapd process is running as.
>
> When I reboot, I get this:
> $ sudo ss -unlp | grep 162
> UNCONN 0      0             0.0.0.0:162         0.0.0.0:* 
>    users:(("systemd",pid=1,fd=83))        
> UNCONN 0      0                [::]:162           [::]:*    
> users:(("systemd",pid=1,fd=84))        
> $ pgrep snmptrapd
>
> So systemd is listening to the socket and snmptrapd is not even running.
> After I send a trap:
> $ sudo ss -unlp | grep 162
> UNCONN 0      0             0.0.0.0:162         0.0.0.0:* 
>    users:(("snmptrapd",pid=7296,fd=3),("systemd",pid=1,fd=83))
> UNCONN 0      0                [::]:162           [::]:*    
> users:(("snmptrapd",pid=7296,fd=4),("systemd",pid=1,fd=84))
> $ pgrep snmptrapd
> 7296
>
> Maybe the issue is that this system does not have IPv6 enabled?  In 
> sysctl.conf it has:
> net.ipv6.conf.all.disable_ipv6 = 1
>
> I added that and rebooted, it worked fine.
>  
>
>
> That has effect on the default snmpd service as well, but I work around 
> that by modifying the listen
> port in the config file.
> The snmptrapd.service file has "udp:162" and "udp6:162" args in the 
> ExecStart but maybe these aren't used at all
> in this config?  In bullseye I think it worked out of the box.
>
> In a way, they're used.
>
> In a proper systemd socket activation, the service doesn't need to know or 
> define the ports it's listening to, they just get a bunch of file descriptors 
> and use those.
>
> So we could have a command line in the service file with no ports and it 
> looks like it works; but that is only because snmptrapd is sort-of looking 
> for UDP port 162.
> If you change the socket definition to another port, say 164 you get this:
> Feb 02 15:57:37 elmo snmptrapd[9123]: couldn't open udp:162 -- errno 13 
> ("Permission denied")
> Feb 02 15:57:37 elmo snmptrapd[9123]: couldn't open udp:162 -- errno 13 
> ("Permission denied")
>
> system opens port 164, a trap to port 164 starts snmptrapd which not only 
> listens to the systemd FD, but opens its own ports too running as a non-priv 
> user.
>  OK, so change the service definiton to listen to some port above 1024, it 
> should be ok because it can open that port and use the systemd FD. For some 
> reason
> it doesn't (as in it starts but there are no messages).
>
> What that means is the ports that are defined in the socket file need to 
> match the ports in the service file. snmptrapd is shipped with that:
> $ grep 162 debian/snmptrapd.s*
> debian/snmptrapd.service:ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162 
> udp6:162
> debian/snmptrapd.socket:ListenDatagram=0.0.0.0:162 
> debian/snmptrapd.socket:ListenDatagram=[::]:162
> debian/snmptrapd.socket:# ListenStream=0.0.0.0:162 
> debian/snmptrapd.socket:# ListenStream=[::]:162
>
> It would be more friendly when snmpd and snmptrapd degrade gracefully to 
> IPv4-only when there is no IPv6 on the system.
>
> I don't think its that. 
>
>  - Craig
>

Bug#1059649: NTP does not keep accurate time on bookworm

[Craig Small]

On Fri, 2 Feb 2024 at 08:54, Rob Janssen  wrote:

> I am using systemd.
>
Where are you seeing this error? The systemd socket is the thing that opens
up the socket, so shouldn't matter what the snmptrapd process is running as.

When I reboot, I get this:
$ sudo ss -unlp | grep 162
UNCONN 0  0 0.0.0.0:1620.0.0.0:*
 users:(("systemd",pid=1,fd=83))
UNCONN 0  0[::]:162   [::]:*
 users:(("systemd",pid=1,fd=84))
$ pgrep snmptrapd

So systemd is listening to the socket and snmptrapd is not even running.
After I send a trap:
$ sudo ss -unlp | grep 162
UNCONN 0  0 0.0.0.0:1620.0.0.0:*
 users:(("snmptrapd",pid=7296,fd=3),("systemd",pid=1,fd=83))
UNCONN 0  0[::]:162   [::]:*
 users:(("snmptrapd",pid=7296,fd=4),("systemd",pid=1,fd=84))
$ pgrep snmptrapd
7296

Maybe the issue is that this system does not have IPv6 enabled?  In
> sysctl.conf it has:
> net.ipv6.conf.all.disable_ipv6 = 1
>
I added that and rebooted, it worked fine.


>
> That has effect on the default snmpd service as well, but I work around
> that by modifying the listen
> port in the config file.
> The snmptrapd.service file has "udp:162" and "udp6:162" args in the
> ExecStart but maybe these aren't used at all
> in this config?  In bullseye I think it worked out of the box.
>
In a way, they're used.

In a proper systemd socket activation, the service doesn't need to know or
define the ports it's listening to, they just get a bunch of file
descriptors and use those.

So we could have a command line in the service file with no ports and it
looks like it works; but that is only because snmptrapd is sort-of looking
for UDP port 162.
If you change the socket definition to another port, say 164 you get this:
Feb 02 15:57:37 elmo snmptrapd[9123]: couldn't open udp:162 -- errno 13
("Permission denied")
Feb 02 15:57:37 elmo snmptrapd[9123]: couldn't open udp:162 -- errno 13
("Permission denied")

system opens port 164, a trap to port 164 starts snmptrapd which not only
listens to the systemd FD, but opens its own ports too running as a
non-priv user.
 OK, so change the service definiton to listen to some port above 1024, it
should be ok because it can open that port and use the systemd FD. For some
reason
it doesn't (as in it starts but there are no messages).

What that means is the ports that are defined in the socket file need to
match the ports in the service file. snmptrapd is shipped with that:
$ grep 162 debian/snmptrapd.s*
debian/snmptrapd.service:ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162
udp6:162
debian/snmptrapd.socket:ListenDatagram=0.0.0.0:162
debian/snmptrapd.socket:ListenDatagram=[::]:162
debian/snmptrapd.socket:# ListenStream=0.0.0.0:162
debian/snmptrapd.socket:# ListenStream=[::]:162

It would be more friendly when snmpd and snmptrapd degrade gracefully to
> IPv4-only when there is no IPv6 on the system.
>
I don't think its that.

 - Craig

Bug#1059649: NTP does not keep accurate time on bookworm

[Rob Janssen]

On 2024-02-01 22:09, Craig Small wrote:
> On Sat, 30 Dec 2023 at 06:15, Rob Janssen  wrote:
>
> After the upgrade, the snmptrapd service no longer starts.
> The error message is: couldn't open udp:162 -- errno 13 ("Permission 
> denied")
>
> Could you tell me how you start snmptrapd?
> There are two ways:
> The default systemd way. The socket is created with a snmptrap.socket and 
> then passed onto the snmpdtrap server. I have tested this just now and it 
> works fine with 5.9.4
> The init way. In this case snmptrapd starts as root and changes to 
> Debian-snmp after binding to the sockets.
I am using systemd.

>  
>
> To cover possible upgrade issues I removed and purged the snmptrapd 
> package
> and re-installed it, but the new install suffered the same issue.
>
> What, specifically, are you doing to see this message? Running what you see 
> in the service file on the command line won't work and will give this result.
At first I just upgraded a bullseye system where everything was working, to 
bookworm, and it failed to start snmptrapd.
Then I did the uninstall/re-install of snmptrapd and it still failed.

Maybe the issue is that this system does not have IPv6 enabled?  In sysctl.conf 
it has:
net.ipv6.conf.all.disable_ipv6 = 1

That has effect on the default snmpd service as well, but I work around that by 
modifying the listen
port in the config file.
The snmptrapd.service file has "udp:162" and "udp6:162" args in the ExecStart 
but maybe these aren't used at all
in this config?  In bullseye I think it worked out of the box.

It would be more friendly when snmpd and snmptrapd degrade gracefully to 
IPv4-only when there is no IPv6 on the system.

Rob

Bug#1059649: NTP does not keep accurate time on bookworm

[Craig Small]

On Sat, 30 Dec 2023 at 06:15, Rob Janssen  wrote:

> After the upgrade, the snmptrapd service no longer starts.
> The error message is: couldn't open udp:162 -- errno 13 ("Permission
> denied")
>
Could you tell me how you start snmptrapd?
There are two ways:
The default systemd way. The socket is created with a snmptrap.socket and
then passed onto the snmpdtrap server. I have tested this just now and it
works fine with 5.9.4
The init way. In this case snmptrapd starts as root and changes to
Debian-snmp after binding to the sockets.

It appears that this is caused by the service being started as user
> Debian-snmp
>
and thus not being able to open privileged ports.
>
As you can see, it doesn't need to normally.


> To cover possible upgrade issues I removed and purged the snmptrapd package
> and re-installed it, but the new install suffered the same issue.
>
What, specifically, are you doing to see this message? Running what you see
in the service file on the command line won't work and will give this
result.

 - Craig

Bug#1059649: NTP does not keep accurate time on bookworm

[Rob Janssen]

Package: snmptrapd
Version: 5.9.3+dfsg-2

I upgraded a system from bullseye to bookworm.
It had snmptrapd installed.
Before the upgrade, all was OK.

After the upgrade, the snmptrapd service no longer starts.
The error message is: couldn't open udp:162 -- errno 13 ("Permission denied")

It appears that this is caused by the service being started as user Debian-snmp
and thus not being able to open privileged ports.
I could work around it by editing the systemd service file to run snmptrapd as
User=root but this is likely not what was intended.

To cover possible upgrade issues I removed and purged the snmptrapd package
and re-installed it, but the new install suffered the same issue.