I believe I have reproduced this crash and it's related to syslog. I
have run into this before, probably with the amplified noise the crashes
were happening more often.
Reproducing it I can cause the crash if I stop/restart the syslog
service on the host at the right time.
Workaround: Disabled syslog and just set the log to file for now.
On 3/16/24 09:00, Robert Blayzor via RPKI wrote:
I have been running into stability issues since the release of 0.13 and
it's been getting worse. I'm having routinator servers crash multiple
times a day now and I just had two servers crash basically at the same
time, so it's getting some outside information that it doesn't like.
The servers are running FreeBSD 14.0 and Rust 1.76 is installed on the
system. This has been happening since 0.13 and is now still happening in
0.13.2, but now even more frequently it appears.
Unfortunately when routinator core dumps, it's not leaving a core file
behind, I'm just seeing:
pid 2982 (routinator), jid 0, uid 245: exited on signal 6 (no core dump
- bad address)
pid 93548 (routinator), jid 0, uid 245: exited on signal 6 (no core dump
- bad address)
pid 75744 (routinator), jid 0, uid 245: exited on signal 6 (no core dump
- bad address)
Literally the last noise in the logs on each server. This level of
"noise" and failure looks like something is broken....
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/EA021982D2C311EE98D88865775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/6667703ED47D11EEAA6502BE775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/86D788C8D48111EE80528CC4775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/EFBE9CB6D16911EEA51004A8775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/8668158AD17311EEB69D94B8775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/B7C19E14D2CF11EEBB636E78775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/B8A5AEA0D48311EE85B40BC8775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/0B3122E8D17911EE951057C1775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/385F1D8AD2D011EE9BBD5C79775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 routinator[75742]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/ED2CF494D2C611EEAA8E776A775412E6.roa: certificate has
expired.
Mar 16 12:43:43 rs0 kernel: pid 75744 (routinator), jid 0, uid 245:
exited on signal 6 (no core dump - bad address)
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/11A8344ED2C411EEB833F865775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/3E62BCE0D2C711EEAEF6EA6A775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/C5EC9C26D2CC11EE8E67C973775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/8C32E506D2CE11EEA31CD576775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/F8491F0ED2C811EE8D92D66D775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/1543656CD2C711EE9822A16A775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/2040974A0F1911EE83AEF94F4AD9E6FC.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/A940268AD15D11EE8290C094775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 routinator[95204]:
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/5812BEAED20111EE9D9842A6775412E6.roa: certificate has
expired.
Mar 16 08:48:48 rs0 kernel: pid 95206 (routinator), jid 0, uid 245:
exited on signal 6 (no core dump - bad address)
--
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP: https://pgp.inoc.net/rblayzor/
--
RPKI mailing list
RPKI@lists.nlnetlabs.nl
https://lists.nlnetlabs.nl/mailman/listinfo/rpki
