Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Yes! At long last 231-6 seems to have cleared the systemd issue I was observing on Raspberry Pi: -- root@pi ~ # systemctl status ● pi State: running Jobs: 0 queued Failed: 0 units -- Many thanks to the people who spent time on this! Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 30 August 2016 at 13:43, Felipe Satelerwrote: > On 30 August 2016 at 13:32, Pete Batard wrote: >> On 2016.08.30 17:30, Felipe Sateler wrote: >>> >>> I managed to get the config from a jessie rpi by loading the 'configs' >>> module (sudo modprobe configs). After that the config is found on >>> /proc/config.gz >> >> >> Yeah, I just discovered the same after I replied. >> This is also documented on the official rpi github repo: >> https://github.com/raspberrypi/firmware/issues/442 >> >> I am therefore attaching the config.gz for my current kernel. > > OK. So there is CONFIG_SECCOMP but no CONFIG_SECCOMP_FILTER. The first > leads to having Seccomp: 0 in the status file, but then no filter > support means that any actual program systemd tries to load fails. > BTW, it might be a good idea to file an issue to the raspbian people > so that they enable CONFIG_SECCOMP_FILTER, or disable SECCOMP > entirely. > > I think I will be able to reproduce the issue now. It took a while to reproduce (it was not obvious how to produce a kernel with seccomp but no filtering), but I managed to reproduce. Patch is up for review upstream: https://github.com/systemd/systemd/pull/4087 -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Okay. I have logged issue #651 (https://github.com/raspberrypi/firmware/issues/651) with the rpi team so that they try to sort their SECCOMP configuration in future kernels. Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 30 August 2016 at 13:32, Pete Batardwrote: > On 2016.08.30 17:30, Felipe Sateler wrote: >> >> I managed to get the config from a jessie rpi by loading the 'configs' >> module (sudo modprobe configs). After that the config is found on >> /proc/config.gz > > > Yeah, I just discovered the same after I replied. > This is also documented on the official rpi github repo: > https://github.com/raspberrypi/firmware/issues/442 > > I am therefore attaching the config.gz for my current kernel. OK. So there is CONFIG_SECCOMP but no CONFIG_SECCOMP_FILTER. The first leads to having Seccomp: 0 in the status file, but then no filter support means that any actual program systemd tries to load fails. BTW, it might be a good idea to file an issue to the raspbian people so that they enable CONFIG_SECCOMP_FILTER, or disable SECCOMP entirely. I think I will be able to reproduce the issue now. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 2016.08.30 17:30, Felipe Sateler wrote: I managed to get the config from a jessie rpi by loading the 'configs' module (sudo modprobe configs). After that the config is found on /proc/config.gz Yeah, I just discovered the same after I replied. This is also documented on the official rpi github repo: https://github.com/raspberrypi/firmware/issues/442 I am therefore attaching the config.gz for my current kernel. Regards, /Pete config.gz Description: application/gzip
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 30 August 2016 at 13:26, Pete Batardwrote: > Thanks Felipe. > > I guess with the new detection process, the resurgence of the issue is > starting to make sense now. > > For the record, I am using one of the latest official Raspberry Pi kernels > from https://github.com/raspberrypi/firmware (which I get indirectly through > the https://github.com/Hexxeh/rpi-firmware repo and its rpi-update script at > https://github.com/Hexxeh/rpi-update). Ah, I thought you were using the debian kernels. > > Unfortunately, these kernels do not provide a config.gz in /proc, so I can't > tell you precisely what compilation options were used in case it matters. > But I'll be happy to run more tests on my machine as needed. I managed to get the config from a jessie rpi by loading the 'configs' module (sudo modprobe configs). After that the config is found on /proc/config.gz -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Thanks Felipe. I guess with the new detection process, the resurgence of the issue is starting to make sense now. For the record, I am using one of the latest official Raspberry Pi kernels from https://github.com/raspberrypi/firmware (which I get indirectly through the https://github.com/Hexxeh/rpi-firmware repo and its rpi-update script at https://github.com/Hexxeh/rpi-update). Unfortunately, these kernels do not provide a config.gz in /proc, so I can't tell you precisely what compilation options were used in case it matters. But I'll be happy to run more tests on my machine as needed. Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 29 August 2016 at 13:13, Pete Batardwrote: > Well, adding 'systemd.log_level=debug' to my kernel options made the system > enter emergency mode... > > And I can't get the very beginning of the log then (even though I also tried > to increase the buffer size by adding 'log_buf_len=19'), so I'm attaching > the best I can provide in terms of verbose log, with the first 4.5 seconds > missing. > > Also attached is the status (from the system running without > 'systemd.log_level=debug', since I don't really want to keep it in emergency > mode). Thanks. This status file reports that Seccomp: 0. But your kernel doesn't really have seccomp, so that is weird. > But again, as far as I am concerned, one of the changes that intervened > between 231-4 and 231-5 seems to have reverted the SECCOMP issue, so I would > invite you to closely review what was changed between these 2 revisions, as > maybe you'll find a clue there. Well, to work around this problem we (temporarily) disabled all seccomp security filters. With 231-5 we replaced that in favor of runtime detection of seccomp availability. Clearly the runtime detection is insufficient, but we really need to fix the detection: disabling security features is a bug too! I'm trying to install a vm using the debian kernel. This should make it easier to test stuff. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 29 August 2016 at 12:10, Pete Batardwrote: > Please find my boot log. Note that I've tried commenting out > 'MemoryDenyWriteExecute=yes' in the various .service config files, but it > didn't help. > > If you need anything else, please let me know. This log is not verbose. Could you boot with kernel argument systemd.log_level=debug ? Also, on the arm machine could you attach the contents of /proc/self/status ? -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Please find my boot log. Note that I've tried commenting out 'MemoryDenyWriteExecute=yes' in the various .service config files, but it didn't help. If you need anything else, please let me know. Regards, /Pete [0.00] Booting Linux on physical CPU 0xf00 [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.4.19-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611) ) #906 SMP Tue Aug 23 15:53:06 BST 2016 [0.00] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d [0.00] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [0.00] Machine model: Raspberry Pi 2 Model B Rev 1.1 [0.00] cma: Reserved 8 MiB at 0x3a80 [0.00] Memory policy: Data cache writealloc [0.00] On node 0 totalpages: 241664 [0.00] free_area_init_node: node 0, pgdat 808c2f40, node_mem_map b9fa6000 [0.00] Normal zone: 2124 pages used for memmap [0.00] Normal zone: 0 pages reserved [0.00] Normal zone: 241664 pages, LIFO batch:31 [0.00] [bcm2709_smp_init_cpus] enter (9520->f3003010) [0.00] [bcm2709_smp_init_cpus] ncores=4 [0.00] PERCPU: Embedded 13 pages/cpu @b9f62000 s22592 r8192 d22464 u53248 [0.00] pcpu-alloc: s22592 r8192 d22464 u53248 alloc=13*4096 [0.00] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0.00] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 239540 [0.00] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1680 bcm2708_fb.fbheight=1050 bcm2709.boardrev=0x2a01041 bcm2709.serial=0x66b48257 smsc95xx.macaddr=B8:27:EB:B4:82:57 bcm2708_fb.fbswap=1 bcm2709.uart_clock=4800 bcm2709.disk_led_gpio=47 bcm2709.disk_led_active_low=0 vc_mem.mem_base=0x3dc0 vc_mem.mem_size=0x3f00 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootwait net.ifnames=1 [0.00] PID hash table entries: 4096 (order: 2, 16384 bytes) [0.00] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) [0.00] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) [0.00] Memory: 939076K/966656K available (6348K kernel code, 432K rwdata, 1716K rodata, 476K init, 764K bss, 19388K reserved, 8192K cma-reserved) [0.00] Virtual kernel memory layout: vector : 0x - 0x1000 ( 4 kB) fixmap : 0xffc0 - 0xfff0 (3072 kB) vmalloc : 0xbb80 - 0xff80 (1088 MB) lowmem : 0x8000 - 0xbb00 ( 944 MB) modules : 0x7f00 - 0x8000 ( 16 MB) .text : 0x80008000 - 0x807e8510 (8066 kB) .init : 0x807e9000 - 0x8086 ( 476 kB) .data : 0x8086 - 0x808cc250 ( 433 kB) .bss : 0x808cf000 - 0x8098e1ec ( 765 kB) [0.00] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [0.00] Hierarchical RCU implementation. [0.00] Build-time adjustment of leaf fanout to 32. [0.00] NR_IRQS:16 nr_irqs:16 16 [0.00] Architected cp15 timer(s) running at 19.20MHz (phys). [0.00] clocksource: arch_sys_counter: mask: 0xff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns [0.07] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns [0.21] Switching to timer-based delay loop, resolution 52ns [0.000249] Console: colour dummy device 80x30 [0.001031] console [tty1] enabled [0.001069] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000) [0.001121] pid_max: default: 32768 minimum: 301 [0.001408] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes) [0.001443] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes) [0.002303] Disabling cpuset control group subsystem [0.002365] Initializing cgroup subsys io [0.002408] Initializing cgroup subsys memory [0.002466] Initializing cgroup subsys devices [0.002504] Initializing cgroup subsys freezer [0.002536] Initializing cgroup subsys net_cls [0.002603] CPU: Testing write buffer coherency: ok [0.002685] ftrace: allocating 21217 entries in 63 pages [0.037656] CPU0: update cpu_capacity 1024 [0.037712] CPU0: thread -1, cpu 0, socket 15, mpidr 8f00 [0.037739] [bcm2709_smp_prepare_cpus] enter [0.037858] Setting up static identity map for 0x8240 - 0x8274 [0.039516] [bcm2709_boot_secondary] cpu:1 started (0) 16 [0.039854] [bcm2709_secondary_init] enter cpu:1 [0.039898] CPU1: update cpu_capacity 1024 [0.039904] CPU1: thread -1, cpu 1, socket 15, mpidr 8f01 [0.040305] [bcm2709_boot_secondary] cpu:2 started (0) 17 [
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Control: tags -1 moreinfo Control: found -1 231-5 On 28 August 2016 at 17:46, Pete Batardwrote: > On 2016.08.28 15:37, Felipe Sateler wrote: >> >> Could you try 231-5 from unstable? > > > Oops, I mean 231-5 broke it. I'm seeing the issue back in 231-5 and not > 231-4 as I mentioned earlier. > > 231-4 seemed to be okay, and it's only when I upgraded to 231-5 that the > problem reappeared. Hmm. Strange. I cannot reproduce on a vm with seccomp disabled. Could you please attach a verbose log? -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 2016.08.28 15:37, Felipe Sateler wrote: Could you try 231-5 from unstable? Oops, I mean 231-5 broke it. I'm seeing the issue back in 231-5 and not 231-4 as I mentioned earlier. 231-4 seemed to be okay, and it's only when I upgraded to 231-5 that the problem reappeared. Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 27 August 2016 at 20:20, Pete Batardwrote: > ...and 231-4 broke the whole thing down again. :( Could you try 231-5 from unstable? -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
...and 231-4 broke the whole thing down again. :( Same error: root@pi ~ # systemctl status systemd-journald.service ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled) Active: failed (Result: start-limit-hit) since Mon 2016-07-25 20:49:50 IST; 1 months 2 days ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 200 (code=exited, status=228/SECCOMP) 231-3 was actually fine as far as I'm concerned (but I wasn't able to actually test it until a few days ago). Did you inadvertently revert one of the fixes that were applied in 231-3? Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 14 août 2016 23:38, Pete Batardwrote: > Didn't fix it for me... :( Same for me, still broken. Christian
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Didn't fix it for me... :( I went to an 'apt-get --reinstall install' of every single package that is listed under 'Description' above (expect -udeb and -dev), and got the 231-2 version alright, but the issue remains exactly the same, even after a full reboot. I should point out that I did get the following warning when updating udev, which I gather is due to using 'rpi-update' (https://github.com/Hexxeh/rpi-update) to run a recent kernel: - # uname -a Linux pi 4.4.17-v7+ #901 SMP Fri Aug 12 17:57:27 BST 2016 armv7l GNU/Linux # apt-get --reinstall install udev Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 50 not upgraded. Need to get 0 B/1,045 kB of archives. After this operation, 0 B of additional disk space will be used. (Reading database ... 130230 files and directories currently installed.) Preparing to unpack .../archives/udev_231-2_armhf.deb ... Unpacking udev (231-2) over (231-2) ... Processing triggers for systemd (231-2) ... Processing triggers for man-db (2.7.5-1) ... Setting up udev (231-2) ... addgroup: The group `input' already exists as a system group. Exiting. update-initramfs: deferring update (trigger activated) insserv: warning: current start runlevel(s) (empty) of script `udev' overrides LSB defaults (S). insserv: warning: current stop runlevel(s) (S) of script `udev' overrides LSB defaults (empty). Processing triggers for initramfs-tools (0.125) ... /boot/initrd.img-3.18.0-trunk-rpi2 does not exist. Cannot update. - I guess I'm just going to have to revert those systemd components to 231-1 then... Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On Aug 12, 2016, at 9:21 AM, Pete Batardwrote: > Okay. I have now gone through a dpkg -i install of all the (non dbgsym) .deb > I see on your server, and also issued a reboot for good measure, but I still > see the same problem with journald being failed, along with dependent services Hi Filipe, Would you like me to also do what pete describes (all the .deb files) or is there a more fine-grained test you’d like me to try? Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Okay. I have now gone through a dpkg -i install of all the (non dbgsym) .deb I see on your server, and also issued a reboot for good measure, but I still see the same problem with journald being failed, along with dependent services - root@pi ~ # systemctl start systemd-journald.service Job for systemd-journald.service failed because the control process exited with error code. See "systemctl status systemd-journald.service" and "journalctl -xe" for details. root@pi ~ # systemctl status systemd-journald.service ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled) Active: failed (Result: start-limit-hit) since Fri 2016-08-12 17:17:12 IST; 5s ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Process: 1560 ExecStart=/lib/systemd/systemd-journald (code=exited, status=228/SECCOMP) Main PID: 1560 (code=exited, status=228/SECCOMP) root@pi ~ # systemctl --failed UNITLOAD ACTIVE SUBDESCRIPTION ● systemd-journald.serviceloaded failed failed Journal Service ● systemd-logind.service loaded failed failed Login Service ● systemd-networkd.serviceloaded failed failed Network Service ● systemd-resolved.serviceloaded failed failed Network Name Resolution ● systemd-journald-dev-log.socket loaded failed failed Journal Socket (/dev/log) ● systemd-journald.socket loaded failed failed Journal Socket ● systemd-networkd.socket loaded failed failed Network Service Netlink Socket LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB= The low-level unit activation state, values depend on unit type. 7 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. - One thing I still notice on the download server is that there exists a 'systemd-journal-remote-dbgsym_231-2_armhf.deb' but no 'systemd-journal-remote_231-2_armhf.deb'. There's also no 'systemd-sysv' despite being listed in the .dsc. Maybe those .deb's are still missing? Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 12 August 2016 at 11:05, Pete Batardwrote: > Thanks Felipe. > > I installed libsystemd0_231-2_armhf.deb but shouldn't there be a > systemd_231-2_armhf.deb as well? Strangely there's a > systemd-dbgsym_231-2_armhf.deb but no non-dbgsym version, as opposed to > other .debs. Sorry about that. Somehow that deb didn't get uploaded from my machine. > The .dsc does mention a 'systemd' binary, and earlier, the idea was to > install 'libsystemd0' and 'systemd' from the generated debs. > > For the time being, I'm not sure if I should just go ahead and install the > dbgsym version, or if there's still one piece missing. > > Please let me know before I proceed. No, the -dbgsym version only contains debugging symbols, not the actual executables. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Thanks Felipe. I installed libsystemd0_231-2_armhf.deb but shouldn't there be a systemd_231-2_armhf.deb as well? Strangely there's a systemd-dbgsym_231-2_armhf.deb but no non-dbgsym version, as opposed to other .debs. The .dsc does mention a 'systemd' binary, and earlier, the idea was to install 'libsystemd0' and 'systemd' from the generated debs. For the time being, I'm not sure if I should just go ahead and install the dbgsym version, or if there's still one piece missing. Please let me know before I proceed. Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 8 August 2016 at 10:03, Felipe Satelerwrote: > On 8 August 2016 at 02:59, Pete Batard wrote: >> Thanks Felipe. >> >> I'm afraid it's still not completing successfully though: > > Gah, I'm terribly sorry. I have a fix here, but I'm testbuilding first > before asking you to do that again. I'll let you know as soon as I > have built it. It has built fine. You can grab it again from https://people.debian.org/~fsateler/systemd/ . This time I have prebuilt armhf binaries so you can skip the build part :) -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 8 August 2016 at 02:59, Pete Batardwrote: > Thanks Felipe. > > I'm afraid it's still not completing successfully though: Gah, I'm terribly sorry. I have a fix here, but I'm testbuilding first before asking you to do that again. I'll let you know as soon as I have built it. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Thanks Felipe. I'm afraid it's still not completing successfully though: ../src/test/test-execute.c: In function ‘test_exec_systemcallfilter’: ../src/test/test-execute.c:120:14: error: implicit declaration of function ‘is_seccomp_enabled’ [-Werror=implicit-function-declaration] if (!is_seccomp_enabled()) ^ ../src/test/test-execute.c:120:9: warning: nested extern declaration of ‘is_seccomp_enabled’ [-Wnested-externs] if (!is_seccomp_enabled()) ^ ../src/test/test-execute.c: In function ‘test_exec_systemcall_system_mode_with_user’: ../src/test/test-execute.c:141:9: error: expected expression before ‘if’ if (getpwnam("nobody")) ^ ../src/test/test-execute.c:141:9: warning: ‘return’ with a value, in function returning void cc1: some warnings being treated as errors Makefile:20308: recipe for target 'src/test/test_execute-test-execute.o' failed make[4]: *** [src/test/test_execute-test-execute.o] Error 1 Makefile:21743: recipe for target 'all-recursive' failed make[3]: *** [all-recursive] Error 1 Makefile:9783: recipe for target 'all' failed make[2]: *** [all] Error 2 make[2]: Leaving directory '/root/systemd-231/build-deb' dh_auto_build: make -j1 returned exit code 2 debian/rules:194: recipe for target 'override_dh_auto_build' failed make[1]: *** [override_dh_auto_build] Error 2 make[1]: Leaving directory '/root/systemd-231' debian/rules:376: recipe for target 'build' failed make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 6 August 2016 at 05:20, Pete Batardwrote: > Since Rick hasn't replied, and I've been encountering the same issue on a > Raspberry Pi 2 model B running sid, I've been trying the steps mentioned > above. Hi Pete, I have updated the source package hopefully fixing this issue. Please re-download and try again. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Since Rick hasn't replied, and I've been encountering the same issue on a Raspberry Pi 2 model B running sid, I've been trying the steps mentioned above. However the post-build tests failed with the following: - test-watchdog.log Hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0 Set hardware watchdog to 10s. Pinging... Pinging... Pinging... Pinging... Pinging... PASS test-watchdog (exit status: 0) test-web-util.log PASS test-web-util (exit status: 0) test-xattr-util.log PASS test-xattr-util (exit status: 0) test-xml.log PASS test-xml (exit status: 0) debian/rules:362: recipe for target 'override_dh_auto_test' failed make[1]: *** [override_dh_auto_test] Error 1 make[1]: Leaving directory '/root/systemd-231' debian/rules:376: recipe for target 'build' failed make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 - Not sure if it's something I missed, or something you want to look into. Regards, /Pete
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
> On Aug 3, 2016, at 3:58 PM, Felipe Satelerwrote: > > On 3 August 2016 at 16:44, Rick Thomas wrote: >> >> On Aug 3, 2016, at 9:06 AM, Felipe Sateler wrote: >> >>> On 1 August 2016 at 18:32, Rick Thomas wrote: Thanks, Filipe! What do we have to do at this point to test this and then translate it into a patch? >>> >>> OK, so I have a proof-of-concept patch. Rick, could you test it in your >>> machine? >> >> I’m afraid I don’t have the facilities or the expertise to turn this into a >> loadable kernel .deb . >> >> If someone can provide an installable kernel .deb file for a Cuboxi4-Pro >> machine, I’ll he happy to test it. > > This is a patch for systemd, not the kernel. I prepared a source > package for easier testing. Do the following: > > apt-get install dpkg-dev devscripts > apt-get build-dep systemd > dget https://people.debian.org/~fsateler/systemd_231-2.dsc > cd systemd-231 > dpkg-buildpackage > > That should leave you with several .debs (after a long time). Install > libsystemd0 and systemd from those debs. Great! Thanks for the instructions. I’ll give it a try soon. Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 3 August 2016 at 16:44, Rick Thomaswrote: > > On Aug 3, 2016, at 9:06 AM, Felipe Sateler wrote: > >> On 1 August 2016 at 18:32, Rick Thomas wrote: >>> >>> Thanks, Filipe! >>> >>> What do we have to do at this point to test this and then translate it into >>> a patch? >> >> OK, so I have a proof-of-concept patch. Rick, could you test it in your >> machine? > > I’m afraid I don’t have the facilities or the expertise to turn this into a > loadable kernel .deb . > > If someone can provide an installable kernel .deb file for a Cuboxi4-Pro > machine, I’ll he happy to test it. This is a patch for systemd, not the kernel. I prepared a source package for easier testing. Do the following: apt-get install dpkg-dev devscripts apt-get build-dep systemd dget https://people.debian.org/~fsateler/systemd_231-2.dsc cd systemd-231 dpkg-buildpackage That should leave you with several .debs (after a long time). Install libsystemd0 and systemd from those debs. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On Aug 3, 2016, at 9:06 AM, Felipe Satelerwrote: > On 1 August 2016 at 18:32, Rick Thomas wrote: >> >> Thanks, Filipe! >> >> What do we have to do at this point to test this and then translate it into >> a patch? > > OK, so I have a proof-of-concept patch. Rick, could you test it in your > machine? I’m afraid I don’t have the facilities or the expertise to turn this into a loadable kernel .deb . If someone can provide an installable kernel .deb file for a Cuboxi4-Pro machine, I’ll he happy to test it. Thanks! Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Am 03.08.2016 um 17:26 schrieb Felipe Sateler: > Control: forwarded -1 https://github.com/systemd/systemd/issues/3882 > Control: severity -1 serious > Yes, I'd like upstream's opinion on this first. However, I think this > bug should be bumped to RC to prevent migration to testing, as this > makes arm systems basically unusable. I have marked the bug as serios. > Feel free to downgrade if you disagree. No, I agree with the severity bump. Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 1 August 2016 at 18:32, Rick Thomaswrote: > > On Aug 1, 2016, at 2:40 PM, Felipe Sateler wrote: > >> On 28 July 2016 at 17:04, Michael Biebl wrote: >>> Am 28.07.2016 um 22:50 schrieb Rick Thomas: In the interest of having a working system, I reverted that machine to systemd version 230-7. Unsurprisingly, the problem went away. I’ll try re-installing 231-1 and commenting that line. I’ll probably have a chance tonight. I’ll report when I have something. It may be worth noticing that other things failed as well when 231-1 was in. I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note are “Failed to start Raise network interfaces” and “Failed to start Login Service.” Are there other places where I should remove a “SystemCallFilter” ? >>> >>> Various units were locked down like e.g. in >>> https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736 >>> >>> If the SystemCallFilter= is what causes journald to fail, it's likely it >>> also affects those other services. >> >> Turns out seccomp is disabled in the arm* kernels: >> >> % grep SECCOMP boot/config-4.6.0-1-marvell >> CONFIG_HAVE_ARCH_SECCOMP_FILTER=y >> # CONFIG_SECCOMP is not set >> >> % grep SECCOMP boot/config-4.6.0-1-armmp >> CONFIG_HAVE_ARCH_SECCOMP_FILTER=y >> # CONFIG_SECCOMP is not set >> >> So I think the kernel should enable SECCOMP. >> >> However, I think systemd should also simply (warn and) ignore seccomp >> calls if seccomp is not available in the current kernel. >> >> -- >> >> Saludos, >> Felipe Sateler > > Thanks, Filipe! > > What do we have to do at this point to test this and then translate it into a > patch? OK, so I have a proof-of-concept patch. Rick, could you test it in your machine? -- Saludos, Felipe Sateler diff --git a/src/core/execute.c b/src/core/execute.c index 7c178b9..2d45bc9 100644 --- a/src/core/execute.c +++ b/src/core/execute.c @@ -2103,35 +2103,37 @@ static int exec_child( } #ifdef HAVE_SECCOMP -if (use_address_families) { -r = apply_address_families(context); -if (r < 0) { -*exit_status = EXIT_ADDRESS_FAMILIES; -return r; +if (is_seccomp_enabled()) { +if (use_address_families) { +r = apply_address_families(context); +if (r < 0) { +*exit_status = EXIT_ADDRESS_FAMILIES; +return r; +} } -} -if (context->memory_deny_write_execute) { -r = apply_memory_deny_write_execute(context); -if (r < 0) { -*exit_status = EXIT_SECCOMP; -return r; +if (context->memory_deny_write_execute) { +r = apply_memory_deny_write_execute(context); +if (r < 0) { +*exit_status = EXIT_SECCOMP; +return r; +} } -} -if (context->restrict_realtime) { -r = apply_restrict_realtime(context); -if (r < 0) { -*exit_status = EXIT_SECCOMP; -return r; +if (context->restrict_realtime) { +r = apply_restrict_realtime(context); +if (r < 0) { +*exit_status = EXIT_SECCOMP; +return r; +} } -} -if (use_syscall_filter) { -r = apply_seccomp(context); -if (r < 0) { -*exit_status = EXIT_SECCOMP; -return r; +if (use_syscall_filter) { +r = apply_seccomp(context); +if (r < 0) { +*exit_status = EXIT_SECCOMP; +return r; +} } } #endif diff --git a/src/shared/seccomp-util.c b/src/shared/seccomp-util.c index 8656d11..41e22a4 100644 --- a/src/shared/seccomp-util.c +++ b/src/shared/seccomp-util.c @@ -21,6 +21,8 @@ #include #include +#include "alloc-util.h" +#include "fileio.h" #include "macro.h"
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Control: forwarded -1 https://github.com/systemd/systemd/issues/3882 Control: severity -1 serious On 3 August 2016 at 02:15, Michael Bieblwrote: > Am 01.08.2016 um 23:40 schrieb Felipe Sateler: >> So I think the kernel should enable SECCOMP. > > I agree, unless SECCOMP on arm has some unwanted side-effects. > Felipe, can you file a bug report against the linux package accordingly? Already reported https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833183 > >> However, I think systemd should also simply (warn and) ignore seccomp >> calls if seccomp is not available in the current kernel. > > I agree as well, Systemd should log an error and continue. > We need to file a bug report upstream for that? Any takers? I have forwarded this report and marked this bug as such. > Assuming upstream actually decides to make SECCOMP (once enabled) a hard > dependency, we need to find a different solution, like a check in the > maintainer scripts. That said, we should first try to get that adressed > upstream. Yes, I'd like upstream's opinion on this first. However, I think this bug should be bumped to RC to prevent migration to testing, as this makes arm systems basically unusable. I have marked the bug as serios. Feel free to downgrade if you disagree. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On Aug 2, 2016, at 11:15 PM, Michael Bieblwrote: > Am 01.08.2016 um 23:40 schrieb Felipe Sateler: >> So I think the kernel should enable SECCOMP. > > I agree, unless SECCOMP on arm has some unwanted side-effects. > Felipe, can you file a bug report against the linux package accordingly? It may be that there are size considerations that preclude turning on such features for kernels on certain arm devices (armel comes to mind with SheevaPlug and OpenRD as examples). I’m not sure of that — and definitely unsure of the details, but I guess the linux package maintainers will know for sure. Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Am 01.08.2016 um 23:40 schrieb Felipe Sateler: > So I think the kernel should enable SECCOMP. I agree, unless SECCOMP on arm has some unwanted side-effects. Felipe, can you file a bug report against the linux package accordingly? > However, I think systemd should also simply (warn and) ignore seccomp > calls if seccomp is not available in the current kernel. I agree as well, Systemd should log an error and continue. We need to file a bug report upstream for that? Any takers? Assuming upstream actually decides to make SECCOMP (once enabled) a hard dependency, we need to find a different solution, like a check in the maintainer scripts. That said, we should first try to get that adressed upstream. Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On Aug 1, 2016, at 2:40 PM, Felipe Satelerwrote: > On 28 July 2016 at 17:04, Michael Biebl wrote: >> Am 28.07.2016 um 22:50 schrieb Rick Thomas: >>> In the interest of having a working system, I reverted that machine to >>> systemd version 230-7. Unsurprisingly, the problem went away. >>> >>> I’ll try re-installing 231-1 and commenting that line. I’ll probably have >>> a chance tonight. I’ll report when I have something. >>> >>> It may be worth noticing that other things failed as well when 231-1 was >>> in. I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular >>> note are “Failed to start Raise network interfaces” and “Failed to start >>> Login Service.” >>> >>> Are there other places where I should remove a “SystemCallFilter” ? >>> >> >> Various units were locked down like e.g. in >> https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736 >> >> If the SystemCallFilter= is what causes journald to fail, it's likely it >> also affects those other services. > > Turns out seccomp is disabled in the arm* kernels: > > % grep SECCOMP boot/config-4.6.0-1-marvell > CONFIG_HAVE_ARCH_SECCOMP_FILTER=y > # CONFIG_SECCOMP is not set > > % grep SECCOMP boot/config-4.6.0-1-armmp > CONFIG_HAVE_ARCH_SECCOMP_FILTER=y > # CONFIG_SECCOMP is not set > > So I think the kernel should enable SECCOMP. > > However, I think systemd should also simply (warn and) ignore seccomp > calls if seccomp is not available in the current kernel. > > -- > > Saludos, > Felipe Sateler Thanks, Filipe! What do we have to do at this point to test this and then translate it into a patch? Michael, do you have any suggestions? Thanks! Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
On 28 July 2016 at 17:04, Michael Bieblwrote: > Am 28.07.2016 um 22:50 schrieb Rick Thomas: >> In the interest of having a working system, I reverted that machine to >> systemd version 230-7. Unsurprisingly, the problem went away. >> >> I’ll try re-installing 231-1 and commenting that line. I’ll probably have a >> chance tonight. I’ll report when I have something. >> >> It may be worth noticing that other things failed as well when 231-1 was in. >> I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note >> are “Failed to start Raise network interfaces” and “Failed to start Login >> Service.” >> >> Are there other places where I should remove a “SystemCallFilter” ? >> > > Various units were locked down like e.g. in > https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736 > > If the SystemCallFilter= is what causes journald to fail, it's likely it > also affects those other services. Turns out seccomp is disabled in the arm* kernels: % grep SECCOMP boot/config-4.6.0-1-marvell CONFIG_HAVE_ARCH_SECCOMP_FILTER=y # CONFIG_SECCOMP is not set % grep SECCOMP boot/config-4.6.0-1-armmp CONFIG_HAVE_ARCH_SECCOMP_FILTER=y # CONFIG_SECCOMP is not set So I think the kernel should enable SECCOMP. However, I think systemd should also simply (warn and) ignore seccomp calls if seccomp is not available in the current kernel. -- Saludos, Felipe Sateler
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Thanks for you help so far, Michael! Can I ask one last favor on this? It seems that the bug (whatever it is) depends on things like kernel version and machine architecture. So, can you suggest someone who can take this further? Thanks! Rick
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
As I replied to that (did you see it? There may have been some problems with my email about that time), commenting out the “SystemCallFilter” did not make the problem go away on either of the ARM systems. Another interesting datapoint: I upgraded my PowerMac G4 powerpc machine to latest Sid (including the systemd version in question) and it does *not* have the problem. This is the same result I saw on the amd64 (virtual) machine. So it seems to be arm specific… Any thoughts? On Jul 29, 2016, at 3:02 PM, Michael Bieblwrote: > Am 29.07.2016 um 23:29 schrieb Rick Thomas: >> Hmmm… Curiouser and curiouser! >> >> I upgraded a VM (amd64) to latest Sid (with systemd version 231-1). The >> problem is *not* present there. >> >> The problem may be specific to arm hardware? I’ll try it on a PowerPC G4 >> (Apple Mac PPC) machine later today. > > I think this might be arch specific, yes. > See my reply earlier: > >> SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount >> @obsolete @raw-io >> >> is causing the problem? If you comment out that line from >> systemd-journald.service, does it start properly? >> >> SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully >> functional and we need to reassign this. > > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? >
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Am 29.07.2016 um 23:29 schrieb Rick Thomas: > Hmmm… Curiouser and curiouser! > > I upgraded a VM (amd64) to latest Sid (with systemd version 231-1). The > problem is *not* present there. > > The problem may be specific to arm hardware? I’ll try it on a PowerPC G4 > (Apple Mac PPC) machine later today. I think this might be arch specific, yes. See my reply earlier: > SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount > @obsolete @raw-io > > is causing the problem? If you comment out that line from > systemd-journald.service, does it start properly? > > SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully > functional and we need to reassign this. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Hmmm… Curiouser and curiouser! I upgraded a VM (amd64) to latest Sid (with systemd version 231-1). The problem is *not* present there. The problem may be specific to arm hardware? I’ll try it on a PowerPC G4 (Apple Mac PPC) machine later today. Rick On Jul 28, 2016, at 5:18 PM, Rick Thomaswrote: > I upgraded one of my test systems (an armhf Cubox-i4Pro) to Sid. > After rebooting, I saw the same symptoms as on the SheevaPlug.
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
I upgraded one of my test systems (an armhf Cubox-i4Pro) to Sid. After rebooting, I saw the same symptoms as on the SheevaPlug. I commented the line in question in /lib/systemd/system/systemd-logind.service and /lib/systemd/system/systemd-journald.service There was no corresponding line in /lib/systemd/system/networking.service so I did nothing there. I then did ‘update-initramfs -u’. Then I rebooted. Symptoms still persist, so that doesn’t seem to be it. I still have root@cube:~# systemctl status systemd-journald.service * systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled) Active: failed (Result: start-limit-hit) since Thu 2016-07-28 17:05:29 PDT; 8min ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 608 (code=exited, status=228/SECCOMP) Any other thoughts? Rick On Jul 28, 2016, at 2:04 PM, Michael Bieblwrote: > Am 28.07.2016 um 22:50 schrieb Rick Thomas: >> In the interest of having a working system, I reverted that machine to >> systemd version 230-7. Unsurprisingly, the problem went away. >> >> I’ll try re-installing 231-1 and commenting that line. I’ll probably have a >> chance tonight. I’ll report when I have something. >> >> It may be worth noticing that other things failed as well when 231-1 was in. >> I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note >> are “Failed to start Raise network interfaces” and “Failed to start Login >> Service.” >> >> Are there other places where I should remove a “SystemCallFilter” ? >> > > Various units were locked down like e.g. in > https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736 > > If the SystemCallFilter= is what causes journald to fail, it's likely it > also affects those other services. > > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? >
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Am 28.07.2016 um 22:50 schrieb Rick Thomas: > In the interest of having a working system, I reverted that machine to > systemd version 230-7. Unsurprisingly, the problem went away. > > I’ll try re-installing 231-1 and commenting that line. I’ll probably have a > chance tonight. I’ll report when I have something. > > It may be worth noticing that other things failed as well when 231-1 was in. > I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note > are “Failed to start Raise network interfaces” and “Failed to start Login > Service.” > > Are there other places where I should remove a “SystemCallFilter” ? > Various units were locked down like e.g. in https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736 If the SystemCallFilter= is what causes journald to fail, it's likely it also affects those other services. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
In the interest of having a working system, I reverted that machine to systemd version 230-7. Unsurprisingly, the problem went away. I’ll try re-installing 231-1 and commenting that line. I’ll probably have a chance tonight. I’ll report when I have something. It may be worth noticing that other things failed as well when 231-1 was in. I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note are “Failed to start Raise network interfaces” and “Failed to start Login Service.” Are there other places where I should remove a “SystemCallFilter” ? screenlog.xz Description: Binary data On Jul 28, 2016, at 1:24 PM, Michael Bieblwrote: > Am 28.07.2016 um 22:01 schrieb Rick Thomas: >> No. This is a stock kernel. It’s available from both testing and unstable >> repos. The machine itself is a SheevaPlug. Nothing custom about it. >> >> What makes you think it’s custom? > > This was more of a question then a statement. > I wanted to rule out, that the kernel was missing any relevant features. > > I assume the previous version worked properly and the addition of > > SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount > @obsolete @raw-io > > is causing the problem? If you comment out that line from > systemd-journald.service, does it start properly? > > SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully > functional and we need to reassign this. > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? >
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
Am 28.07.2016 um 22:01 schrieb Rick Thomas: > No. This is a stock kernel. It’s available from both testing and unstable > repos. The machine itself is a SheevaPlug. Nothing custom about it. > > What makes you think it’s custom? This was more of a question then a statement. I wanted to rule out, that the kernel was missing any relevant features. I assume the previous version worked properly and the addition of SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io is causing the problem? If you comment out that line from systemd-journald.service, does it start properly? SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully functional and we need to reassign this. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
No. This is a stock kernel. It’s available from both testing and unstable repos. The machine itself is a SheevaPlug. Nothing custom about it. What makes you think it’s custom? Rick On Jul 28, 2016, at 3:59 AM, Michael Bieblwrote: > control: tags -1 + moreinfo > Am 28.07.2016 um 12:08 schrieb Rick Thomas: >> Main PID: 477 (code=exited, status=228/SECCOMP) > ... >> Kernel: Linux 4.6.0-1-marvell > > That looks like you are using a custom kernel. Is the problem > reproducible with the default Debian Linux kernel? > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? >
Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start
control: tags -1 + moreinfo Am 28.07.2016 um 12:08 schrieb Rick Thomas: > Main PID: 477 (code=exited, status=228/SECCOMP) ... > Kernel: Linux 4.6.0-1-marvell That looks like you are using a custom kernel. Is the problem reproducible with the default Debian Linux kernel? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature