Hello I have a system, running on a 2 core device, doing packet processing. The kernel version is 4.9. 2 applications communicate with each other via IPC. One application is receiving packets from an ethernet interface and after doing some preprocessing it forwards the packets to the other application. Next to these 2 applications a lot of other applications are running, so the system is under high load.
Depending on the IPC mechanism I use between the 2 apps, I see a completely different CPU load pattern. The total load consumed by the 2 applications when using stream based Unix Domain sockets (UDS) is 130% (200% = full load on the 2 cores). The load consumed when using posix message queues is only 90%. Apart from some IPC implementation details, the logic is identical in the 2 tests so I cannot explain the 40% difference on the total load. I was doing some monitoring with top and in the /proc/<pid>/stat files of the 2 applications and I have the feeling that in the case of UDS, a lot more system processing is done in the application threads. I'm talking about the "stime" info the stat file. When I read "kernel-hacking/hacking.html" in the linux documentation I see: "Whenever a system call is about to return to userspace, or a hardware interrupt handler exits, any ‘software interrupts’ which are marked pending (usually by hardware interrupts) are run (kernel/softirq.c)." Can it be that this is the case for the read/write system calls used by UDS while this is not the case for the mqueue system calls (mq_timedsend/mq_timedreceive)? BTW the system is running with "threadirqs" enabled so that the irq work is done in dedicated kernel threads. Thanks Best regards, Ronny
