Hi,
I've been using pstreams library for some time, but when I upgraded
the kernel to 4.19 / 5.0 (I need patches for nftables API) I started
noticing problems with blocked read in glibc. I've created class on
top of pstreams since I needed timeout functionality etc. After I
found the problem with blocking I extended the conditions to check
rdbuf()->in_avail() > 0 before I call getline, but even then it
blocks. It happens mostly in async threads, which then hang till I
restart whole daemon. I've tried to inspect more in depth states and
I'm suspecting some buffer problems. As you can see, there is
malformed string at line no.2 `s=0x7f87e4000d12 " 7.526 ms.101.84
7.339 ms 1975 ", n=30)`. It should be something similar to ` 4
10.26.101.84 7.339 ms 7.526 ms` because it is output of traceroute
command. I'm not really sure it is problem of glibc. I honestly don't
know. I have my own old library to handle external processes
FILE* pipe = popen(command, "r");
if (!pipe)
{
return false;
}
while (fgets(buffer.data(), sizeof(buffer), pipe) != NULL)
{
returnSstream << buffer.data();
}
int processReturnValue = pclose(pipe);
It worked without any problems for many months on kernels 4.9, even
self compiled to apply patches for HFSC / fq_codel. Since I switched
to 4.19 it started to happen randomly.
Now even my library has similar problems, but in different parts of
glibc functions. It always stucks at some kind of read operation on
buffer of external process. I've tried to upgrade distribution from
Debian Stretch to Debian testing and I'm trying it with stock kernel
(4.19) instead of compiled one. It happens too. PStream.hpp is copy of
pstream.h, but I removed some doxygen lines to prevent warnings. I
didn't make any other modifications to it, I just prepared it in case
I would have to. I'm including it because line numbers would probably
mismatch with original. https://pastebin.com/uWYKacfr
I can recompile it with original file if needed. I just can't say when
it will hang. It happens randomly in range of hours and not on every
testing machine I run my code on. I'll ask this to creator of pstreams
library too, but it seems to be somewhere deeper than his or my
libraries. Version of glibc was always stock one of Stretch / Buster.
I'm trying to create reproducer, but it can take some time since it
happens totally randomly.
if (cReadDone() == false &&
rBuffer.rdbuf()->in_avail() > 0 &&
std::getline(rBuffer, rOutput).good() == true) - this where it is
stuck - line #9 in gdb back trace
gdb:
where
#0 __libc_read (nbytes=30, buf=0x7f87e4000d12, fd=3) at
../sysdeps/unix/sysv/linux/read.c:26
#1 __libc_read (fd=3, buf=0x7f87e4000d12, nbytes=30) at
../sysdeps/unix/sysv/linux/read.c:24
#2 0x00005594c844075d in redi::basic_pstreambuf<char,
std::char_traits<char> >::read (this=0x7f87fa6c10d0, s=0x7f87e4000d12
" 7.526 ms.101.84 7.339 ms 1975 ", n=30) at
/root/shaperd/projects/include/PStream.hpp:1980
#3 0x00005594c84405d1 in redi::basic_pstreambuf<char,
std::char_traits<char> >::fill_buffer (this=0x7f87fa6c10d0,
non_blocking=false) at /root/shaperd/projects/include/PStream.hpp:1927
#4 0x00005594c8440186 in redi::basic_pstreambuf<char,
std::char_traits<char> >::underflow (this=0x7f87fa6c10d0) at
/root/shaperd/projects/include/PStream.hpp:1839
#5 0x00007f87fb9bd39e in std::basic_streambuf<char,
std::char_traits<char> >::sgetc (this=0x7f87fa6c10d0) at
/build/gcc-8-XAxxDW/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:345
#6 std::basic_streambuf<char, std::char_traits<char> >::sgetc
(this=<optimized out>) at
/build/gcc-8-XAxxDW/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:345
#7 std::basic_streambuf<char, std::char_traits<char> >::snextc
(this=<optimized out>) at
/build/gcc-8-XAxxDW/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:310
#8 std::getline<char, std::char_traits<char>, std::allocator<char> >
(__in=..., __str=" 4 10.26.101.84 7.339 ms 7.526 ms", __delim=10
'\n') at ../../../../../src/libstdc++-v3/src/c++98/istream-string.cc:170
#9 0x00005594c84c0f83 in PStreams::Class::GetLine
(this=0x7f87fa6c1090, rBuffer=..., rOutput=" 4 10.26.101.84 7.339 ms
7.526 ms") at /root/shaperd/projects/lib/libPStreams.cpp:73
----
S pozdravem / Best Regards
Vaclav Zindulka
