Hi all,
After some while I got back to building qpidc (just the client part actually)
on AIX. In the meanwhile I got an upgraded box with AIX 7.1 (on POWER CPU) and
latest compiler xlc 12.1. I then built boost 1.47.0 (with some IBM patch
required for xlc) and after a few more problems managed to build qpid (make
all) v0.24 to the point of linking qpidd, which fails because of some undefined
boost program_options symbols. I'll deal with this later, since I need the
client and required libraries libqpidtypes.so, libqpidcommon.so,
libqpidclient.so and libqpidmessaging.so got built OK AFAICT.
So, I try to build a simple client using example from the web site, and the
executable is created OK. When I run it though, it freezes at one point and
stays like this indefinetly. It has to do something with connection and threads
but I haven't been able to figure out the cause. I did some analysis and here
are the facts:
gdb output (we can see where it waits in main thread):
++++++++++++++++++
#4 0xd051d660 in pthread_cond_wait () from /usr/lib/libpthreads.a(shr_xpg5.o)
#5 0xd6c83cc8 in qpid::sys::Condition::wait( (qpid::sys::Mutex &))
(this=0x201b97b0, mutex=@0x201b977c: ???)
at
/temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/posix/Condition.h:63
#6 0xd6c83d9c in qpid::sys::Monitor::wait( (void)) (this=0x201b977c) at
/temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/Monitor.h:41
#7 0xd6c83e60 in
waitFor__Q3_4qpid6client12StateManagerFQ2_3std3setXTiTQ2_3std4lessXTi_TQ2_3std9allocatorXTi__
(this=0x201b9778, desired=@0x2ff21910: ???)
at /temporar/qpid_build/qpidc-0.24/src/qpid/client/StateManager.cpp:51
...
++++++++++++++++++
Specifically this is here:
++++++++++++++++++
qpid/sys/posix/Condition.h:
62 void Condition::wait(Mutex& mutex) {
63 QPID_POSIX_ASSERT_THROW_IF(pthread_cond_wait(&condition,
&mutex.mutex)); // <====
64 }
++++++++++++++++++
Qpid logs:
Client log:
++++++++++++++++++
2013-12-10 17:35:47 [Client] debug Created connection 192.168.0.142:5672 with {}
2013-12-10 17:35:47 [Client] debug Starting connection,
urls=[192.168.0.142:5672]
2013-12-10 17:35:47 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-10 17:35:47 [Client] debug Created IO thread: 0
2013-12-10 17:35:47 [Network] debug TCPConnector created for 0-10
2013-12-10 17:35:47 [Client] info Set TCP_NODELAY
2013-12-10 17:35:47 [System] info Connecting: 192.168.0.142:5672
++++++++++++++++++
Server log (this is trace level!):
++++++++++++++++++
2013-12-11 16:41:32 [System] error Connection
qpid.192.168.0.142:5672-192.168.0.135:47415 No protocol received closing
++++++++++++++++++
Netstat first shows established connection and then close_wait:
++++++++++++++++++
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 ESTABLISHED
...
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 CLOSE_WAIT
...
++++++++++++++++++
Output of truss (similar to strace on Linuxes):
++++++++++++++++++
...
8454278: 29950155: socket(2, 1, 0) = 5
8454278: 29950155: kioctl(5, -2147195266, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kioctl(5, -2147195267, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kfcntl(5, F_SETFL, 0x00000004) = 0
8454278: 29950155: getsockopt(5, 65535, 4104, 0x2FF21374, 0x2FF21370) = 0
8454278: 29950155: connext(5, 0x201FB688, 16) Err#55 EINPROGRESS
8454278: 29950155: ngetsockname(5, 0x2FF20BD0, 0x2FF210DC) = 0
8454278: 29950155: kwrite(4, " ", 1) = 1
[and then nothing]
++++++++++++++++++
Based on all this it seems that the connection get established, but then
something weird is happening with the Poller thread. When I try to debug the
run() function it gets stuck on if clause in line 724:
++++++++++++++++++
if (lastReturnedHandle) { // <== here
impl->resetMode(*lastReturnedHandle);
lastReturnedHandle = 0;
}
++++++++++++++++++
gdb session with trace log on stdout:
++++++++++++++++++
(gdb) break PosixPoller.cpp:724
No source file named PosixPoller.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (PosixPoller.cpp:724) pending.
(gdb) run
Starting program: /temporar/atk/qtmin
[New Thread 1]
entry
args
2013-12-11 18:52:23 [Unspecified] debug Config file not read:
/usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read:
/usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read:
/usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read:
/usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Client] debug Created connection 192.168.0.142:5672 with {}
conn
2013-12-11 18:52:23 [Client] debug Starting connection,
urls=[192.168.0.142:5672]
2013-12-11 18:52:23 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-11 18:52:23 [Client] debug Created IO thread: 0
[New Thread 258]
[Switching to Thread 258]
Breakpoint 1, qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98,
event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
724 if (lastReturnedHandle) {
(gdb) bt
#0 qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98,
event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
#1 0xd6070dd4 in qpid::sys::Poller::run( (void)) (this=0x201b7e98) at
/temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:690
#2 0xd622fb44 in qpid::sys::Thread.cpp-0::runRunnable( (void *))
(p=0x201b7e98) at
/temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/Thread.cpp:35
#3 0xd04fad88 in _pthread_body () from /usr/lib/libpthreads.a(shr_xpg5.o)
#4 0x00000000 in ?? ()
(gdb) step
2013-12-11 18:52:33 [Network] debug TCPConnector created for 0-10
2013-12-11 18:52:33 [Client] info Set TCP_NODELAY
2013-12-11 18:52:33 [System] info Connecting: 192.168.0.142:5672
[and then the endless wait]
++++++++++++++++++
If you managed to stay with me to this point, any ideas? :) Please note that I
had to hack quite some things to make it build on AIX, though semantically
there should be no change. Beside this I had to take ifaddrs.h and ifaddrs.cpp
from somebody on samba.org list and link with it, since this is not included in
AIX - I don't know if this should have any impact.
Btw, this is how I built the exe:
xlC_r -g -o qtmin ifaddrs.c qtest.cpp -I/temporar/qpid_build/qpidc-0.24/include
-L/temporar/qpid_build/qpidc-0.24/build/src/ -lqpidcommon -lqpidmessaging
-lpthreads -Wl,-brtl
I really appreciate any help with this.
Regards,
Ales
