[
https://issues.apache.org/jira/browse/DISPATCH-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620303#comment-16620303
]
Carsten Lohmann edited comment on DISPATCH-1086 at 9/19/18 2:06 PM:
--------------------------------------------------------------------
Here are backtraces from separate core dumps:
1:
{noformat}
#0 0x00007f4a0578f4fd in pn_list_get (list=0x1424900, index=24140)
at /build/qpid-proton-src/c/src/core/object/list.c:42
#1 0x00007f4a0579866f in pni_session_bound (ssn=<optimized out>)
at /build/qpid-proton-src/c/src/core/engine.c:1021
#2 0x00007f4a0579892b in pn_connection_bound (
connection=connection@entry=0x14244a0)
at /build/qpid-proton-src/c/src/core/engine.c:157
#3 0x00007f4a0579e268 in pn_transport_bind (transport=0xf1f760,
connection=0x14244a0) at /build/qpid-proton-src/c/src/core/transport.c:706
#4 0x00007f4a05797d35 in batch_next (batch=0x1461330)
at /build/qpid-proton-src/c/src/core/connection_driver.c:41
#5 0x00007f4a0557a6f1 in pconnection_batch_next ()
at /build/qpid-proton-src/c/src/proactor/epoll.c:948
#6 0x00007f4a05a0c2fb in thread_run (arg=arg@entry=0xf0bf60)
at /build/qpid-dispatch-src/src/server.c:976
#7 0x00007f4a05a0c590 in qd_server_run (qd=<optimized out>)
at /build/qpid-dispatch-src/src/server.c:1247
#8 0x000000000040182c in main_process (
config_path=0x7fff809c4965 "/tmp/qdrouterd.conf",
python_pkgdir=<optimized out>, fd=2)
at /build/qpid-dispatch-src/router/src/main.c:112
#9 0x0000000000401589 in main (argc=3, argv=0x7fff809c3ba8)
at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
2:
{noformat}
#0 ssl_cert_dup (cert=0x0) at ssl/ssl_cert.c:89
#1 0x00007fda9b487b98 in SSL_new (ctx=0x1fb8a40) at ssl/ssl_lib.c:716
#2 0x00007fda9c95b0c8 in init_ssl_socket (transport=0x7fda8014ec60,
ssl=ssl@entry=0x7fda8004dc70)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#3 0x00007fda9c95bc18 in init_ssl_socket (ssl=0x7fda8004dc70,
transport=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#4 process_input_ssl (transport=0x7fda8014ec60, layer=0,
input_data=0x7fda80163f90 "\220\a", available=0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:963
#5 0x00007fda9c95317a in transport_consume (
transport=transport@entry=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/core/transport.c:1821
#6 0x00007fda9c954eda in pn_transport_close_tail (transport=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/core/transport.c:2972
#7 0x00007fda9c94cdea in pn_connection_driver_read_close (
d=d@entry=0x7fda800e8968)
at /build/qpid-proton-src/c/src/core/connection_driver.c:114
#8 0x00007fda9c72f5f8 in pconnection_process (pc=pc@entry=0x7fda800e83c0,
events=events@entry=0, timeout=timeout@entry=false,
topup=topup@entry=true, is_io_2=is_io_2@entry=false)
at /build/qpid-proton-src/c/src/proactor/epoll.c:1230
#9 0x00007fda9c72f754 in pconnection_batch_next ()
at /build/qpid-proton-src/c/src/proactor/epoll.c:953
#10 0x00007fda9cbc12fb in thread_run (arg=0x1bd0f60)
at /build/qpid-dispatch-src/src/server.c:976
#11 0x00007fda9c512594 in start_thread (arg=<optimized out>)
at pthread_create.c:463
#12 0x00007fda9b7bbe6f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
3:
{noformat}
#0 0x00007f761a027858 in EVP_PKEY_up_ref (pkey=0x1a0bf8b480166)
at crypto/evp/p_lib.c:160
#1 0x00007f761a3898a4 in ssl_cert_dup (cert=0x7f761b85a6a0 <pn_link_incref>)
at ssl/ssl_cert.c:99
#2 0x00007f761a394b98 in SSL_new (ctx=0x7f761ba77940 <clazz>)
at ssl/ssl_lib.c:716
#3 0x00007f761b8680c8 in init_ssl_socket (transport=0xcbede0, ssl=0xc92d70)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#4 0x00007f761b869464 in init_ssl_socket (ssl=<optimized out>,
transport=<optimized out>)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#5 pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
session_id=session_id@entry=0x0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#6 0x00007f761bab09ef in qdr_handle_authentication_service_connection_event (
e=e@entry=0xc78900) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#7 0x00007f761bacda2c in handle (qd_server=qd_server@entry=0x85bf60,
e=e@entry=0xc78900, pn_conn=pn_conn@entry=0xc35ba0, ctx=ctx@entry=0x0)
at /build/qpid-dispatch-src/src/server.c:864
#8 0x00007f761bace2e4 in thread_run (arg=arg@entry=0x85bf60)
at /build/qpid-dispatch-src/src/server.c:973
#9 0x00007f761bace590 in qd_server_run (qd=<optimized out>)
at /build/qpid-dispatch-src/src/server.c:1247
#10 0x000000000040182c in main_process (
config_path=0x7ffecc430965 "/tmp/qdrouterd.conf",
python_pkgdir=<optimized out>, fd=2)
at /build/qpid-dispatch-src/router/src/main.c:112
#11 0x0000000000401589 in main (argc=3, argv=0x7ffecc42e848)
at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
4:
{noformat}
#0 __GI_abort () at abort.c:107
#1 0x00007f3308a8e7b7 in __libc_message (action=action@entry=do_abort,
fmt=fmt@entry=0x7f3308b98359 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#2 0x00007f3308a94c4c in malloc_printerr (
str=str@entry=0x7f3308b99c40 "munmap_chunk(): invalid pointer")
at malloc.c:5350
#3 0x00007f3308a951a4 in munmap_chunk (p=<optimized out>) at malloc.c:2846
#4 0x00007f33084d39f2 in OPENSSL_sk_free (st=0x7f32f00cc910)
at crypto/stack/stack.c:273
#5 0x00007f33084718e4 in sk_void_free (sk=<optimized out>)
at include/openssl/crypto.h:94
#6 CRYPTO_free_ex_data (class_index=class_index@entry=1,
obj=obj@entry=0x1bcd2c0, ad=ad@entry=0x1bcd390) at crypto/ex_data.c:361
#7 0x00007f33087d84be in SSL_CTX_free (a=0x1bcd2c0) at ssl/ssl_lib.c:2718
#8 0x00007f3309caecb7 in pn_ssl_domain_free (domain=0x1a240d0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:558
#9 0x00007f3309caf6b7 in pn_ssl_free (
transport=transport@entry=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:887
#10 0x00007f3309ca5108 in pn_transport_finalize (object=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/core/transport.c:663
#11 0x00007f3309c971f9 in pn_class_decref (clazz=0x7f3309ebdcc0 <clazz>,
clazz@entry=0x7f3309ebd540 <PN_OBJECT>, object=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/core/object/object.c:95
#12 0x00007f3309c9740f in pn_decref (object=<optimized out>)
at /build/qpid-proton-src/c/src/core/object/object.c:253
#13 0x00007f3309ca4dd1 in pn_transport_free (transport=<optimized out>)
at /build/qpid-proton-src/c/src/core/transport.c:643
#14 0x00007f3309c9ff14 in pn_connection_driver_destroy (
d=d@entry=0x7f32f0293f18)
at /build/qpid-proton-src/c/src/core/connection_driver.c:94
#15 0x00007f3309a805f4 in pconnection_final_free (pc=pc@entry=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:889
#16 0x00007f3309a814ec in pconnection_cleanup (pc=pc@entry=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:905
#17 0x00007f3309a841d2 in pconnection_done (pc=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:1045
#18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f32f0293f10)
at /build/qpid-proton-src/c/src/proactor/epoll.c:2167
#19 0x00007f3309f1431c in thread_run (arg=0x16a4f60)
at /build/qpid-dispatch-src/src/server.c:990
#20 0x00007f3309865594 in start_thread (arg=<optimized out>)
at pthread_create.c:463
#21 0x00007f3308b0ee6f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
5:
{noformat}
#0 0x00007fd87854eae3 in SSL_new (ctx=0x712f646c6975622f) at ssl/ssl_lib.c:677
#1 0x00007fd879a220c8 in init_ssl_socket (transport=0x7fd8600eb4e0,
ssl=0x7fd85c0ec390) at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#2 0x00007fd879a23464 in init_ssl_socket (ssl=<optimized out>,
transport=<optimized out>)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#3 pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
session_id=session_id@entry=0x0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#4 0x00007fd879c6a9ef in qdr_handle_authentication_service_connection_event (
e=e@entry=0x7fd85c055040) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#5 0x00007fd879c87a2c in handle (qd_server=qd_server@entry=0xc9ef60,
e=e@entry=0x7fd85c055040, pn_conn=pn_conn@entry=0x7fd860121790,
ctx=ctx@entry=0x0) at /build/qpid-dispatch-src/src/server.c:864
#6 0x00007fd879c882e4 in thread_run (arg=0xc9ef60)
at /build/qpid-dispatch-src/src/server.c:973
#7 0x00007fd8795d9594 in start_thread (arg=<optimized out>)
at pthread_create.c:463
#8 0x00007fd878882e6f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
was (Author: calohmn):
Here are backtraces from separate core dumps:
1:
{noformat}
#0 0x00007f4a0578f4fd in pn_list_get (list=0x1424900, index=24140)
at /build/qpid-proton-src/c/src/core/object/list.c:42
#1 0x00007f4a0579866f in pni_session_bound (ssn=<optimized out>)
at /build/qpid-proton-src/c/src/core/engine.c:1021
#2 0x00007f4a0579892b in pn_connection_bound (
connection=connection@entry=0x14244a0)
at /build/qpid-proton-src/c/src/core/engine.c:157
#3 0x00007f4a0579e268 in pn_transport_bind (transport=0xf1f760,
connection=0x14244a0) at /build/qpid-proton-src/c/src/core/transport.c:706
#4 0x00007f4a05797d35 in batch_next (batch=0x1461330)
at /build/qpid-proton-src/c/src/core/connection_driver.c:41
#5 0x00007f4a0557a6f1 in pconnection_batch_next ()
at /build/qpid-proton-src/c/src/proactor/epoll.c:948
#6 0x00007f4a05a0c2fb in thread_run (arg=arg@entry=0xf0bf60)
at /build/qpid-dispatch-src/src/server.c:976
#7 0x00007f4a05a0c590 in qd_server_run (qd=<optimized out>)
at /build/qpid-dispatch-src/src/server.c:1247
#8 0x000000000040182c in main_process (
config_path=0x7fff809c4965 "/tmp/qdrouterd.conf",
python_pkgdir=<optimized out>, fd=2)
at /build/qpid-dispatch-src/router/src/main.c:112
#9 0x0000000000401589 in main (argc=3, argv=0x7fff809c3ba8)
at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
2:
{noformat}
#0 ssl_cert_dup (cert=0x0) at ssl/ssl_cert.c:89
#1 0x00007fda9b487b98 in SSL_new (ctx=0x1fb8a40) at ssl/ssl_lib.c:716
#2 0x00007fda9c95b0c8 in init_ssl_socket (transport=0x7fda8014ec60,
ssl=ssl@entry=0x7fda8004dc70)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#3 0x00007fda9c95bc18 in init_ssl_socket (ssl=0x7fda8004dc70,
transport=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#4 process_input_ssl (transport=0x7fda8014ec60, layer=0,
input_data=0x7fda80163f90 "\220\a", available=0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:963
#5 0x00007fda9c95317a in transport_consume (
transport=transport@entry=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/core/transport.c:1821
#6 0x00007fda9c954eda in pn_transport_close_tail (transport=0x7fda8014ec60)
at /build/qpid-proton-src/c/src/core/transport.c:2972
#7 0x00007fda9c94cdea in pn_connection_driver_read_close (
d=d@entry=0x7fda800e8968)
at /build/qpid-proton-src/c/src/core/connection_driver.c:114
#8 0x00007fda9c72f5f8 in pconnection_process (pc=pc@entry=0x7fda800e83c0,
events=events@entry=0, timeout=timeout@entry=false,
topup=topup@entry=true, is_io_2=is_io_2@entry=false)
at /build/qpid-proton-src/c/src/proactor/epoll.c:1230
#9 0x00007fda9c72f754 in pconnection_batch_next ()
at /build/qpid-proton-src/c/src/proactor/epoll.c:953
#10 0x00007fda9cbc12fb in thread_run (arg=0x1bd0f60)
at /build/qpid-dispatch-src/src/server.c:976
#11 0x00007fda9c512594 in start_thread (arg=<optimized out>)
at pthread_create.c:463
#12 0x00007fda9b7bbe6f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
3:
{noformat}
#0 0x00007f761a027858 in EVP_PKEY_up_ref (pkey=0x1a0bf8b480166)
at crypto/evp/p_lib.c:160
#1 0x00007f761a3898a4 in ssl_cert_dup (cert=0x7f761b85a6a0 <pn_link_incref>)
at ssl/ssl_cert.c:99
#2 0x00007f761a394b98 in SSL_new (ctx=0x7f761ba77940 <clazz>)
at ssl/ssl_lib.c:716
#3 0x00007f761b8680c8 in init_ssl_socket (transport=0xcbede0, ssl=0xc92d70)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#4 0x00007f761b869464 in init_ssl_socket (ssl=<optimized out>,
transport=<optimized out>)
at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#5 pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
session_id=session_id@entry=0x0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#6 0x00007f761bab09ef in qdr_handle_authentication_service_connection_event (
e=e@entry=0xc78900) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#7 0x00007f761bacda2c in handle (qd_server=qd_server@entry=0x85bf60,
e=e@entry=0xc78900, pn_conn=pn_conn@entry=0xc35ba0, ctx=ctx@entry=0x0)
at /build/qpid-dispatch-src/src/server.c:864
#8 0x00007f761bace2e4 in thread_run (arg=arg@entry=0x85bf60)
at /build/qpid-dispatch-src/src/server.c:973
#9 0x00007f761bace590 in qd_server_run (qd=<optimized out>)
at /build/qpid-dispatch-src/src/server.c:1247
#10 0x000000000040182c in main_process (
config_path=0x7ffecc430965 "/tmp/qdrouterd.conf",
python_pkgdir=<optimized out>, fd=2)
at /build/qpid-dispatch-src/router/src/main.c:112
#11 0x0000000000401589 in main (argc=3, argv=0x7ffecc42e848)
at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
4:
{noformat}
#0 __GI_abort () at abort.c:107
#1 0x00007f3308a8e7b7 in __libc_message (action=action@entry=do_abort,
fmt=fmt@entry=0x7f3308b98359 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#2 0x00007f3308a94c4c in malloc_printerr (
str=str@entry=0x7f3308b99c40 "munmap_chunk(): invalid pointer")
at malloc.c:5350
#3 0x00007f3308a951a4 in munmap_chunk (p=<optimized out>) at malloc.c:2846
#4 0x00007f33084d39f2 in OPENSSL_sk_free (st=0x7f32f00cc910)
at crypto/stack/stack.c:273
#5 0x00007f33084718e4 in sk_void_free (sk=<optimized out>)
at include/openssl/crypto.h:94
#6 CRYPTO_free_ex_data (class_index=class_index@entry=1,
obj=obj@entry=0x1bcd2c0, ad=ad@entry=0x1bcd390) at crypto/ex_data.c:361
#7 0x00007f33087d84be in SSL_CTX_free (a=0x1bcd2c0) at ssl/ssl_lib.c:2718
#8 0x00007f3309caecb7 in pn_ssl_domain_free (domain=0x1a240d0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:558
#9 0x00007f3309caf6b7 in pn_ssl_free (
transport=transport@entry=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/ssl/openssl.c:887
#10 0x00007f3309ca5108 in pn_transport_finalize (object=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/core/transport.c:663
#11 0x00007f3309c971f9 in pn_class_decref (clazz=0x7f3309ebdcc0 <clazz>,
clazz@entry=0x7f3309ebd540 <PN_OBJECT>, object=0x7f32f02940e0)
at /build/qpid-proton-src/c/src/core/object/object.c:95
#12 0x00007f3309c9740f in pn_decref (object=<optimized out>)
at /build/qpid-proton-src/c/src/core/object/object.c:253
#13 0x00007f3309ca4dd1 in pn_transport_free (transport=<optimized out>)
at /build/qpid-proton-src/c/src/core/transport.c:643
#14 0x00007f3309c9ff14 in pn_connection_driver_destroy (
d=d@entry=0x7f32f0293f18)
at /build/qpid-proton-src/c/src/core/connection_driver.c:94
#15 0x00007f3309a805f4 in pconnection_final_free (pc=pc@entry=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:889
#16 0x00007f3309a814ec in pconnection_cleanup (pc=pc@entry=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:905
#17 0x00007f3309a841d2 in pconnection_done (pc=0x7f32f0293970)
at /build/qpid-proton-src/c/src/proactor/epoll.c:1045
#18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f32f0293f10)
at /build/qpid-proton-src/c/src/proactor/epoll.c:2167
#19 0x00007f3309f1431c in thread_run (arg=0x16a4f60)
at /build/qpid-dispatch-src/src/server.c:990
#20 0x00007f3309865594 in start_thread (arg=<optimized out>)
at pthread_create.c:463
#21 0x00007f3308b0ee6f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
> Dispatch Router sporadically goes into a state where TLS connections to the
> auth service fail
> ---------------------------------------------------------------------------------------------
>
> Key: DISPATCH-1086
> URL: https://issues.apache.org/jira/browse/DISPATCH-1086
> Project: Qpid Dispatch
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Keith Wall
> Priority: Major
>
> Whilst running performance tests against Enmasse, we periodically see a
> problem where Dispatch Router (1.1.0) goes into a state where fails to form
> TLS connections to the authservice. When this occurs, the router needs to be
> restarted to restore service. There does not seem to be a pattern to when the
> issue occurs, but in all cases where it has been seen, the test case included
> tens or hundreds of concurrently formed connections.
> The following message is written to the log:
>
> {noformat}
> 2018-07-06 10:38:45.543519 +0000 AUTHSERVICE (warning) Cannot initialise
> SSL{noformat}
> Unfortunately turning up the router logging (using the following command)
> reveal no more useful information. This Proton improvement JIRA was raised to
> include the diagnostics from OpenSSL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
