[ https://issues.apache.org/jira/browse/MESOS-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210343#comment-17210343 ]
Benjamin Mahler commented on MESOS-10186: ----------------------------------------- [~jpreddy] have you looked at what the error:1408F09C:SSL routines:ssl3_get_record:http request error means? Sounds like something is trying to speak the wrong protocol perhaps.. my suggestion would be to take a look at the bytes being sent to see if they look right for TLS. > Segmentation fault while running mesos in SSL mode > -------------------------------------------------- > > Key: MESOS-10186 > URL: https://issues.apache.org/jira/browse/MESOS-10186 > Project: Mesos > Issue Type: Bug > Affects Versions: 1.7.0, 1.9.0 > Reporter: Jaya Prasad Reddy G > Priority: Blocker > > Hello, > I've been runnning into segmentation faults while running mesos in SSL mode. > After backtracing the coredump, this is what I found: > {code:java} > #0 0x00007f93060b7592 in free () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f9303ebe8cd in CRYPTO_free () from > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 > #2 0x00007f9303f7bfde in EVP_CIPHER_CTX_cleanup () from > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 > #3 0x00007f93042e4275 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 > #4 0x00007f93042e59ca in SSL_set_accept_state () from > /lib/x86_64-linux-gnu/libssl.so.1.0.0 > #5 0x00007f93059a5c72 in ?? () from > /usr/lib/x86_64-linux-gnu/libevent_openssl-2.0.so.5 > #6 0x00007f930a801a74 in > process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback > (request=request@entry=0x7f92f8000a80) at > src/posix/libevent/libevent_ssl_socket.cpp:1172 > #7 0x00007f930a8021ea in > process::network::internal::LibeventSSLSocketImpl::accept_callback > (this=this@entry=0x55569161b430, request=request@entry=0x7f92f8000a80) at > src/posix/libevent/libevent_ssl_socket.cpp:1124 > #8 0x00007f930a8027bc in > process::network::internal::LibeventSSLSocketImpl::<lambda(evconnlistener*, > int, sockaddr*, int, void*)>::operator()(evconnlistener *, int, sockaddr *, > void *, int) (listener=0x555691667980, socket=24, > addr=<optimized out>, arg=0x55569161be30, addr_length=<optimized out>, > __closure=<optimized out>) at src/posix/libevent/libevent_ssl_socket.cpp:988 > #9 0x00007f9305e0829c in ?? () from > /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 > #10 0x00007f9305dfa639 in event_base_loop () from > /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 > #11 0x00007f930a81c41d in process::EventLoop::run () at > src/posix/libevent/libevent.cpp:98 > #12 0x00007f93066cbd00 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > #13 0x00007f930699c6ba in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #14 0x00007f930613a60d in clone () from /lib/x86_64-linux-gnu/libc.so.6 > {code} > The environment variables exported before running mesos are: > {code:java} > MESOS_SSL_CIPHERS=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA > > LIBPROCESS_SSL_ENABLED=true > LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false > LIBPROCESS_SSL_CA_DIR=/opt/mesos/secrets/certs/ > LIBPROCESS_SSL_KEY_FILE=/opt/mesos/secrets/certs/mesos.private_key > LIBPROCESS_SSL_CERT_FILE=/opt/mesos/secrets/certs/mesos.certificate > LIBPROCESS_SSL_VERIFY_CERT=false > LIBPROCESS_SSL_REQUIRE_CERT=false > LIBPROCESS_SSL_VERIFY_IPADD=false > LIBPROCESS_SSL_CA_FILE=/etc/ssl_ca/ca_list.pem > LIBPROCESS_SSL_CIPHERS=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA > > LIBPROCESS_SSL_ENABLE_SSL_V3=false > LIBPROCESS_SSL_ENABLE_TLS_V1_0=false > LIBPROCESS_SSL_ENABLE_TLS_V1_1=false > LIBPROCESS_SSL_ENABLE_TLS_V1_2=true > ZOOKEEPER_SSL_ENABLED=0 > ZOOKEEPER_SSL_VERIFY_CERT=false > ZOOKEEPER_SSL_REQUIRE_CERT=false > ZOOKEEPER_SSL_CA_FILE=/etc/ssl_ca/ca_list.pem > ZOOKEEPER_SSL_KEY_FILE=/opt/mesos/secrets/certs/mesos.private_key > ZOOKEEPER_SSL_CERT_FILE=/opt/mesos/secrets/certs/mesos.certificate > {code} > And the command used to run mesos: > {code:java} > mesos_master --ip=172.24.51.99 --advertise_ip=172.24.51.99 > --hostname_lookup=false --acls=file:///opt/mesos/etc/acl.json > --modules=file:///opt/mesos/etc/modules.json --port=5050 --quorum=3 > --work_dir=/ghostcache2/mesos/registry > --zk=zk://mycluster1.random.cluster.com:2181,mycluster2.random.cluster.com:2181,mycluster3.random.cluster.com:2181,mycluster4.random.cluster.com:2181,mycluster5.random.cluster.com:2181/mesos.mycluster > --cluster=mycluster --hostname=mycluster2.random.cluster.com > --offer_timeout=30secs --webui_dir=/opt/mesos/share/mesos/webui > --whitelist=file:///opt/mesos/etc/whitelist --authenticate_agents=true > --authenticators=crammd5 --authorizers=local > --credentials=file:///opt/mesos/secrets/all_credentials.json > --authenticate_frameworks=true{code} > lddtree of the mesos_master > {code:java} > ~# lddtree /a/sbin/mesos_master > mesos_master => /a/sbin/mesos_master (interpreter => > /lib64/ld-linux-x86-64.so.2) > libmesos-1.9.0.so => /usr/local/myproject/opt/mesos/lib/libmesos-1.9.0.so > libevent-2.0.so.5 => /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 > libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 > libevent_openssl-2.0.so.5 => > /usr/lib/x86_64-linux-gnu/libevent_openssl-2.0.so.5 > libevent_core-2.0.so.5 => > /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5 > libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 > libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 > libevent_pthreads-2.0.so.5 => > /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5 > libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 > libcrypto.so.111.0.0 => > /usr/local/myproject/lib/openssl/x86_64-linux-gnu/libcrypto.so.111.0.0 > libssl.so.111.0.0 => > /usr/local/myproject/lib/openssl/x86_64-linux-gnu/libssl.so.111.0.0 > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 > ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6{code} > > I tried updating mesos(from 1.7.0 to 1.9.0), libevent(2.0.21 to 2.0.22, 2.1.8 > and 2.1.12) to latest versions but still no luck. Not sure what I'm missing. > Any help is appreciated. > -- This message was sent by Atlassian Jira (v8.3.4#803005)