Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

> sgtm - got a patch series up now;  see https://reviews.apache.org/r/68906/

Have you seen 2 patches for cmake builds in the git ref?
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

> Note that I moved the setup of EV_READ | EV_WRITE setup a bit further up on 
> the connect part. I was worried that without doing so, we could possibly run 
> into races caused by data being available early. This would be matching our 
> callback setup timing.

Does event_base_loop() and
bufferevent_socket_connect()/bufferevent_enable() runs in different
threads? If so this can be true.

But this will not break anything, since they will be read after, that's
it. IOW both patches is fine.

-- 
  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

> Also it contains some fixes for cmake (since I don't like autotools due
> to how difficult to handle every single dependency right there, in
> particular "make -C 3rdparty/libprocess/ -j4 libprocess-tests" will not
> built all dependencies, if you curious).

That part I have yet to look into but I believe that we are still incomplete on 
our CMake adoption - specifically for building our own libprocess without 
building the entire Mesos project. Will pick up on that soonish.

> Thanks but this contribution guide requires too much actions for this
> small patches, so I would rather prefer the second option.

sgtm - got a patch series up now;  see https://reviews.apache.org/r/68906/

Note that I moved the setup of EV_READ | EV_WRITE setup a bit further up on the 
connect part. I was worried that without doing so, we could possibly run into 
races caused by data being available early. This would be matching our callback 
setup timing. Please let me know if that was not in line with best practices on 
the use of libevent.

> 
>>> Also since you wrote that "reverting makes thing worse" could you verify 
>>> referenced patches (including osx) ?
>> 
>> Am doing that right now. Getting back to you later today.
>> 
>> Thanks so much again, I am dancing in circles right now, that is how much 
>> this issue bothered me.
> 
> Great! Let's see if he issue will gone! (obvious it should)

I can confirm it works as expected in the following combinations;

macOS 10.14.1 - libevent 2.2
macOS 10.14.1 - libevent 2.1.8
macOS 10.14.1 - libevent 2.0.22 
Ubuntu 16.04 - libevent 2.1.8

You nailed it! :)

Thanks again!

Till***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

On Tue, Oct 02, 2018 at 11:49:47AM +0200, Till Toenshoff wrote:
> Thanks so much Azat —  and sorry for not answering right away on these great 
> news. I was buried in work — well, I bet you know the drill...
> 
> I now wonder if you would like to submit your patches yourself, making you a 
> highly valued Apache Mesos contributor :). Alternatively, I would just go 
> ahead, create a patch within our review / patch mechanics and reference your 
> invaluable help. Whatever suits your time and wishes.
> Here is how we commonly do those things: 
> https://github.com/apache/mesos/blob/master/docs/advanced-contribution.md

Hi Till,

Thanks but this contribution guide requires too much actions for this
small patches, so I would rather prefer the second option.

> >Also since you wrote that "reverting makes thing worse" could you verify 
> >referenced patches (including osx) ?
> 
> Am doing that right now. Getting back to you later today.
> 
> Thanks so much again, I am dancing in circles right now, that is how much 
> this issue bothered me.

Great! Let's see if he issue will gone! (obvious it should)

Regards,
Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

Thanks so much Azat —  and sorry for not answering right away on these great 
news. I was buried in work — well, I bet you know the drill...

I now wonder if you would like to submit your patches yourself, making you a 
highly valued Apache Mesos contributor :). Alternatively, I would just go 
ahead, create a patch within our review / patch mechanics and reference your 
invaluable help. Whatever suits your time and wishes.
Here is how we commonly do those things: 
https://github.com/apache/mesos/blob/master/docs/advanced-contribution.md

>Also since you wrote that "reverting makes thing worse" could you verify 
>referenced patches (including osx) ?

Am doing that right now. Getting back to you later today.

Thanks so much again, I am dancing in circles right now, that is how much this 
issue bothered me.

Till***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Alexander Rojas]

Thanks Azat!

This was a great catch!

Alexander Rojas
alexan...@mesosphere.io




> On 30. Sep 2018, at 23:52, Azat Khuzhin  wrote:
> 
> On Thu, Sep 13, 2018 at 03:05:41AM +0200, Till Toenshoff wrote:
>>> On 10. Sep 2018, at 12:02, Azat Khuzhin  wrote:
>>> 
> commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
>  be_openssl: don't add events during bev creation (like be_sock)
>>> 
>>> Thanks for the bisect!
>>> 
>>> Interesting, just revering it doesn't fixes the issue to me (on linux).
>>> And to reset to this commit I need openssl 1.0 (since I have 1.1), will
>>> prepare env for it and get back.
>>> 
 The above bisect was done on macOS 10.14 - which works fine with Mesos l+ 
 libevent 2.1.5 but breaks with libevent 2.1.8.
 
 Will do the same bisect on Ubuntu 18.04 as well - just to be sure.
>>> 
>> Bisect on Ubuntu 18.04 resulted in the very same offender.
>> 
>> Reverting that commit makes things worse, not better. Instead of a timeout 
>> on the receive, we then get a failure already on the accept.
>> 
>> ../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
>> (socket).failure(): Failed accept: connection error: 
>> error::lib(0):func(0):reason(0)
>> 
>> This bisect and the revert results are based on libssl 1.0.
>> 
>> Will also keep pondering … and get back with new results.
> 
> Hi Till,
> 
> So finally I had time to investigate this, and after looking into mesos
> sources I find out that you libevent wrappers do not enable needable
> events for openssl bufferevents. And that is why everything hangs (why I
> didn't check this before?).
> 
> I just added bufferevent_enable() into two places and now SSLTest.*
> passes.
> 
> You will find patches for mesos here:
>  https://github.com/azat-archive/mesos.git le-dev
> 
> Also it contains some fixes for cmake (since I don't like autotools due
> to how difficult to handle every single dependency right there, in
> particular "make -C 3rdparty/libprocess/ -j4 libprocess-tests" will not
> built all dependencies, if you curious).
> 
> And here is startup script for your docker image to pass libevent/mesos
> sources into it that I came up with:
>  https://gist.github.com/azat/41ee673d2b5dec9f1ac14a5970265be2
> 
> Before this I used your autotools build and hacks around to pass my
> libevent version into the image (things got complicated because that
> unit tests start binaries which reset environ) but maybe I messed up
> maybe not, but you bisect indeed correct. Anyway you should enable
> events manually if you want reliable code.
> 
> Also since you wrote that "reverting makes thing worse" could you
> verify referenced patches (including osx) ?
> 
> And thanks again for the docker image, details and your correct bisect!
> 
> Regards,
> Azat.
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.

***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

On Thu, Sep 13, 2018 at 03:05:41AM +0200, Till Toenshoff wrote:
> > On 10. Sep 2018, at 12:02, Azat Khuzhin  wrote:
> > 
> >>> commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
> >>>   be_openssl: don't add events during bev creation (like be_sock)
> > 
> > Thanks for the bisect!
> > 
> > Interesting, just revering it doesn't fixes the issue to me (on linux).
> > And to reset to this commit I need openssl 1.0 (since I have 1.1), will
> > prepare env for it and get back.
> > 
> >> The above bisect was done on macOS 10.14 - which works fine with Mesos l+ 
> >> libevent 2.1.5 but breaks with libevent 2.1.8.
> >> 
> >> Will do the same bisect on Ubuntu 18.04 as well - just to be sure.
> > 
> Bisect on Ubuntu 18.04 resulted in the very same offender.
> 
> Reverting that commit makes things worse, not better. Instead of a timeout on 
> the receive, we then get a failure already on the accept.
> 
> ../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
> (socket).failure(): Failed accept: connection error: 
> error::lib(0):func(0):reason(0)
> 
> This bisect and the revert results are based on libssl 1.0.
> 
> Will also keep pondering … and get back with new results.

Hi Till,

So finally I had time to investigate this, and after looking into mesos
sources I find out that you libevent wrappers do not enable needable
events for openssl bufferevents. And that is why everything hangs (why I
didn't check this before?).

I just added bufferevent_enable() into two places and now SSLTest.*
passes.

You will find patches for mesos here:
  https://github.com/azat-archive/mesos.git le-dev

Also it contains some fixes for cmake (since I don't like autotools due
to how difficult to handle every single dependency right there, in
particular "make -C 3rdparty/libprocess/ -j4 libprocess-tests" will not
built all dependencies, if you curious).

And here is startup script for your docker image to pass libevent/mesos
sources into it that I came up with:
  https://gist.github.com/azat/41ee673d2b5dec9f1ac14a5970265be2

Before this I used your autotools build and hacks around to pass my
libevent version into the image (things got complicated because that
unit tests start binaries which reset environ) but maybe I messed up
maybe not, but you bisect indeed correct. Anyway you should enable
events manually if you want reliable code.

Also since you wrote that "reverting makes thing worse" could you
verify referenced patches (including osx) ?

And thanks again for the docker image, details and your correct bisect!

Regards,
Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

> On 10. Sep 2018, at 12:02, Azat Khuzhin  wrote:
> 
>>> commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
>>>   be_openssl: don't add events during bev creation (like be_sock)
> 
> Thanks for the bisect!
> 
> Interesting, just revering it doesn't fixes the issue to me (on linux).
> And to reset to this commit I need openssl 1.0 (since I have 1.1), will
> prepare env for it and get back.
> 
>> The above bisect was done on macOS 10.14 - which works fine with Mesos l+ 
>> libevent 2.1.5 but breaks with libevent 2.1.8.
>> 
>> Will do the same bisect on Ubuntu 18.04 as well - just to be sure.
> 
Bisect on Ubuntu 18.04 resulted in the very same offender.

Reverting that commit makes things worse, not better. Instead of a timeout on 
the receive, we then get a failure already on the accept.

../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
(socket).failure(): Failed accept: connection error: 
error::lib(0):func(0):reason(0)

This bisect and the revert results are based on libssl 1.0.

Will also keep pondering … and get back with new results.

***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

> > commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
> >be_openssl: don't add events during bev creation (like be_sock)

Thanks for the bisect!

Interesting, just revering it doesn't fixes the issue to me (on linux).
And to reset to this commit I need openssl 1.0 (since I have 1.1), will
prepare env for it and get back.

> The above bisect was done on macOS 10.14 - which works fine with Mesos l+ 
> libevent 2.1.5 but breaks with libevent 2.1.8.
> 
> Will do the same bisect on Ubuntu 18.04 as well - just to be sure.

Regards,
Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

> On 10. Sep 2018, at 02:58, Till Toenshoff  wrote:
> 
> f4b6284b8393dbabf389ddce734a30f4cdeffa17 is the first bad commit
> commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
> Author: Azat Khuzhin mailto:a3at.m...@gmail.com>>
> Date:   Thu Nov 5 17:40:25 2015 +0300
> 
>be_openssl: don't add events during bev creation (like be_sock)
> 
>Using the following examples you can get changes between be_openssl and
>be_sock:
>$ function diff_addr()
>{
>eval diff -u $(printf "<(strip_addr %s) " "$@")
>}
>$ function strip_addr()
>{
>sed 's/0x[a-zA-Z0-9]*/0x/g' "$@"
>}
>$ EVENT_DEBUG_LOGGING_ALL= regress --verbose --no-fork 
> +http/https_connection_retry 2> /tmp/https-retry.log >&2
>$ EVENT_DEBUG_LOGGING_ALL= regress --verbose --no-fork 
> +http/connection_retry 2> /tmp/http-retry.log >&2
>$ diff_addr /tmp/http-retry.log /tmp/https-retry.log
> 
> :100644 100644  4afdde27b2ab63cb92de31922688e77d97fc037b 
> 7d469bfad00806fb4f124e55619bfa59813b09f6 M  bufferevent_openssl.c
> bisect run success

The above bisect was done on macOS 10.14 - which works fine with Mesos l+ 
libevent 2.1.5 but breaks with libevent 2.1.8.

Will do the same bisect on Ubuntu 18.04 as well - just to be sure.

Till

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

> On 10. Sep 2018, at 01:13, Azat Khuzhin  wrote:
> 
>> ```
>> docker run -it docker.io/tillt/mesos-debug-libevent-ubuntu18:version1 
>> /root/mesos/build/3rdparty/libprocess/libprocess-tests 
>> --gtest_filter="SSLTest.SSLSocket" --verbose
>> ```
> 
> That is great!
> 
>> Any help debugging further into this issue is highly welcome.
> 
> Can you bisect libevent to see the specific commit/commits which breaks
> this? Plus I'm looking into this too and will get back once I will find
> something.
> 

The offending commit is:
```
f4b6284b8393dbabf389ddce734a30f4cdeffa17 is the first bad commit
commit f4b6284b8393dbabf389ddce734a30f4cdeffa17
Author: Azat Khuzhin 
Date:   Thu Nov 5 17:40:25 2015 +0300

be_openssl: don't add events during bev creation (like be_sock)

Using the following examples you can get changes between be_openssl and
be_sock:
$ function diff_addr()
{
eval diff -u $(printf "<(strip_addr %s) " "$@")
}
$ function strip_addr()
{
sed 's/0x[a-zA-Z0-9]*/0x/g' "$@"
}
$ EVENT_DEBUG_LOGGING_ALL= regress --verbose --no-fork 
+http/https_connection_retry 2> /tmp/https-retry.log >&2
$ EVENT_DEBUG_LOGGING_ALL= regress --verbose --no-fork 
+http/connection_retry 2> /tmp/http-retry.log >&2
$ diff_addr /tmp/http-retry.log /tmp/https-retry.log

:100644 100644 4afdde27b2ab63cb92de31922688e77d97fc037b 
7d469bfad00806fb4f124e55619bfa59813b09f6 M  bufferevent_openssl.c
bisect run success
```

Will also look closer at that….

Thanks for your help!
Till


***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

> ```
> docker run -it docker.io/tillt/mesos-debug-libevent-ubuntu18:version1 
> /root/mesos/build/3rdparty/libprocess/libprocess-tests 
> --gtest_filter="SSLTest.SSLSocket" --verbose
> ```

That is great!

> Any help debugging further into this issue is highly welcome.

Can you bisect libevent to see the specific commit/commits which breaks
this? Plus I'm looking into this too and will get back once I will find
something.

Regards,
Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

After some time, I would like to revive this thread as the problem has not been 
fixed within Mesos.

So we still fail to support Ubuntu 17, 18.04 or macOS with Mesos in combination 
with libevent 2.1.8. Interestingly I was able to get the test that fails on the 
above to work on macOS 10.14 with libevent 2.1.5 - it fails on macOS 10.13 
though - very puzzling. And finally, libevent 2.0.22 works accross all 
platforms in combination with Mesos.

I have gone ahead and created a docker image that contains a compiled libevent 
2.1.8;

```
cd /root/libevent-2.1.8-stable
./autogen.sh
./configure  --enable-debug-mode --enable-verbose-debug --prefix=/root/usr
make -j16
make install
```

Additionally, the image contains a prebuilt Mesos including debug symbols;

```
cd /root/mesos
./bootstrap
mkdir build
cd build
../configure --enable-ssl --enable-libevent --disable-optimize --enable-debug 
--disable-java --disable-python --disable-werror --with-libevent=/root/usr
make tests -j16
```

All of the above details can also be seen on 
https://hub.docker.com/r/tillt/mesos-debug-libevent-ubuntu18/


For running the test that Alexander mentioned before;


```
docker run -it docker.io/tillt/mesos-debug-libevent-ubuntu18:version1 
/root/mesos/build/3rdparty/libprocess/libprocess-tests 
--gtest_filter="SSLTest.SSLSocket" --verbose
```


Any help debugging further into this issue is highly welcome.


Thanks,
Till

***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Till Toenshoff]

After some time, I would like to revive this thread as the problem has not
been fixed within Mesos.

So we still fail to support Ubuntu 17, 18.04 or macOS with Mesos in
combination with libevent 2.1.8. Interestingly I was able to get the test
that fails on the above to work on macOS 10.14 with libevent 2.1.5 - it
fails on macOS 10.13 though - very puzzling. And finally, libevent 2.0.22
works accross all platforms in combination with Mesos.

I have gone ahead and created a docker image that contains a compiled
libevent 2.1.8;

```
cd /root/libevent-2.1.8-stable
./autogen.sh
./configure  --enable-debug-mode --enable-verbose-debug --prefix=/root/usr
make -j16
make install
```

Additionally, the image contains a prebuilt Mesos including debug symbols;

```
cd /root/mesos
./bootstrap
mkdir build
cd build
../configure --enable-ssl --enable-libevent --disable-optimize
--enable-debug --disable-java --disable-python --disable-werror
--with-libevent=/root/usr
make tests -j16
```

All of the above details can also be seen on
https://hub.docker.com/r/tillt/mesos-debug-libevent-ubuntu18/


For running the test that Alexander mentioned before;


```
docker run -it docker.io/tillt/mesos-debug-libevent-ubuntu18:version1
/root/mesos/build/3rdparty/libprocess/libprocess-tests
--gtest_filter="SSLTest.SSLSocket" --verbose
```


Any help debugging further into this issue is highly welcome.


Thanks,
Till

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Alexander Rojas]

> On 14. Feb 2018, at 16:52, Azat Khuzhin  wrote:
> 
> On Wed, Feb 14, 2018 at 04:33:07PM +0100, Alexander Rojas wrote:
>> Really hard to achieve, I was playing around and failed to get a minimal 
>> reproducer.
> 
> Maybe you have a container with all this stuff (+debug symbols) ?
> So that I can play in it.

I’ll try to build that soon.

> 
>> [debug] ../event.c: asked to terminate loop.
> 
> H, who asked this?

From the codebase, it should only happen here 
https://github.com/apache/mesos/blob/e36e3b9989b1097dca9597097dd51d24aaba48fa/3rdparty/libprocess/src/process.cpp#L2420
 

 
which is called in teardown. I don’t see any other place which calls 
`event_base_loopexit()`

> 
>> [debug] timeout_next: event: 0x7f70ddf0, in 0 seconds, 71990 useconds
>> Error was 2
>> Error was 5
> 
> So:
> - 2 is SSL_ERROR_WANT_READ
> - 5 is SSL_ERROR_SYSCALL
> (according to openssl/ssl.h)
> 
> So can you add printing of errno along with SSL_get_error() in print_err()?
> 
> It looks like it got SSL_ERROR_WANT_READ and then tried to read() but
> failed, and hence SSL_ERROR_SYSCALL.
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.

I also redirect the output of the forked client, the output is now mixed between
server and client but may be useful:

```
[debug] event_add: event: 0x19804c0 (fd 5), EV_READcall 0x7ff904576f10
[debug] ../event.c: no events registered.
[debug] event_add: event: 0x1972cc0 (fd 6), EV_READcall 0xd58520
[debug] event_add: event: 0x7ff8d4000df0 (fd -1),EV_TIMEOUT call 0xd56130
[debug] event_add: event 0x7ff8d4000df0, timeout in 0 seconds 0 useconds, 
call 0xd56130
[debug] ../poll.c: poll reports 1
[debug] event_active: 0x19804c0 (fd 5), res 2, callback 0x7ff904576f10
[debug] event_process_active: event: 0x19804c0, EV_READ   call 0x7ff904576f10
[debug] timeout_next: event: 0x7ff8d4000df0, in 0 seconds, 0 useconds
Note: Google Test filter = SSLTest.SSLSocket-
[==] Running 1 test from 1 test case.
[--] Global test environment set-up.
[--] 1 test from SSLTest
[ RUN  ] SSLTest.SSLSocket
[debug] ../poll.c: poll reports 0
[debug] event_del: 0x7ff8d4000df0 (fd -1), callback 0xd56130
[debug] timeout_process: event: 0x7ff8d4000df0, call 0xd56130
[debug] event_active: 0x7ff8d4000df0 (fd -1), res 1, callback 0xd56130
[debug] event_del: 0x7ff8d4000df0 (fd -1), callback 0xd56130
[debug] event_process_active: event: 0x7ff8d4000df0,call 0xd56130
[debug] event_del: 0x7ff8d4000df0 (fd -1), callback 0xd56130
[debug] ../event.c: asked to terminate loop.
[debug] event_add: event: 0x7ff8cc000df0 (fd -1),EV_TIMEOUT call 0xd56130
[debug] event_add: event 0x7ff8cc000df0, timeout in 0 seconds 2 useconds, 
call 0xd56130
[debug] ../poll.c: poll reports 1
[debug] event_active: 0x19804c0 (fd 5), res 2, callback 0x7ff904576f10
[debug] event_process_active: event: 0x19804c0, EV_READ   call 0x7ff904576f10
[debug] timeout_next: event: 0x7ff8cc000df0, in 0 seconds, 2 useconds
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0214 16:49:48.387568  7747 openssl.cpp:504] CA file path is unspecified! NOTE: 
Set CA file path with LIBPROCESS_SSL_CA_FILE=
I0214 16:49:48.387590  7747 openssl.cpp:509] CA directory path unspecified! 
NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
I0214 16:49:48.387595  7747 openssl.cpp:514] Will not verify peer certificate!
NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate verification
I0214 16:49:48.387600  7747 openssl.cpp:520] Will only verify peer certificate 
if presented!
NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate verification
[debug] event_add: event: 0x19aeb18 (fd 7), EV_READcall 0x7ff904580b50
[debug] ../poll.c: poll reports 1
[debug] event_active: 0x19804c0 (fd 5), res 2, callback 0x7ff904576f10
[debug] event_process_active: event: 0x19804c0, EV_READ   call 0x7ff904576f10
[debug] timeout_next: event: 0x7ff8cc000df0, in 0 seconds, 27992 useconds
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0214 16:49:48.394479  7761 openssl.cpp:504] CA file path is unspecified! NOTE: 
Set CA file path with LIBPROCESS_SSL_CA_FILE=
I0214 16:49:48.394526  7761 openssl.cpp:509] CA directory path unspecified! 
NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
I0214 16:49:48.394531  7761 openssl.cpp:514] Will not verify peer certificate!
NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate verification
I0214 16:49:48.394534  7761 openssl.cpp:520] Will only verify peer certificate 
if presented!
NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate verification
[debug] event_add: event: 0x2c56700 (fd 5), EV_READcall 0x7f47e2940f10
[debug] ../event.c: no events registered.
[debug] ..

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

On Wed, Feb 14, 2018 at 04:33:07PM +0100, Alexander Rojas wrote:
> Really hard to achieve, I was playing around and failed to get a minimal 
> reproducer.

Maybe you have a container with all this stuff (+debug symbols) ?
So that I can play in it.

> [debug] ../event.c: asked to terminate loop.

H, who asked this?

> [debug] timeout_next: event: 0x7f70ddf0, in 0 seconds, 71990 useconds
> Error was 2
> Error was 5

So:
- 2 is SSL_ERROR_WANT_READ
- 5 is SSL_ERROR_SYSCALL
(according to openssl/ssl.h)

So can you add printing of errno along with SSL_get_error() in print_err()?

It looks like it got SSL_ERROR_WANT_READ and then tried to read() but
failed, and hence SSL_ERROR_SYSCALL.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Alexander Rojas]

Hi Azat

> On 4. Feb 2018, at 20:49, Azat Khuzhin  wrote:
> 
> Hi Alexander,
> 
> Sorry for the delay,
No problem, I have also been fixing other issues. Crunch time!
> 
>> So far, yes
> 
> Great.
> 
>> NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate
>> verification
> 
> Hm, maybe something with require-cert (is this SSL_VERIFY_PEER?) ?

Not really an issue, we can configure if verify peer is active or not.

> 
>> [pid 13305] --- SIGPIPE (Broken pipe) ---
>> [pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
>> -192, 0)= 5
>> [pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
>> 0x140750dd, 0)  = 1
>> [pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60,
>> 0x, -192, 0)   = 2
>> [pid 13305] --- SIGSEGV (Segmentation fault) ---
>> [pid 13292] +++ killed by SIGSEGV +++
>> [pid 13278] --- SIGCHLD (Child exited) ---
>> [pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60, 0, 11,
>> 0)  = 5
>> ../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
>> (socket).failure(): Failed accept: connection error:
>> error::lib(0):func(0):reason(0)
>> ```
> 
> I don't like SIGPIPE here, seems that under ltrace it got SIGPIPE and
> you do not ignore it, am I right [1]? Could you add code to ignore it and
> retry again?
> 

Our code should ignore SIGPIPE by default. I added signal(SIGPIPE, SIG_IGN) 
close 
to the call site and nothing changed.

>  [1]: 
> https://github.com/apache/mesos/blob/abd1751aa9a305b0648f4a630acbc84e7d837783/3rdparty/libprocess/src/process.cpp#L1029
> 
> I think that something like this before start in bash will do the trick:
>  trap '' SIGPIPE
> 
> Or maybe it will be better if you will remove #ifdef 0/#endif for
> print_err() in bufferevent_openssl.c and recompile it with it?
> 
> And the best thing that you can do is to produce a sample that I can
> compile and dig into, because I think that mesos is not the thing that I
> want to compile from sources for this.

Really hard to achieve, I was playing around and failed to get a minimal 
reproducer.

> 
>> I also verified our use of  `BEV_OPT_DEFER_CALLBACKS` and we do not use it
>> for listening sockets, so my backtrace should be alright. Connecting
>> sockets do use the flag however.
> 
> Regards,
> Azat.
> 
> P.S. you email client again messed up with long lines from ltrace
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.

What I did do was to recompile libevent from the git tag `release-2.1.8-stable`
with the following flags:

```
cmake -DCMAKE_BUILD_TYPE=Debug \
-GNinja \
-DEVENT__DISABLE_THREAD_SUPPORT=off \
-DEVENT__ENABLE_VERBOSE_DEBUG=on ..
```

I also enabled print_err(), so here is the output I got:

```
[debug] event_add: event: 0x1b094c0 (fd 5), EV_READcall 0x7f710f1d5f10
[debug] ../event.c: no events registered. # This line repeats ~2000 times
...
[debug] event_add: event: 0x1afbcc0 (fd 6), EV_READcall 0xd58520
[debug] event_add: event: 0x7f70dc000df0 (fd -1),EV_TIMEOUT call 0xd56130
[debug] event_add: event 0x7f70dc000df0, timeout in 0 seconds 0 useconds, 
call 0xd56130
[debug] ../poll.c: poll reports 1
[debug] event_active: 0x1b094c0 (fd 5), res 2, callback 0x7f710f1d5f10
[debug] event_process_active: event: 0x1b094c0, EV_READ   call 0x7f710f1d5f10
[debug] timeout_next: event: 0x7f70dc000df0, in 0 seconds, 0 useconds
Note: Google Test filter = SSLTest.SSLSocket-
[==] Running 1 test from 1 test case.
[--] Global test environment set-up.
[--] 1 test from SSLTest
[ RUN  ] SSLTest.SSLSocket
[debug] ../poll.c: poll reports 0
[debug] event_del: 0x7f70dc000df0 (fd -1), callback 0xd56130
[debug] timeout_process: event: 0x7f70dc000df0, call 0xd56130
[debug] event_active: 0x7f70dc000df0 (fd -1), res 1, callback 0xd56130
[debug] event_del: 0x7f70dc000df0 (fd -1), callback 0xd56130
[debug] event_process_active: event: 0x7f70dc000df0,call 0xd56130
[debug] event_del: 0x7f70dc000df0 (fd -1), callback 0xd56130
[debug] ../event.c: asked to terminate loop.
[debug] event_add: event: 0x7f70d4000df0 (fd -1),EV_TIMEOUT call 0xd56130
[debug] event_add: event 0x7f70d4000df0, timeout in 0 seconds 3 useconds, 
call 0xd56130
[debug] ../poll.c: poll reports 1
[debug] event_active: 0x1b094c0 (fd 5), res 2, callback 0x7f710f1d5f10
[debug] event_process_active: event: 0x1b094c0, EV_READ   call 0x7f710f1d5f10
[debug] timeout_next: event: 0x7f70d4000df0, in 0 seconds, 3 useconds
[debug] ../poll.c: poll reports 0
[debug] event_del: 0x7f70d4000df0 (fd -1), callback 0xd56130
[debug] timeout_process: event: 0x7f70d4000df0, call 0xd56

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

Hi Alexander,

Sorry for the delay,

> So far, yes

Great.

> NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate
> verification

Hm, maybe something with require-cert (is this SSL_VERIFY_PEER?) ?

> [pid 13305] --- SIGPIPE (Broken pipe) ---
> [pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
> -192, 0)= 5
> [pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
> 0x140750dd, 0)  = 1
> [pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60,
> 0x, -192, 0)   = 2
> [pid 13305] --- SIGSEGV (Segmentation fault) ---
> [pid 13292] +++ killed by SIGSEGV +++
> [pid 13278] --- SIGCHLD (Child exited) ---
> [pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60, 0, 11,
> 0)  = 5
> ../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
> (socket).failure(): Failed accept: connection error:
> error::lib(0):func(0):reason(0)
> ```

I don't like SIGPIPE here, seems that under ltrace it got SIGPIPE and
you do not ignore it, am I right [1]? Could you add code to ignore it and
retry again?

  [1]: 
https://github.com/apache/mesos/blob/abd1751aa9a305b0648f4a630acbc84e7d837783/3rdparty/libprocess/src/process.cpp#L1029

I think that something like this before start in bash will do the trick:
  trap '' SIGPIPE

Or maybe it will be better if you will remove #ifdef 0/#endif for
print_err() in bufferevent_openssl.c and recompile it with it?

And the best thing that you can do is to produce a sample that I can
compile and dig into, because I think that mesos is not the thing that I
want to compile from sources for this.

> I also verified our use of  `BEV_OPT_DEFER_CALLBACKS` and we do not use it
> for listening sockets, so my backtrace should be alright. Connecting
> sockets do use the flag however.

Regards,
Azat.

P.S. you email client again messed up with long lines from ltrace
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Alexander Rojas]

On 24. Jan 2018, at 16:53, Azat Khuzhin  wrote:

On Wed, Jan 24, 2018 at 09:55:15AM +0100, Alexander Rojas wrote:

I did the update, the problem changed from the accept to the recv call.
While now it can `accept()` socket connections, the `recv` callback fails.
I need to investigate further here, but it behaves just like the macosx
problem.


And every accept() fails, yes?


So far, yes


#2  0x7725dc48 in do_handshake (bev_ssl=0x7fffe4001a10) at
bufferevent_openssl.c:1053


Can you print the value of "err" here?

If not (it is optimized out or smth else) maybe you can use ltrace to
get it i.e.
 ltrace -f -e SSL_get_error ...


It was definitely optimized out, however with the ltrace this is what I get
from the test output:

 ```
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0129 11:01:11.040802 13278 openssl.cpp:504] CA file path is unspecified!
NOTE: Set CA file path with LIBPROCESS_SSL_CA_FILE=
I0129 11:01:11.040874 13278 openssl.cpp:509] CA directory path unspecified!
NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
I0129 11:01:11.040904 13278 openssl.cpp:514] Will not verify peer
certificate!
NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate
verification
I0129 11:01:11.040935 13278 openssl.cpp:520] Will only verify peer
certificate if presented!
NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate
verification
[pid 13292] --- Called exec() ---
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0129 11:01:11.090569 13292 openssl.cpp:504] CA file path is unspecified!
NOTE: Set CA file path with LIBPROCESS_SSL_CA_FILE=
I0129 11:01:11.090746 13292 openssl.cpp:509] CA directory path unspecified!
NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
I0129 11:01:11.090777 13292 openssl.cpp:514] Will not verify peer
certificate!
NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate
verification
I0129 11:01:11.090806 13292 openssl.cpp:520] Will only verify peer
certificate if presented!
NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate
verification
[pid 13305] --- SIGPIPE (Broken pipe) ---
[pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
-192, 0)= 5
[pid 13305] libevent_openssl-2.1.so.6->SSL_get_error(0x204e3d0, 0x,
0x140750dd, 0)  = 1
[pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60,
0x, -192, 0)   = 2
[pid 13305] --- SIGSEGV (Segmentation fault) ---
[pid 13304] +++ killed by SIGSEGV +++
[pid 13303] +++ killed by SIGSEGV +++
[pid 13302] +++ killed by SIGSEGV +++
[pid 13301] +++ killed by SIGSEGV +++
[pid 13300] +++ killed by SIGSEGV +++
[pid 13299] +++ killed by SIGSEGV +++
[pid 13298] +++ killed by SIGSEGV +++
[pid 13297] +++ killed by SIGSEGV +++
[pid 13296] +++ killed by SIGSEGV +++
[pid 13295] +++ killed by SIGSEGV +++
[pid 13294] +++ killed by SIGSEGV +++
[pid 13293] +++ killed by SIGSEGV +++
[pid 13305] +++ killed by SIGSEGV +++
[pid 13292] +++ killed by SIGSEGV +++
[pid 13278] --- SIGCHLD (Child exited) ---
[pid 13291] libevent_openssl-2.1.so.6->SSL_get_error(0x7f95b8000e60, 0, 11,
0)  = 5
../../../3rdparty/libprocess/src/tests/ssl_tests.cpp:254: Failure
(socket).failure(): Failed accept: connection error:
error::lib(0):func(0):reason(0)
```


(I verified and SSL_get_error() should be dynamic symbol, not macro, so
this should helps)


I also verified our use of  `BEV_OPT_DEFER_CALLBACKS` and we do not use it
for listening sockets, so my backtrace should be alright. Connecting
sockets do use the flag however.

Thanks again for your time.


P.S. BTW you client corrupts traces (by wrapping lines)
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

On Wed, Jan 24, 2018 at 09:55:15AM +0100, Alexander Rojas wrote:
> I did the update, the problem changed from the accept to the recv call.
> While now it can `accept()` socket connections, the `recv` callback fails.
> I need to investigate further here, but it behaves just like the macosx
> problem.

And every accept() fails, yes?

> #2  0x7725dc48 in do_handshake (bev_ssl=0x7fffe4001a10) at
> bufferevent_openssl.c:1053

Can you print the value of "err" here?

If not (it is optimized out or smth else) maybe you can use ltrace to
get it i.e.
  ltrace -f -e SSL_get_error ...

(I verified and SSL_get_error() should be dynamic symbol, not macro, so
this should helps)

P.S. BTW you client corrupts traces (by wrapping lines)
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Alexander Rojas]

Hi Azat,

thanks for looking into this.

On Mon, Jan 22, 2018 at 9:27 PM, Azat Khuzhin  wrote:

> On Mon, Jan 22, 2018 at 09:55:49AM +0100, Alexander Rojas wrote:
> > Hey guys,
> >
> > Over the past couple of weeks I've been struggling with an old bug of
> Apache Mesos related to our implementation of SSL Sockets which are based
> on libevent, you can see the whole implementation in [1] and [2]. It
> affects later versions of Linux, like Ubuntu 17.10 and OSX, full
> description of the packages at MESOS-8271 [3].
>
> Hi Alexander,
>
> Can you try with openssl 1.1? (from the issue I saw openssl 1.0)
>

I did the update, the problem changed from the accept to the recv call.
While now it can `accept()` socket connections, the `recv` callback fails.
I need to investigate further here, but it behaves just like the macosx
problem.

>
> > The problem can be triggered simply by updating libevent from 2.0.22 to
> 2.18.
>
> Wow, there were a lot of changes since 2.0.22. So I don't think that
> bisect is a good idea here.
>
> > In linux we see the issue when we call `accept()`. When we reach the
> event callback we keep getting `events & BEV_EVENT_ERROR && events &
> BEV_EVENT_READ` [4] however `EVUTIL_SOCKET_ERROR()` keeps returning 0, and
> `bufferevent_get_openssl_error(bev)` also shows no error.
> >
> > On the client we find that the connection was closed, but the symptoms
> are similar, we get `BEV_EVENT_ERROR` but no other diagnostics.
> >
> > I include a full stack trace of a run where the problem appears (Line
> numbers may be somewhat off due to debug output lines added). Any more info
> I am happy to contribute.
> >
> > ```
> > process::network::internal::LibeventSSLSocketImpl::accept_
> SSL_callback(process::network::internal::LibeventSSLSocketImpl::
> AcceptRequest*)::$_10::operator()(bufferevent*, short, void*) const
> (this=0x160eae0, bev=0x7fffe4001a10, events=33, arg=0x7fffc4000c10) at
> ../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1152
> > process::network::internal::LibeventSSLSocketImpl::accept_
> SSL_callback(process::network::internal::LibeventSSLSocketImpl::
> AcceptRequest*)::$_10::__invoke(bufferevent*, short, void*)
> (bev=0x7fffe4001a10, events=33, arg=0x7fffc4000c10) at
> ../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1143
> > ?? () from /usr/lib/x86_64-linux-gnu/libevent_openssl-2.1.so.6
> > ?? () from /usr/lib/x86_64-linux-gnu/libevent_openssl-2.1.so.6
> > ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
> > event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
>
> Can you install debug symbols for libevent? So that we can see line
> numbers? But it looks like it called from do_read() -> conn_closed(), if
> this is true I need "err" that obtained from SSL_get_error() in
> do_read().
>

I installed debug symbols, this is a newer stack trace:

```
#0
process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback(process::network::internal::LibeventSSLSocketImpl::AcceptRequest*)::$_10::operator()(bufferevent*,
short, void*) const (this=0x1608fe0, bev=0x7fffe4001a10, events=33,
arg=0x7fffc8000c10) at
../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1190
#1  0x00d1876a in
process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback(process::network::internal::LibeventSSLSocketImpl::AcceptRequest*)::$_10::__invoke(bufferevent*,
short, void*) (bev=0x7fffe4001a10, events=33, arg=0x7fffc8000c10) at
../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1101
#2  0x7725dc48 in do_handshake (bev_ssl=0x7fffe4001a10) at
bufferevent_openssl.c:1053
#3  0x7725dd80 in be_openssl_handshakeeventcb (fd=,
what=, ptr=0x7fffe4001a10) at bufferevent_openssl.c:1075
#4  0x765616aa in event_persist_closure (ev=,
base=0x1608d40) at event.c:1580
#5  event_process_active_single_queue (base=base@entry=0x1608d40,
activeq=0x15c98a0, max_to_process=max_to_process@entry=2147483647,
endtime=endtime@entry=0x0) at event.c:1639
#6  0x76562227 in event_process_active (base=0x1608d40) at
event.c:1738
#7  event_base_loop (base=0x1608d40, flags=) at event.c:1961
#8  0x00d4f8f5 in process::EventLoop::run () at
../../../3rdparty/libprocess/src/libevent.cpp:98
#9  0x00c82b47 in std::__invoke_impl
(__f=@0x15cb3e8: 0xd4f870 ) at
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/invoke.h:60
#10 0x00c82add in std::__invoke (__fn=@0x15cb3e8:
0xd4f870 ) at
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/invoke.h:95
#11 0x00c82ab5 in std::thread::_Invoker
>::_M_invoke<0ul> (this=0x15cb3e8) at
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/thread:234
#12 0x00c82a85 in std::thread::_Invoker
>::operator() (this=0x15cb3e8) at
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/thread:243
#13 0x00c82979 in
std::thread::_State_impl >
>::_M_run (this=0x15cb3e0) at
/usr/bin/../lib/gcc/x86_64-

Re: [Libevent-users] SSL problems after update to libeven 2.1.8

[Azat Khuzhin]

On Mon, Jan 22, 2018 at 09:55:49AM +0100, Alexander Rojas wrote:
> Hey guys,
> 
> Over the past couple of weeks I've been struggling with an old bug of Apache 
> Mesos related to our implementation of SSL Sockets which are based on 
> libevent, you can see the whole implementation in [1] and [2]. It affects 
> later versions of Linux, like Ubuntu 17.10 and OSX, full description of the 
> packages at MESOS-8271 [3].

Hi Alexander,

Can you try with openssl 1.1? (from the issue I saw openssl 1.0)

> The problem can be triggered simply by updating libevent from 2.0.22 to 2.18.

Wow, there were a lot of changes since 2.0.22. So I don't think that
bisect is a good idea here.

> In linux we see the issue when we call `accept()`. When we reach the event 
> callback we keep getting `events & BEV_EVENT_ERROR && events & 
> BEV_EVENT_READ` [4] however `EVUTIL_SOCKET_ERROR()` keeps returning 0, and 
> `bufferevent_get_openssl_error(bev)` also shows no error.
> 
> On the client we find that the connection was closed, but the symptoms are 
> similar, we get `BEV_EVENT_ERROR` but no other diagnostics.
> 
> I include a full stack trace of a run where the problem appears (Line numbers 
> may be somewhat off due to debug output lines added). Any more info I am 
> happy to contribute.
> 
> ```
> process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback(process::network::internal::LibeventSSLSocketImpl::AcceptRequest*)::$_10::operator()(bufferevent*,
>  short, void*) const (this=0x160eae0, bev=0x7fffe4001a10, events=33, 
> arg=0x7fffc4000c10) at 
> ../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1152
> process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback(process::network::internal::LibeventSSLSocketImpl::AcceptRequest*)::$_10::__invoke(bufferevent*,
>  short, void*) (bev=0x7fffe4001a10, events=33, arg=0x7fffc4000c10) at 
> ../../../3rdparty/libprocess/src/libevent_ssl_socket.cpp:1143
> ?? () from /usr/lib/x86_64-linux-gnu/libevent_openssl-2.1.so.6
> ?? () from /usr/lib/x86_64-linux-gnu/libevent_openssl-2.1.so.6
> ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
> event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6

Can you install debug symbols for libevent? So that we can see line
numbers? But it looks like it called from do_read() -> conn_closed(), if
this is true I need "err" that obtained from SSL_get_error() in
do_read().

And the next step will be to try to turn BEV_OPT_DEFER_CALLBACKS off (so
that we could see more reliable call sequence).

> process::EventLoop::run () at ../../../3rdparty/libprocess/src/libevent.cpp:98

Thanks,
Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.