Re: [Libevent-users] filter bufferevents and timeout

[Bruno CARLUS]

Thank you Azat.

So I ran my test code with Libevent version (2.2.0-alpha-dev):
it seems that there is no difference because the call to bufferevent_flush 
triggers filter cb but still not read cb.

Concerning evbuffer, you mean using evbuffer_remove with a data_ptr refencing 
the mmaped file ?
That's what i'm doing from inside the read cb

- Mail original -
De: "Azat Khuzhin" <a3at.m...@gmail.com>
À: libevent-users@freehaven.net
Envoyé: Lundi 19 Mars 2018 19:25:08
Objet: Re: [Libevent-users] filter bufferevents and timeout

On Mon, Mar 19, 2018 at 12:26 PM, Bruno CARLUS <b.car...@ipnl.in2p3.fr> wrote:
> Hello,
> Ok thank you!
> For the moment I just bypassed the flush: because i know when my stream is 
> over i trigger a BEV_EVENT_EOF from the filter callback, and then handle the 
> remaining data in input buffer within the event callback.

So could you try next patch please?
  
https://github.com/azat/libevent/commit/ed38bfb4ea46e3a53c0dba0c7cabd0dfbd30f047.patch

Or, if you need the whole tree:
  https://github.com/azat/libevent.git be-filter-watermarks

(It is just a WIP, I will rebase it later).

> I was thinking of a file as a bufferevent output-only:
> we read data from a socket into input buffer, filter it then add it to the 
> output buffer which writes into the file.

Why not just using evbuffer for this?

  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] filter bufferevents and timeout

[Bruno CARLUS]

Hello,
Ok thank you!
For the moment I just bypassed the flush: because i know when my stream is over 
i trigger a BEV_EVENT_EOF from the filter callback, and then handle the 
remaining data in input buffer within the event callback.

I was thinking of a file as a bufferevent output-only:
we read data from a socket into input buffer, filter it then add it to the 
output buffer which writes into the file.
It could be with a filter bufferevent or a pair of bufferevents for which one 
could choose to enable input or output only?
Of course one can bypass that by writing into the file from inside the read 
callback as i'm doing for the time being...

Bruno

- Mail original -
De: "Azat Khuzhin" <a3at.m...@gmail.com>
À: libevent-users@freehaven.net
Envoyé: Lundi 19 Mars 2018 09:55:52
Objet: Re: [Libevent-users] filter bufferevents and timeout

On Tue, Mar 13, 2018 at 7:42 PM, Bruno CARLUS <b.car...@ipnl.in2p3.fr> wrote:
> Hello,
>
> I just identified a little problem in my code:
> I use a filter bufferevent wrapped on a socket bufferevent, for example:
>
> bev = bufferevent_socket_new(base, listener, BEV_OPT_CLOSE_ON_FREE);
> struct timeval timeout_read = {5, 0};
> bufferevent_set_timeouts(bev, _read, NULL);
> bufferevent_setwatermark(bev, EV_READ, 1500, 0);
>
> bev_filter = bufferevent_filter_new(bev, readandfiltercb, NULL, 
> BEV_OPT_CLOSE_ON_FREE, NULL, NULL);
> bufferevent_setwatermark(bev_filter, EV_READ, 3000, 0);
> bufferevent_setcb(bev_filter, freadcb, fwritecb, eventcb, NULL);
>
> I'm sending 1000 bytes packets on this socket.
> readandfiltercb is called every 2 packets: it filters data and copy it to the 
> bev_filter input buffer.
> freadcb is then called ok.
>
> But when udp stream is over, it remains one 1000 bytes packet in the buffer.
> eventcb is called because of the timeout: then it flushes bufferevent
> readandfiltercb is called ok: it filters and copies last bytes to bev_filter 
> input buffer.
> But then freadcb is not called at all.
>
> Is it normal?
> How to force bypass the watermark and flush bev_filter input buffer as well?

Hello,

I don't think so, my guess is that during flush it doesn't call readcb
(in be_filter_flush() there is no bufferevent_trigger_nolock_() like
in be_filter_read_nolock_()).
I will write a test today and will come back to you later, in the evening.

> Last point: Is the underlying fd of a bufferevent can be a mmap file?
> I could not find reference to the use of EVBUFFER_SENDFILE in the doc...

How you will poll mmaped fd?
Can you explain this a little bit more? Maybe I'm missing something.

-- 
  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] filter bufferevents and timeout

[Azat Khuzhin]

On Tue, Mar 13, 2018 at 7:42 PM, Bruno CARLUS  wrote:
> Hello,
>
> I just identified a little problem in my code:
> I use a filter bufferevent wrapped on a socket bufferevent, for example:
>
> bev = bufferevent_socket_new(base, listener, BEV_OPT_CLOSE_ON_FREE);
> struct timeval timeout_read = {5, 0};
> bufferevent_set_timeouts(bev, _read, NULL);
> bufferevent_setwatermark(bev, EV_READ, 1500, 0);
>
> bev_filter = bufferevent_filter_new(bev, readandfiltercb, NULL, 
> BEV_OPT_CLOSE_ON_FREE, NULL, NULL);
> bufferevent_setwatermark(bev_filter, EV_READ, 3000, 0);
> bufferevent_setcb(bev_filter, freadcb, fwritecb, eventcb, NULL);
>
> I'm sending 1000 bytes packets on this socket.
> readandfiltercb is called every 2 packets: it filters data and copy it to the 
> bev_filter input buffer.
> freadcb is then called ok.
>
> But when udp stream is over, it remains one 1000 bytes packet in the buffer.
> eventcb is called because of the timeout: then it flushes bufferevent
> readandfiltercb is called ok: it filters and copies last bytes to bev_filter 
> input buffer.
> But then freadcb is not called at all.
>
> Is it normal?
> How to force bypass the watermark and flush bev_filter input buffer as well?

Hello,

I don't think so, my guess is that during flush it doesn't call readcb
(in be_filter_flush() there is no bufferevent_trigger_nolock_() like
in be_filter_read_nolock_()).
I will write a test today and will come back to you later, in the evening.

> Last point: Is the underlying fd of a bufferevent can be a mmap file?
> I could not find reference to the use of EVBUFFER_SENDFILE in the doc...

How you will poll mmaped fd?
Can you explain this a little bit more? Maybe I'm missing something.

-- 
  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] filter bufferevents and timeout

[Bruno CARLUS]

Hi,

I have few questions as I go further in my project:

- regarding bufferevent_filter: I need to retrieve udp data on a socket, filter 
it according to my protocol and write received data to a file. I was thinking 
on a read cb tranfering data (evbuffer_add_buffer or 
evbuffer_add_buffer_reference) from the input buffer of my bufferevent_filter 
to the output_buffer of another bufferevent linked to my file (mmaped). Is it 
the right method? How to minimize memory allocation in these operations for a 
high throughput?

- i read that bufferevent does not support udp yet but it seems to work in my 
tests (i receive data only, never send) ... What are exacly the limitations ?

Thanks again for your help!
Bruno.

- Mail original -
De: "Azat Khuzhin" <a3at.m...@gmail.com>
À: libevent-users@freehaven.net
Envoyé: Lundi 6 Novembre 2017 12:30:05
Objet: Re: [Libevent-users] filter bufferevents and timeout

Hi,

> But when done properly it works indeed, thank you!

Great

> - in my input filter callback, is the destination buffer the input buffer of 
> my bufferevent_filter ?

yep, the src is the underlying input, and the dst is the input of the
filtered bufferevent

> - how is used the buffbufferevent_filter_result returned by the input filter 
> callback ?

Kind of the same as for output filter:
- it will call you input callback until you returns BEV_OK (i.e. while
you are processing something) and the underlying input has data
- if it processed something it resets timeouts, and triggers readcb of
filtered bufferevent.

(All this for the case if the filtered input does not have watermarks,
since in this case it can stop earlier)

  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] filter bufferevents and timeout

[Bruno CARLUS]

Ok thank you!
I understand it better now.

Bruno.

- Mail original -
De: "Azat Khuzhin" <a3at.m...@gmail.com>
À: libevent-users@freehaven.net
Envoyé: Lundi 6 Novembre 2017 12:30:05
Objet: Re: [Libevent-users] filter bufferevents and timeout

Hi,

> But when done properly it works indeed, thank you!

Great

> - in my input filter callback, is the destination buffer the input buffer of 
> my bufferevent_filter ?

yep, the src is the underlying input, and the dst is the input of the
filtered bufferevent

> - how is used the buffbufferevent_filter_result returned by the input filter 
> callback ?

Kind of the same as for output filter:
- it will call you input callback until you returns BEV_OK (i.e. while
you are processing something) and the underlying input has data
- if it processed something it resets timeouts, and triggers readcb of
filtered bufferevent.

(All this for the case if the filtered input does not have watermarks,
since in this case it can stop earlier)

  Azat.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

[Libevent-users] filter bufferevents and timeout

[Bruno CARLUS]

Hello, 
is there any way to catch and handle a timeout issued by the underlying 
bufferevent when using a filter bufferevent ? 
Thanks! 

Regards, 
Bruno.