Hey Sorin!
First and foremost, thanks a lot for your answer.
I started working on implementing a body filter for my needs, and am already
facing some strange issues. When my body filter is called on the subrequest I
sent, despite it being a body filter, I get a buffer chain containing the whole
upstream response, including headers. Since this is not a header filter, this
has gotten me quite confused... For what it's worth, my module doesn't even
register a header filter. Here is some gdb output:
Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at
/home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
207 {
(gdb) p *in->buf
$1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0,
shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file
= 0, flush = 0,
sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0,
num = 0}
(gdb)
Unless I'm missing something, this makes it a lot harder for me to actually
work on the body of those responses, as I would end up having to parse the HTTP
header to find the part I'm interested in.
I'm also at a loss as to the semantics of the last_buf and last_in_chain flags.
The module development guide doesn't contain any information about the
"last_in_chain" flag; it only talks about last_buf as a marker indicating that
the buffer chain is complete. Intuitively, I would think of "last_in_chain" as
a marker that we're at the end of the buffer list, but that doesn't make a lot
of sense since this would be redundant with the next pointer being NULL. The
ngx_buf.h header is equally unhelpful there and I couldn't locate a piece of
code that would make the semantics clear yet (some modules seem to consider
both flags as being equal, while others aren't).
Last but not least, is there any documentation as to when exactly the output
filters are being called? Mine seems to be called two times for the same
subrequest. Once with the buffer that I showed above, and a second time with a
buffer that is mostly all zero'ed, except for the "sync" and "last_in_chain"
flags both set to 1. That seems to be some kind of a flush request, but again,
I don't see any documentation on that.
Thanks in advance,
Maxime
________________________________
From: nginx-devel [nginx-devel-boun...@nginx.org] on behalf of Sorin Manole
[sorin.v.man...@gmail.com]
Sent: Monday, October 26, 2015 8:05 PM
To: nginx-devel@nginx.org
Subject: Re: Processing responses of unbounded sizes from upstream servers
Hey Maxime! The solution you described makes sense and should work. And yes,
you should probably use HTTP_AUX_FILTER_MODULES to register your handler.
You will have to do a little more juggling with buffers however. From what I
remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY is
to pass on the subrequest response to the client. You probably don't want that,
so you will have to catch the response buffers in the output handler and not
pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c
is probably the best source of inspiration for this.
Anyway, good luck!
2015-10-22 15:30 GMT+03:00 Maxime Henrion
<mhenr...@appnexus.com<mailto:mhenr...@appnexus.com>>:
Hello all,
I am currently developing a module that has to send a number of subrequests to
upstream servers, and aggregate them through application logic. I am currently
doing that through a post-subrequest handler, using the
NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to
receive very large responses from the upstream servers, and I end up with the
"upstream buffer is too small" error, even after bumping the buffer sizes a
number of times.
It is my understanding that if I drop this subrequest flag, nginx wouldn't try
to make the response fit in a single buffer anymore, but then I have no idea
how to get at that buffer chain - my post-subrequest handler only knows about
the single buffer in the upstream structure and I haven't been able to locate a
piece of code that would do things differently.
I suppose it would be possible to use an output filter instead of a
post-subrequest handler for that use case, would that make sense? And last but
not least, if I go down that road, can I just move my module "declaration" to
the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the
rest of module work fine, or will I need to use a second module for that?
Thanks a lot in advance!
Maxime
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org<mailto:nginx-devel@nginx.org>
http://mailman.nginx.org/mailman/listinfo/nginx-devel
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
