Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Stephan Beal]

On Tue, Aug 30, 2016 at 10:10 PM, John Found  wrote:

> Well, my exact opinion is that as long as most web servers allow it, there
> is nothing wrong in compression the data by the CGI script. But IMHO, such
> behavior is half step beyond the CGI specification and can be qualified as
> a "common practice", having its roots probably in Apache project.
>

AFAIR, when CGI was commonplace (the mid/late 1990's) no webserver had the
ability to automatically compress output/throughput. i don't recall seeing
that ability until sometime around 2000. (That doesn't mean it didn't
exist, though, just that i didn't run into it in my own Apaches.)

Also, FYI, the communication between the web server (http server) and the
> CGI script is not by HTTP protocol.
> It is by CGI protocol, that resembles, but is still different from the
> HTTP protocol. In that context, the CGI script is NOT a http server and the
> web server is not a http proxy between the CGI script and the web browser.
>

FWIW, i disagree. You say yourself that communication between server and
CGI process is not via HTTP, but that the client/webserver communication is
(and the remote client, then, implicitly cannot talk directly to the CGI
script). Ergo, the web server is a go-between (i.e. proxy) for the CGI
script. The CGI's output is (or should be) opaque to the web server (and
thus the CGI must be free to compress/mutilate it however he likes).

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Warren Young]

On Aug 29, 2016, at 11:09 PM, John Found  wrote:
> 
>>> IMHO, the cgi script should not return "Content-Encoding” header
>> 
>> Of course it should.  Only the data source knows what content encoding it 
>> used. fcgiwrap and your “experimental” web server should just pass that 
>> header along, since they’re acting as proxies in this transaction.
>> 
> The CGI should not mess with the communication between the client and the 
> server. 
> It is specified in the mentioned RFC3875:
> 
>   The script MUST NOT return any header fields that relate to
>   client-side communication issues and could affect the server's
>   ability to send the response to the client.  The server MAY remove
>   any such header fields returned by the client.  It SHOULD resolve any
>   conflicts between header fields returned by the script and header
>   fields that it would otherwise send itself.
> 
> The compression falls exactly in the term "client-server communication”

You quote “client-server communication,” implying that it is a quote from the 
RFC, but it is not.  The term “client-server” never appears in the RFC.  That 
is probably because, in the case of CGI, the “server” is not an end-point, it 
is an intermediary.  The term “client-server” implies that the server is the 
end-point, the source of all data.  That is not the case in CGI.

What you do see a lot in that RFC — including in your own quote above — is talk 
about communication between the client and the *script*.  Your quote says that 
the script should not insert things into the stream that would confuse that 
intermediary server, but compressed content does not qualify.  The client said 
it could accept compressed content, so the server should not be surprised when 
the CGI script — Fossil in this case — provides that, especially when Fossil 
declares that it has done so in the Content-Encoding header.

Note also that many people use Fossil behind regular HTTP proxies via CGI, and 
it works fine there, too, indicating that those other HTTP servers do somehow 
manage to cope.

While it would be perfectly legal to leave compression to the HTTP proxy server 
sitting between the CGI script and the HTTP client, that is not the only way it 
must be done.

Why, if your view is correct, does the following Perl exist?

  https://metacpan.org/pod/CGI::Compress::Gzip
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[John Found]

On Mon, 29 Aug 2016 18:14:28 -0600
Warren Young  wrote:

> On Aug 29, 2016, at 5:10 PM, John Found  wrote:
> > 
> > I am running fossil in cgi mode on an experimental web server.
> 
> So your first guess as to the source of the problem is Fossil, not the 
> “experimental” web server?  Hm.  :)
> 
It was in order to determine what of the programs compresses the response. Our 
guess was that it is fossil and I asked in order to be sure.

> > When requesting the main page "/fossil/repo/name/", fossil returns 302 and 
> > redirects to "/fossil/repo/name/index". 
> > The 302 response, contains some message body "Moved" which is ignored by 
> > the browser. 
> 
> This is all perfectly fine.
> 
> > But if the request contains the header "Accept-Encoding: gzip", the 
> > response contains the header "Content-Encoding: gzip" and
> > it seems that fossil compresses the response body.
> 
> That is also as it should be.
> 
> > Is this the case, or the mess happens somewhere else. 
> 
> Your told HTTP that it could accept gzip-encoded data, and Fossil can provide 
> it, so it does, for efficiency.  The CPU time required to gzip data is 
> generally much lower than the increased network I/O time it takes to send the 
> uncompressed version of the data, so it’s faster overall.
> 
> > IMHO, the cgi script should not return "Content-Encoding” header
> 
> Of course it should.  Only the data source knows what content encoding it 
> used. fcgiwrap and your “experimental” web server should just pass that 
> header along, since they’re acting as proxies in this transaction.
> 
> > this is the function of the web server.
> 
> Only in the case of static content.
> 
The CGI should not mess with the communication between the client and the 
server. 
It is specified in the mentioned RFC3875:

   The script MUST NOT return any header fields that relate to
   client-side communication issues and could affect the server's
   ability to send the response to the client.  The server MAY remove
   any such header fields returned by the client.  It SHOULD resolve any
   conflicts between header fields returned by the script and header
   fields that it would otherwise send itself.

The compression falls exactly in the term "client-server communication" - the 
client specifies what compression it can accept and the server decide whether 
to provide it, or not. 

> > And this hangs, waiting for the body: 
> >  $curl -L -v -v -v --header 'Accept-Encoding: gzip' 
> > http://fresh.flatassembler.net/fossil/repo/fresh
> 
> It doesn’t hang when you connect to Fossil’s own internal HTTP server.  
> Therefore, the problem is either in fcgiwrap or your “experimental” web 
> server.

It is fixed now and nothing hangs anymore.

-- 
http://fresh.flatassembler.net
http://asm32.info
John Found 
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Warren Young]

On Aug 29, 2016, at 6:14 PM, Warren Young  wrote:
> 
> Your told HTTP…

  ^ client
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Warren Young]

On Aug 29, 2016, at 6:11 PM, John Found  wrote:
> 
> https://tools.ietf.org/html/rfc3875#section-6.3

That section doesn’t limit the type of headers that the CGI application can 
return.  It just defines the format of any headers that do exist, and requires 
that certain headers always get sent.

If it were otherwise, a CGI application could never return an X-Foo type header.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Warren Young]

On Aug 29, 2016, at 5:10 PM, John Found  wrote:
> 
> I am running fossil in cgi mode on an experimental web server.

So your first guess as to the source of the problem is Fossil, not the 
“experimental” web server?  Hm.  :)

> When requesting the main page "/fossil/repo/name/", fossil returns 302 and 
> redirects to "/fossil/repo/name/index". 
> The 302 response, contains some message body "Moved" which is ignored by the 
> browser. 

This is all perfectly fine.

> But if the request contains the header "Accept-Encoding: gzip", the response 
> contains the header "Content-Encoding: gzip" and
> it seems that fossil compresses the response body.

That is also as it should be.

> Is this the case, or the mess happens somewhere else. 

Your told HTTP that it could accept gzip-encoded data, and Fossil can provide 
it, so it does, for efficiency.  The CPU time required to gzip data is 
generally much lower than the increased network I/O time it takes to send the 
uncompressed version of the data, so it’s faster overall.

> IMHO, the cgi script should not return "Content-Encoding” header

Of course it should.  Only the data source knows what content encoding it used. 
fcgiwrap and your “experimental” web server should just pass that header along, 
since they’re acting as proxies in this transaction.

> this is the function of the web server.

Only in the case of static content.

> And this hangs, waiting for the body: 
>  $curl -L -v -v -v --header 'Accept-Encoding: gzip' 
> http://fresh.flatassembler.net/fossil/repo/fresh

It doesn’t hang when you connect to Fossil’s own internal HTTP server.  
Therefore, the problem is either in fcgiwrap or your “experimental” web server.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[John Found]

On Mon, 29 Aug 2016 19:27:00 -0400
Richard Hipp  wrote:

> On 8/29/16, John Found  wrote:
> > I am running fossil in cgi mode on an experimental web server. The web
> > server supports only FastCGI interface, so the fossil is spawned using
> > "fcgiwrap" tool. Anyway, we are hunting for a problem.
> >
> > When requesting the main page "/fossil/repo/name/", fossil returns 302 and
> > redirects to "/fossil/repo/name/index".
> > The 302 response, contains some message body "Moved" which is ignored by the
> > browser.
> >
> > But if the request contains the header "Accept-Encoding: gzip", the response
> > contains the header "Content-Encoding: gzip" and
> > it seems that fossil compresses the response body. Is this the case, or the
> > mess happens somewhere else.
> 
> Yes, Fossil compresses the reply in CGI mode.

OK, that should be enough to track the problem further. Thanks.

> 
> > IMHO, the cgi script should not return "Content-Encoding" header, because
> > this is the function of the web server.
> >
> 
> I have never before encountered that opinion.
> 
> It would be a problem for me, because my bespoke webserver used on the
> www.fossil-scm.org website does NOT attempt to compress CGI content.
> 

For me, it seems, such behavior conflict with the RFC 3875, but I might be 
wrong, because the specification is a little bit vague in that part: Section 
6.3: https://tools.ietf.org/html/rfc3875#section-6.3

> -- 
> D. Richard Hipp
> d...@sqlite.org
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users


-- 
http://fresh.flatassembler.net
http://asm32.info
John Found 
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Do, fossil, when running in cgi mode, compress the response body?

[Richard Hipp]

On 8/29/16, John Found  wrote:
> I am running fossil in cgi mode on an experimental web server. The web
> server supports only FastCGI interface, so the fossil is spawned using
> "fcgiwrap" tool. Anyway, we are hunting for a problem.
>
> When requesting the main page "/fossil/repo/name/", fossil returns 302 and
> redirects to "/fossil/repo/name/index".
> The 302 response, contains some message body "Moved" which is ignored by the
> browser.
>
> But if the request contains the header "Accept-Encoding: gzip", the response
> contains the header "Content-Encoding: gzip" and
> it seems that fossil compresses the response body. Is this the case, or the
> mess happens somewhere else.

Yes, Fossil compresses the reply in CGI mode.

> IMHO, the cgi script should not return "Content-Encoding" header, because
> this is the function of the web server.
>

I have never before encountered that opinion.

It would be a problem for me, because my bespoke webserver used on the
www.fossil-scm.org website does NOT attempt to compress CGI content.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users