Re: mod_mime_magic, gzipped tarballs and Docker
Accept On Jan 19, 2016 4:22 AM, "William A Rowe Jr" wrote: > On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža wrote: > >> On 01/08/2016 07:44 PM, William A Rowe Jr wrote: >> >>> Do we have to repeat the softmagic call if checkzmagic resolves to >>> x-gzip/x-deflate and the internal content type needs to be deciphered? >>> >> >> That's true. >> >> I think that Yann's patch moving the zmagic call after the softmagic call >> would just mean that zmagic won't be called at all if the softmagic >> recognizes the file format. >> >> We would have to check the softmagic result by something similar to >> "magic_rsl_to_request" and if it's type we want to decompress, we would >> have to run zmagic. >> >> Before really trying to do so, I want to ask if I understand the >> reasoning right. Do we consider this way because users can then remove >> x-gzip from mime magic and will be able to use it to disable the >> mod_mime_magic behaviour discussed in this thread? >> > > Yes... and in a more flexible manner that allows us to override any > compression mode, not only deflate, by simply tweaking the magic entries. > > Putting magic file entry overrides into the mod_mime_magic directives is a > lot more interesting than simply toggling compression recognition. >
Re: mod_mime_magic, gzipped tarballs and Docker
On 01/18/2016 09:22 PM, William A Rowe Jr wrote: On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža mailto:[email protected]>> wrote: On 01/08/2016 07:44 PM, William A Rowe Jr wrote: Do we have to repeat the softmagic call if checkzmagic resolves to x-gzip/x-deflate and the internal content type needs to be deciphered? That's true. I think that Yann's patch moving the zmagic call after the softmagic call would just mean that zmagic won't be called at all if the softmagic recognizes the file format. We would have to check the softmagic result by something similar to "magic_rsl_to_request" and if it's type we want to decompress, we would have to run zmagic. Before really trying to do so, I want to ask if I understand the reasoning right. Do we consider this way because users can then remove x-gzip from mime magic and will be able to use it to disable the mod_mime_magic behaviour discussed in this thread? Yes... and in a more flexible manner that allows us to override any compression mode, not only deflate, by simply tweaking the magic entries. Putting magic file entry overrides into the mod_mime_magic directives is a lot more interesting than simply toggling compression recognition. Okay... I have something half-way right now. Attached patch changes the order of softmagic and zmagic. When softmagic recognizes a file, the zmagic is called. zmagic method checks if the file has been recognized as "x-gzip" and it tries to uncompress it and call "tryit" again with the uncompressed data as before. I'm now playing with an idea to extend the magic file so we wouldn't have to keep the "x-gzip" -> "gzip -dcq" relation hardcoded, but it could be defined in the magic file instead. William, is the patch close to the behaviour you were describing? Regards, Jan Kaluza Index: modules/metadata/mod_mime_magic.c === --- modules/metadata/mod_mime_magic.c (revision 1725450) +++ modules/metadata/mod_mime_magic.c (working copy) @@ -463,6 +463,8 @@ magic_rsl *head; /* result string list */ magic_rsl *tail; unsigned suf_recursion; /* recursion depth in suffix check */ +char *type; /* Content-Type cached value */ +char *encoding; /* Content-Encoding cached value */ } magic_req_rec; /* @@ -534,6 +536,8 @@ sizeof(magic_req_rec)); req_dat->head = req_dat->tail = (magic_rsl *) NULL; +req_dat->type = NULL; +req_dat->encoding = NULL; ap_set_module_config(r->request_config, &mime_magic_module, req_dat); return req_dat; } @@ -656,13 +660,26 @@ return result; } -/* states for the state-machine algorithm in magic_rsl_to_request() */ +static int magic_rsl_clear(request_rec *r) +{ +magic_req_rec *req_dat = (magic_req_rec *) +ap_get_module_config(r->request_config, &mime_magic_module); +req_dat->head = NULL; +req_dat->tail = NULL; +req_dat->type = NULL; +req_dat->encoding = NULL; + +/* success */ +return 0; +} + +/* states for the state-machine algorithm in magic_rsl_parse() */ typedef enum { rsl_leading_space, rsl_type, rsl_subtype, rsl_separator, rsl_encoding } rsl_states; /* process the RSL and set the MIME info in the request record */ -static int magic_rsl_to_request(request_rec *r) +static int magic_rsl_parse(request_rec *r, char **type, char **encoding) { int cur_frag, /* current fragment number/counter */ cur_pos, /* current position within fragment */ @@ -673,7 +690,6 @@ encoding_pos, /* content encoding starting point: position */ encoding_len; /* content encoding length */ -char *tmp; magic_rsl *frag; /* list-traversal pointer */ rsl_states state; @@ -686,6 +702,13 @@ return DECLINED; } +/* check for cached values filled in by previous run */ +if (req_dat->type) { +*type = req_dat->type; +*encoding = req_dat->encoding; +return OK; +} + /* start searching for the type and encoding */ state = rsl_leading_space; type_frag = type_pos = type_len = 0; @@ -785,24 +808,24 @@ } /* save the info in the request record */ -tmp = rsl_strdup(r, type_frag, type_pos, type_len); +*type = rsl_strdup(r, type_frag, type_pos, type_len); /* XXX: this could be done at config time I'm sure... but I'm * confused by all this magic_rsl stuff. -djg */ -ap_content_type_tolower(tmp); -ap_set_content_type(r, tmp); +ap_content_type_tolower(*type); +req_dat->type = *type; if (state == rsl_encoding) { -tmp = rsl_strdup(r, encoding_frag, +*encoding = rsl_strdup(r, encoding_frag, encoding_pos, encoding_len); /* XXX: this could be done at config time
Re: mod_mime_magic, gzipped tarballs and Docker
On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža wrote: > On 01/08/2016 07:44 PM, William A Rowe Jr wrote: > >> Do we have to repeat the softmagic call if checkzmagic resolves to >> x-gzip/x-deflate and the internal content type needs to be deciphered? >> > > That's true. > > I think that Yann's patch moving the zmagic call after the softmagic call > would just mean that zmagic won't be called at all if the softmagic > recognizes the file format. > > We would have to check the softmagic result by something similar to > "magic_rsl_to_request" and if it's type we want to decompress, we would > have to run zmagic. > > Before really trying to do so, I want to ask if I understand the reasoning > right. Do we consider this way because users can then remove x-gzip from > mime magic and will be able to use it to disable the mod_mime_magic > behaviour discussed in this thread? > Yes... and in a more flexible manner that allows us to override any compression mode, not only deflate, by simply tweaking the magic entries. Putting magic file entry overrides into the mod_mime_magic directives is a lot more interesting than simply toggling compression recognition.
Re: mod_mime_magic, gzipped tarballs and Docker
Clash of clans
On Jan 8, 2016 8:43 PM, "Jim Jagielski" wrote:
> -0.9...
>
> This seems a very heavy solution to a specific one-off problem.
>
> > On Jan 8, 2016, at 4:27 AM, Yann Ylavic wrote:
> >
> > Hi,
> >
> > On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža wrote:
> >>
> >> Content-Type: application/x-tar
> >> Content-Encoding: x-gzip
> > []
> >>
> >> So, the mod_mime_magic is saying here that the body is tarball encoded
> by
> >> gzip.
> >
> > AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> > the above headers.
> >
> >>
> >> But I think even we are right here, the Content-Encoding should be used
> only
> >> when httpd itself encodes the original content, otherwise it confuses
> >> clients (I've asked few people and it confuses some web browsers too -
> I can
> >> get more info if needed.).
> >
> > Agreed, this looks wrong, the browsers will likely use the .tar file
> > instead (this adds unnecessary cycles to both the server and client).
> >
> >>
> >> Maybe we could stop setting Content-Encoding in mod_mime_magic and just
> use
> >> Content-Type?
> >
> > Since it's always been there, we probably should add a new
> > mod_mime_magic directive to control the behaviour and avoid breaking
> > cases.
> >
> > Something along:
> >
> > Index: modules/metadata/mod_mime_magic.c
> > ===
> > --- modules/metadata/mod_mime_magic.c(revision 1723283)
> > +++ modules/metadata/mod_mime_magic.c(working copy)
> > @@ -456,6 +456,7 @@ typedef struct {
> > const char *magicfile;/* where magic be found */
> > struct magic *magic; /* head of magic config list */
> > struct magic *last;
> > +int uncompress;
> > } magic_server_config_rec;
> >
> > /* per-request info */
> > @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> > {
> > AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> > "Path to MIME Magic file (in file(1) format)"),
> > +AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> > + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress),
> RSRC_CONF,
> > + "Whether MIME should try to render uncompressed content by
> recognizing "
> > + "the underlying type, or not (default)"),
> > {NULL}
> > };
> >
> > @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
> >
> > static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> > {
> > +magic_server_config_rec *conf = (magic_server_config_rec *)
> > +ap_get_module_config(r->server->module_config,
> > + &mime_magic_module);
> > unsigned char *newbuf;
> > int newsize;
> > int i;
> > @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char
> *b
> > if (i == ncompr)
> > return 0;
> >
> > -if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> > -/* set encoding type in the request record */
> > -r->content_encoding = compr[i].encoding;
> > -
> > +if (!conf->uncompress) {
> > +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> > + compr[i].encoding, NULL));
> > +}
> > +else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> > newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> > /* Try to detect the content type of the uncompressed data */
> > if (tryit(r, newbuf, newsize, 0) != OK) {
> > +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> > + compr[i].encoding, NULL));
> > return 0;
> > }
> > +/* set encoding type in the request record */
> > +r->content_encoding = compr[i].encoding;
> > }
> > return 1;
> > }
> > --
> >
> > Regards,
> > Yann.
>
>
Re: mod_mime_magic, gzipped tarballs and Docker
On 01/08/2016 07:44 PM, William A Rowe Jr wrote: Do we have to repeat the softmagic call if checkzmagic resolves to x-gzip/x-deflate and the internal content type needs to be deciphered? That's true. I think that Yann's patch moving the zmagic call after the softmagic call would just mean that zmagic won't be called at all if the softmagic recognizes the file format. We would have to check the softmagic result by something similar to "magic_rsl_to_request" and if it's type we want to decompress, we would have to run zmagic. Before really trying to do so, I want to ask if I understand the reasoning right. Do we consider this way because users can then remove x-gzip from mime magic and will be able to use it to disable the mod_mime_magic behaviour discussed in this thread? Regards, Jan Kaluza If so, a tweak to this patch sounds like a winner, and could selectively recognize different inflation streams including bzip2 etc. On Fri, Jan 8, 2016 at 10:57 AM, Yann Ylavic mailto:[email protected]>> wrote: On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic mailto:[email protected]>> wrote: > On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr mailto:[email protected]>> wrote: >> >> Agreed it is configuration, but cant we simply tweak our recommended >> conf/magic >> file??? >> >> # standard unix compress >> # Enable the alternate line below to present gzip content as a transfer >> encoded >> # stream of the underlying content; >> #0 string \037\235application/octet-stream >> x-compress >> 0 string \037\235application/octet-stream >> >> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver) >> # Enable the alternate line below to present gzip content as a transfer >> encoded >> # stream of the underlying content; >> #0 string \037\213application/octet-stream >> x-gzip >> 0 string \037\213application/octet-stream >> >> WDYT? > > I wasn't aware of conf/magic file, but it seems not used by zmagic() > which hardcodes its types. > I may be missing something though... Looks like it is already like this in conf/magic of 2.4.x and 2.2.x. My bet is that zmagic() is run before softmagic(), so the magic file is of no help. Maybe a simpler patch could be: Index: modules/metadata/mod_mime_magic.c === --- modules/metadata/mod_mime_magic.c(revision 1723283) +++ modules/metadata/mod_mime_magic.c(working copy) @@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu int checkzmagic) { /* - * Try compression stuff - */ -if (checkzmagic == 1) { -if (zmagic(r, buf, nb) == 1) -return OK; -} - -/* * try tests in /etc/magic (or surrogate magic file) */ if (softmagic(r, buf, nb) == 1) @@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu return OK; /* + * Try compression stuff + */ +if (checkzmagic == 1) { +if (zmagic(r, buf, nb) == 1) +return OK; +} + +/* * abandon hope, all ye who remain here */ return DECLINED; --
Re: mod_mime_magic, gzipped tarballs and Docker
Alternatively, it could use a global extern var and avoid the dup. > On Jan 8, 2016, at 3:00 PM, Jim Jagielski wrote: > > Looks right to me... using the address of a local var is nasty > and broken and for sure will cause grief. > >> On Jan 8, 2016, at 2:30 PM, Christophe JAILLET >> wrote: >> >> Hi, >> >> not directly related to this discussion, but while looking at >> mod_mime_magic, could someone have a look at: >> http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700 >> >> >> I also tried to get some feedback some time ago >> (http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that time, >> got no success. >> >> The fix looks obvious to me but I never took enough time to test it. >> Any one interested? >> >> Best regards, >> CJ >> >> >
Re: mod_mime_magic, gzipped tarballs and Docker
Looks right to me... using the address of a local var is nasty and broken and for sure will cause grief. > On Jan 8, 2016, at 2:30 PM, Christophe JAILLET > wrote: > > Hi, > > not directly related to this discussion, but while looking at mod_mime_magic, > could someone have a look at: > http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700 > > > I also tried to get some feedback some time ago > (http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that time, > got no success. > > The fix looks obvious to me but I never took enough time to test it. > Any one interested? > > Best regards, > CJ > >
Re: mod_mime_magic, gzipped tarballs and Docker
Hi, not directly related to this discussion, but while looking at mod_mime_magic, could someone have a look at: http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700 I also tried to get some feedback some time ago (http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that time, got no success. The fix looks obvious to me but I never took enough time to test it. Any one interested? Best regards, CJ
Re: mod_mime_magic, gzipped tarballs and Docker
Do we have to repeat the softmagic call if checkzmagic resolves to
x-gzip/x-deflate and the internal content type needs to be deciphered?
If so, a tweak to this patch sounds like a winner, and could selectively
recognize different inflation streams including bzip2 etc.
On Fri, Jan 8, 2016 at 10:57 AM, Yann Ylavic wrote:
> On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic wrote:
> > On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr
> wrote:
> >>
> >> Agreed it is configuration, but cant we simply tweak our recommended
> >> conf/magic
> >> file???
> >>
> >> # standard unix compress
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\235application/octet-stream
> >> x-compress
> >> 0 string \037\235application/octet-stream
> >>
> >> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\213application/octet-stream
> >> x-gzip
> >> 0 string \037\213application/octet-stream
> >>
> >> WDYT?
> >
> > I wasn't aware of conf/magic file, but it seems not used by zmagic()
> > which hardcodes its types.
> > I may be missing something though...
>
> Looks like it is already like this in conf/magic of 2.4.x and 2.2.x.
> My bet is that zmagic() is run before softmagic(), so the magic file
> is of no help.
>
> Maybe a simpler patch could be:
>
> Index: modules/metadata/mod_mime_magic.c
> ===
> --- modules/metadata/mod_mime_magic.c(revision 1723283)
> +++ modules/metadata/mod_mime_magic.c(working copy)
> @@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu
> int checkzmagic)
> {
> /*
> - * Try compression stuff
> - */
> -if (checkzmagic == 1) {
> -if (zmagic(r, buf, nb) == 1)
> -return OK;
> -}
> -
> -/*
> * try tests in /etc/magic (or surrogate magic file)
> */
> if (softmagic(r, buf, nb) == 1)
> @@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu
> return OK;
>
> /*
> + * Try compression stuff
> + */
> +if (checkzmagic == 1) {
> +if (zmagic(r, buf, nb) == 1)
> +return OK;
> +}
> +
> +/*
> * abandon hope, all ye who remain here
> */
> return DECLINED;
> --
>
Re: mod_mime_magic, gzipped tarballs and Docker
On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic wrote:
> On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr wrote:
>>
>> Agreed it is configuration, but cant we simply tweak our recommended
>> conf/magic
>> file???
>>
>> # standard unix compress
>> # Enable the alternate line below to present gzip content as a transfer
>> encoded
>> # stream of the underlying content;
>> #0 string \037\235application/octet-stream
>> x-compress
>> 0 string \037\235application/octet-stream
>>
>> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
>> # Enable the alternate line below to present gzip content as a transfer
>> encoded
>> # stream of the underlying content;
>> #0 string \037\213application/octet-stream
>> x-gzip
>> 0 string \037\213application/octet-stream
>>
>> WDYT?
>
> I wasn't aware of conf/magic file, but it seems not used by zmagic()
> which hardcodes its types.
> I may be missing something though...
Looks like it is already like this in conf/magic of 2.4.x and 2.2.x.
My bet is that zmagic() is run before softmagic(), so the magic file
is of no help.
Maybe a simpler patch could be:
Index: modules/metadata/mod_mime_magic.c
===
--- modules/metadata/mod_mime_magic.c(revision 1723283)
+++ modules/metadata/mod_mime_magic.c(working copy)
@@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu
int checkzmagic)
{
/*
- * Try compression stuff
- */
-if (checkzmagic == 1) {
-if (zmagic(r, buf, nb) == 1)
-return OK;
-}
-
-/*
* try tests in /etc/magic (or surrogate magic file)
*/
if (softmagic(r, buf, nb) == 1)
@@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu
return OK;
/*
+ * Try compression stuff
+ */
+if (checkzmagic == 1) {
+if (zmagic(r, buf, nb) == 1)
+return OK;
+}
+
+/*
* abandon hope, all ye who remain here
*/
return DECLINED;
--
Re: mod_mime_magic, gzipped tarballs and Docker
It just seems like feature-creep and YetAnotherDirective for something that is better handled some other way. As OtherBill notes, a small edit to the magic file "fixes" this. > On Jan 8, 2016, at 11:16 AM, Yann Ylavic wrote: > > On Fri, Jan 8, 2016 at 1:43 PM, Jim Jagielski wrote: >> -0.9... >> >> This seems a very heavy solution to a specific one-off problem. > > Not sure this is the patch which is heavy here... > Anytime mod_mime_magic recognizes the type of a gzip file (by the > magic bytes), it unconditionally starts a new piped-process to > uncompress the file so to determine the type of the gzipped content. > Then, it forces "Content-Encoding: x-gzip" and (eg.) "Content-Type: > x-tar", which causes the browser to think this was an original tarball > compressed by the protocol, and then uncompress it for the user to get > the (supposedly) original file. > (Btw the "Content-Encoding" encoding is set whether or not the > underlying type has been determined successfully, that looks buggy to > me since the browser could get a C-E w/o C-T, so I fixed it in the > proposed patch too). > > My patch simply disables this by default (and provides an opt-out to > restore legacy behaviour, but it could easily take the other way too > to preserve existing), such that now mod_mime_magic can also set only > "Content-Type: x-gzip" and let (real) original go out with no special > treatment. > > The patch is actually quite small to handle this IMHO...
Re: mod_mime_magic, gzipped tarballs and Docker
On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr wrote: > On Fri, Jan 8, 2016 at 3:27 AM, Yann Ylavic wrote: >> >> >> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža wrote: >> > >> > Content-Type: application/x-tar >> > Content-Encoding: x-gzip >> > >> > So, the mod_mime_magic is saying here that the body is tarball encoded >> > by >> > gzip. >> >> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds >> the above headers. > > > Well, not strictly true, mod_deflate uncompresses it ;-) Hmm, not sure mod_deflate is in the place, at least it doesn't need to AFAICT from the code. But sure mod_deflate could be configured to uncompress once more :) > >> > Maybe we could stop setting Content-Encoding in mod_mime_magic and just >> > use >> > Content-Type? >> >> Since it's always been there, we probably should add a new >> mod_mime_magic directive to control the behaviour and avoid breaking >> cases. > > > Agreed it is configuration, but cant we simply tweak our recommended > conf/magic > file??? > > # standard unix compress > # Enable the alternate line below to present gzip content as a transfer > encoded > # stream of the underlying content; > #0 string \037\235application/octet-stream > x-compress > 0 string \037\235application/octet-stream > > # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver) > # Enable the alternate line below to present gzip content as a transfer > encoded > # stream of the underlying content; > #0 string \037\213application/octet-stream > x-gzip > 0 string \037\213application/octet-stream > > WDYT? I wasn't aware of conf/magic file, but it seems not used by zmagic() which hardcodes its types. I may be missing something though...
Re: mod_mime_magic, gzipped tarballs and Docker
On Fri, Jan 8, 2016 at 1:43 PM, Jim Jagielski wrote: > -0.9... > > This seems a very heavy solution to a specific one-off problem. Not sure this is the patch which is heavy here... Anytime mod_mime_magic recognizes the type of a gzip file (by the magic bytes), it unconditionally starts a new piped-process to uncompress the file so to determine the type of the gzipped content. Then, it forces "Content-Encoding: x-gzip" and (eg.) "Content-Type: x-tar", which causes the browser to think this was an original tarball compressed by the protocol, and then uncompress it for the user to get the (supposedly) original file. (Btw the "Content-Encoding" encoding is set whether or not the underlying type has been determined successfully, that looks buggy to me since the browser could get a C-E w/o C-T, so I fixed it in the proposed patch too). My patch simply disables this by default (and provides an opt-out to restore legacy behaviour, but it could easily take the other way too to preserve existing), such that now mod_mime_magic can also set only "Content-Type: x-gzip" and let (real) original go out with no special treatment. The patch is actually quite small to handle this IMHO...
Re: mod_mime_magic, gzipped tarballs and Docker
On Fri, Jan 8, 2016 at 3:27 AM, Yann Ylavic wrote:
>
> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža wrote:
> >
> > Content-Type: application/x-tar
> > Content-Encoding: x-gzip
> >
> > So, the mod_mime_magic is saying here that the body is tarball encoded by
> > gzip.
>
> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> the above headers.
>
Well, not strictly true, mod_deflate uncompresses it ;-)
> But I think even we are right here, the Content-Encoding should be used
> only
> > when httpd itself encodes the original content, otherwise it confuses
> > clients (I've asked few people and it confuses some web browsers too - I
> can
> > get more info if needed.).
>
> Agreed, this looks wrong, the browsers will likely use the .tar file
> instead (this adds unnecessary cycles to both the server and client).
>
I've observed this on many clients downloading .tar.gz and storing the file
as the
decompressed .tar when I had configured .gz to report Content-Encoding:
x-gzip,
and this is simply using mod_mime configuration directives. But that was
shooting
myself in the foot, had not stopped to consider what mod_mime_magic might
do.
> Maybe we could stop setting Content-Encoding in mod_mime_magic and just
> use
> > Content-Type?
>
> Since it's always been there, we probably should add a new
> mod_mime_magic directive to control the behaviour and avoid breaking
> cases.
>
Agreed it is configuration, but cant we simply tweak our recommended
conf/magic
file???
# standard unix compress
# Enable the alternate line below to present gzip content as a transfer
encoded
# stream of the underlying content;
#0 string \037\235application/octet-stream
x-compress
0 string \037\235application/octet-stream
# gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
# Enable the alternate line below to present gzip content as a transfer
encoded
# stream of the underlying content;
#0 string \037\213application/octet-stream
x-gzip
0 string \037\213application/octet-stream
WDYT?
> Something along:
>
> Index: modules/metadata/mod_mime_magic.c
> ===
> --- modules/metadata/mod_mime_magic.c(revision 1723283)
> +++ modules/metadata/mod_mime_magic.c(working copy)
> @@ -456,6 +456,7 @@ typedef struct {
> const char *magicfile;/* where magic be found */
> struct magic *magic; /* head of magic config list */
> struct magic *last;
> +int uncompress;
> } magic_server_config_rec;
>
> /* per-request info */
> @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> {
> AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> "Path to MIME Magic file (in file(1) format)"),
> +AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
> + "Whether MIME should try to render uncompressed content by
> recognizing "
> + "the underlying type, or not (default)"),
> {NULL}
> };
>
> @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
>
> static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> {
> +magic_server_config_rec *conf = (magic_server_config_rec *)
> +ap_get_module_config(r->server->module_config,
> + &mime_magic_module);
> unsigned char *newbuf;
> int newsize;
> int i;
> @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
> if (i == ncompr)
> return 0;
>
> -if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> -/* set encoding type in the request record */
> -r->content_encoding = compr[i].encoding;
> -
> +if (!conf->uncompress) {
> +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> +}
> +else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> /* Try to detect the content type of the uncompressed data */
> if (tryit(r, newbuf, newsize, 0) != OK) {
> +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> return 0;
> }
> +/* set encoding type in the request record */
> +r->content_encoding = compr[i].encoding;
> }
> return 1;
> }
>
Just trying to understand why conf/magic can't be the lever for this
behavior.
Are you trying to trip this behavior on a per-file/per-directory basis?
That would
be one case where keeping two entire conf/magic[12] files loaded would seem
excessive.
What about adding a MimeMagicItem directive to add/modify just a few lines
not in the master mime magic file
Re: mod_mime_magic, gzipped tarballs and Docker
-0.9...
This seems a very heavy solution to a specific one-off problem.
> On Jan 8, 2016, at 4:27 AM, Yann Ylavic wrote:
>
> Hi,
>
> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža wrote:
>>
>> Content-Type: application/x-tar
>> Content-Encoding: x-gzip
> []
>>
>> So, the mod_mime_magic is saying here that the body is tarball encoded by
>> gzip.
>
> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> the above headers.
>
>>
>> But I think even we are right here, the Content-Encoding should be used only
>> when httpd itself encodes the original content, otherwise it confuses
>> clients (I've asked few people and it confuses some web browsers too - I can
>> get more info if needed.).
>
> Agreed, this looks wrong, the browsers will likely use the .tar file
> instead (this adds unnecessary cycles to both the server and client).
>
>>
>> Maybe we could stop setting Content-Encoding in mod_mime_magic and just use
>> Content-Type?
>
> Since it's always been there, we probably should add a new
> mod_mime_magic directive to control the behaviour and avoid breaking
> cases.
>
> Something along:
>
> Index: modules/metadata/mod_mime_magic.c
> ===
> --- modules/metadata/mod_mime_magic.c(revision 1723283)
> +++ modules/metadata/mod_mime_magic.c(working copy)
> @@ -456,6 +456,7 @@ typedef struct {
> const char *magicfile;/* where magic be found */
> struct magic *magic; /* head of magic config list */
> struct magic *last;
> +int uncompress;
> } magic_server_config_rec;
>
> /* per-request info */
> @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> {
> AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> "Path to MIME Magic file (in file(1) format)"),
> +AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
> + "Whether MIME should try to render uncompressed content by recognizing "
> + "the underlying type, or not (default)"),
> {NULL}
> };
>
> @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
>
> static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> {
> +magic_server_config_rec *conf = (magic_server_config_rec *)
> +ap_get_module_config(r->server->module_config,
> + &mime_magic_module);
> unsigned char *newbuf;
> int newsize;
> int i;
> @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
> if (i == ncompr)
> return 0;
>
> -if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> -/* set encoding type in the request record */
> -r->content_encoding = compr[i].encoding;
> -
> +if (!conf->uncompress) {
> +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> +}
> +else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> /* Try to detect the content type of the uncompressed data */
> if (tryit(r, newbuf, newsize, 0) != OK) {
> +magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> return 0;
> }
> +/* set encoding type in the request record */
> +r->content_encoding = compr[i].encoding;
> }
> return 1;
> }
> --
>
> Regards,
> Yann.
Re: mod_mime_magic, gzipped tarballs and Docker
On 01/08/2016 08:49 AM, Jan Kaluža wrote: Hi, it seems Docker client has a problem handling httpd responses [1] when you run Docker server behind httpd working as a reverse proxy. It is caused by "mod_mime_magic" adding following Content-Type and Content-Encoding to gzipped tarballs sent as a response by Docker server: Just to correct myself a bit. I have found out it's not a reverse-proxy case - in that case, mod_mime_magic would not be in the loop. They are just serving Docker data using following config: https://github.com/rbarlow/pulp_docker/blob/master/plugins/etc/httpd/conf.d/pulp_docker.conf Content-Type: application/x-tar Content-Encoding: x-gzip Docker client expects gzipped tarballs to be received, but because it receives Content-Encoding: x-gzip, it decompresses the response and receives just plain tar (at least that's how I understand it). Strictly speaking, httpd is probably right here according to RFC: """ The Content-Encoding entity-header field is used as a modifier to the media-type. When present, its value indicates what additional content codings have been applied to the entity-body, and thus what decoding mechanisms must be applied in order to obtain the media-type referenced by the Content-Type header field. """ So, the mod_mime_magic is saying here that the body is tarball encoded by gzip. But I think even we are right here, the Content-Encoding should be used only when httpd itself encodes the original content, otherwise it confuses clients (I've asked few people and it confuses some web browsers too - I can get more info if needed.). Maybe we could stop setting Content-Encoding in mod_mime_magic and just use Content-Type? [1] https://github.com/docker/docker/issues/17595 Regards, Jan Kaluza Regards, Jan Kaluza
Re: mod_mime_magic, gzipped tarballs and Docker
Hi,
On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža wrote:
>
> Content-Type: application/x-tar
> Content-Encoding: x-gzip
[]
>
> So, the mod_mime_magic is saying here that the body is tarball encoded by
> gzip.
AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
the above headers.
>
> But I think even we are right here, the Content-Encoding should be used only
> when httpd itself encodes the original content, otherwise it confuses
> clients (I've asked few people and it confuses some web browsers too - I can
> get more info if needed.).
Agreed, this looks wrong, the browsers will likely use the .tar file
instead (this adds unnecessary cycles to both the server and client).
>
> Maybe we could stop setting Content-Encoding in mod_mime_magic and just use
> Content-Type?
Since it's always been there, we probably should add a new
mod_mime_magic directive to control the behaviour and avoid breaking
cases.
Something along:
Index: modules/metadata/mod_mime_magic.c
===
--- modules/metadata/mod_mime_magic.c(revision 1723283)
+++ modules/metadata/mod_mime_magic.c(working copy)
@@ -456,6 +456,7 @@ typedef struct {
const char *magicfile;/* where magic be found */
struct magic *magic; /* head of magic config list */
struct magic *last;
+int uncompress;
} magic_server_config_rec;
/* per-request info */
@@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
{
AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
"Path to MIME Magic file (in file(1) format)"),
+AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
+ (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
+ "Whether MIME should try to render uncompressed content by recognizing "
+ "the underlying type, or not (default)"),
{NULL}
};
@@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
{
+magic_server_config_rec *conf = (magic_server_config_rec *)
+ap_get_module_config(r->server->module_config,
+ &mime_magic_module);
unsigned char *newbuf;
int newsize;
int i;
@@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
if (i == ncompr)
return 0;
-if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
-/* set encoding type in the request record */
-r->content_encoding = compr[i].encoding;
-
+if (!conf->uncompress) {
+magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
+ compr[i].encoding, NULL));
+}
+else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
/* Try to detect the content type of the uncompressed data */
if (tryit(r, newbuf, newsize, 0) != OK) {
+magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
+ compr[i].encoding, NULL));
return 0;
}
+/* set encoding type in the request record */
+r->content_encoding = compr[i].encoding;
}
return 1;
}
--
Regards,
Yann.
mod_mime_magic, gzipped tarballs and Docker
Hi, it seems Docker client has a problem handling httpd responses [1] when you run Docker server behind httpd working as a reverse proxy. It is caused by "mod_mime_magic" adding following Content-Type and Content-Encoding to gzipped tarballs sent as a response by Docker server: Content-Type: application/x-tar Content-Encoding: x-gzip Docker client expects gzipped tarballs to be received, but because it receives Content-Encoding: x-gzip, it decompresses the response and receives just plain tar (at least that's how I understand it). Strictly speaking, httpd is probably right here according to RFC: """ The Content-Encoding entity-header field is used as a modifier to the media-type. When present, its value indicates what additional content codings have been applied to the entity-body, and thus what decoding mechanisms must be applied in order to obtain the media-type referenced by the Content-Type header field. """ So, the mod_mime_magic is saying here that the body is tarball encoded by gzip. But I think even we are right here, the Content-Encoding should be used only when httpd itself encodes the original content, otherwise it confuses clients (I've asked few people and it confuses some web browsers too - I can get more info if needed.). Maybe we could stop setting Content-Encoding in mod_mime_magic and just use Content-Type? [1] https://github.com/docker/docker/issues/17595 Regards, Jan Kaluza
