Re: Embedded and default MIME types

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 12/4/19 13:23, Mark Thomas wrote:
> On 04/12/2019 18:11, Christopher Schultz wrote:
>> Mark,
>> 
>> On 12/4/19 13:02, Mark Thomas wrote:
>>> Hi all,
>> 
>>> I was looking at this as an off-shoot of looking at BZ 63985.
>> 
>>> Currently, there are 844 MIME mappings in conf/web.xml that
>>> are not included in Tomcat.DEFAULT_MIME_MAPPINGS. Fixing it
>>> isn't too hard. The 844 figure comes from a unit test I could
>>> easily tweak to generate the code needed to paste in to add
>>> these.
>> 
>>> The question is, do we want to keep these in sync?
>> 
>>> If the answer is yes I can get this fixed fairly quickly and
>>> add some unit tests to ensure that they stay in sync.
>> 
>>> Thoughts?
>> 
>> What about reading them from a file instead of hard-coding them?
>> 
>> That same file can be used at build-time to generate the actual 
>> contents of the conf/web.xml that we ship as a binary.
> 
> We could. I'm not convinced the savings of only maintaining a
> single file going forwards is worth the extra work to do that.

So to reduce the amount of work to use a single file, we're going to
write a suite of unit tests to compare the two possibly diverging
sources? Seems like it's roughly equivalent both ways.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3oCkIACgkQHPApP6U8
pFiIChAAi0uJF0GFws/dCd6EP61q5Wy5hWJTJokuhj0tRrIfNmtnS03D2jVqPIiG
i1fHTyeIvNqigY1dOcs+ImD+y4mjF2/dMPyzl7/lCgHjG5ERRD6UGs47oqWJYeXC
kXnm6OnJjBdVO4CtEm7MkQNzGD1YLfhDkunePyHKbuQE4MTYKEG04Sa3Rr9ukIbI
9H0tdhsT0n69ISTMZ7NyDm2Crk9X2tHTUUsZToc+y0xPUTsraxMD4VXfNPYGDW3w
16BIaOqMXyt9FxXfcwfMjtcRdjxGTGa0wFih7aojJfv5g+ErXOGMfTeU671Eaord
x30v9SSlvmxDaGVIxGms46nPG9ZI7mak4qyHJzcKeU/Ivd81mniE5Xq/p1/2QhsY
7+YT2EIobvXGFitzOBgSKDggWj8cvK5a2wKpdwwHssFA4wKN/w+gIybdA0Vgp3Hf
75QG3gzGYJGv0iCLNE6lXfBtL3X9UWbI0lfvhipIUYvEOB1hK5xknIeQUignKXZL
tT1qynyBXkCeG69yszaPAUkk2g0XS2oGRnE9YlfdaDdqqCNGE9Scwlap8Pra9ayj
Un0Ig8CbnUJeJ5VWKKpvGmh98i0HoeXHS53Tn4dC9A992uiO4zen5cxGSAN9WHfb
341bbqf8uoGUW+suQ14dL/0zBD600v/lynorR56V1T/QBGBKAQQ=
=RtcQ
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Embedded and default MIME types

[Romain Manni-Bucau]

+1 to align both, really reduces diff in tests and prod

Le mer. 4 déc. 2019 à 19:23, Mark Thomas  a écrit :

> On 04/12/2019 18:11, Christopher Schultz wrote:
> > Mark,
> >
> > On 12/4/19 13:02, Mark Thomas wrote:
> >> Hi all,
> >
> >> I was looking at this as an off-shoot of looking at BZ 63985.
> >
> >> Currently, there are 844 MIME mappings in conf/web.xml that are
> >> not included in Tomcat.DEFAULT_MIME_MAPPINGS. Fixing it isn't too
> >> hard. The 844 figure comes from a unit test I could easily tweak to
> >> generate the code needed to paste in to add these.
> >
> >> The question is, do we want to keep these in sync?
> >
> >> If the answer is yes I can get this fixed fairly quickly and add
> >> some unit tests to ensure that they stay in sync.
> >
> >> Thoughts?
> >
> > What about reading them from a file instead of hard-coding them?
> >
> > That same file can be used at build-time to generate the actual
> > contents of the conf/web.xml that we ship as a binary.
>
> We could. I'm not convinced the savings of only maintaining a single
> file going forwards is worth the extra work to do that.
>
> Mark
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Embedded and default MIME types

[Mark Thomas]

On 04/12/2019 18:11, Christopher Schultz wrote:
> Mark,
> 
> On 12/4/19 13:02, Mark Thomas wrote:
>> Hi all,
> 
>> I was looking at this as an off-shoot of looking at BZ 63985.
> 
>> Currently, there are 844 MIME mappings in conf/web.xml that are
>> not included in Tomcat.DEFAULT_MIME_MAPPINGS. Fixing it isn't too
>> hard. The 844 figure comes from a unit test I could easily tweak to
>> generate the code needed to paste in to add these.
> 
>> The question is, do we want to keep these in sync?
> 
>> If the answer is yes I can get this fixed fairly quickly and add
>> some unit tests to ensure that they stay in sync.
> 
>> Thoughts?
> 
> What about reading them from a file instead of hard-coding them?
> 
> That same file can be used at build-time to generate the actual
> contents of the conf/web.xml that we ship as a binary.

We could. I'm not convinced the savings of only maintaining a single
file going forwards is worth the extra work to do that.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Embedded and default MIME types

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 12/4/19 13:02, Mark Thomas wrote:
> Hi all,
> 
> I was looking at this as an off-shoot of looking at BZ 63985.
> 
> Currently, there are 844 MIME mappings in conf/web.xml that are
> not included in Tomcat.DEFAULT_MIME_MAPPINGS. Fixing it isn't too
> hard. The 844 figure comes from a unit test I could easily tweak to
> generate the code needed to paste in to add these.
> 
> The question is, do we want to keep these in sync?
> 
> If the answer is yes I can get this fixed fairly quickly and add
> some unit tests to ensure that they stay in sync.
> 
> Thoughts?

What about reading them from a file instead of hard-coding them?

That same file can be used at build-time to generate the actual
contents of the conf/web.xml that we ship as a binary.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3n9tsACgkQHPApP6U8
pFg6nQ/8CwL8kb5V0t/3or5VmbzM0gkpPsHvRYawXSsgUXxgjk+6fErKH43oEBrX
MDSUJzysMn/LooYBrwhv4dgi1CyiI/75vRHQ0bzYNlOaqRT3Ne+H7Y3SyWWBl6zH
Xmrdoj1dzxqbRJeAv5emn+BvFLcxCdt6aFMLT5lJBUkTwtHn5dQK4D26CDmgaSM/
NABwV0UHt2UOhslI7vk/7+CB/t5V7iuAFbAId8qq6Ydk67uO/Lv01HMUJdyd3bd1
WKQaM99yx8ENMjTQzOEQ8KqItQ9pIn08JAqIE23Yc8noxCW7LYIGKfh/254vFR+K
x+K+DMnn9TDx97ucH7mujVyOcRE/gkHm9vJNqULRSah8k9W/W9STC5xxp2ecplvP
ynVabwLEWD3mPuUrN43uWGJD93JTXmuGFIgusbeWEbYCUIB132+Vy5lYOH+2Dsqj
X5Wt9tXfdEX3dTYhwk0VaoU8LeN2TpwTR2kIgyHWmQDYLAMG6ubkOfxoRB5jQ8BN
uUWhxSbBh1PLDzd5hicmEN5xWJTBmeGY8phDFdrP5GPef8fvPMYHhsWThvoZvxrT
Q3Rs9Ii1NNt63i7JjWcsTXlPtR+NqtIUvUeF+EGl8QHEbOlcjIGwaP3gKYYfUXpE
JA467ZyhUgLIP3jH00tNvveEJDpo0Ta558YAph/+XpeKMBzFR0Y=
=gfpu
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Embedded and default MIME types

[Mark Thomas]

Hi all,

I was looking at this as an off-shoot of looking at BZ 63985.

Currently, there are 844 MIME mappings in conf/web.xml that are not
included in Tomcat.DEFAULT_MIME_MAPPINGS. Fixing it isn't too hard. The
844 figure comes from a unit test I could easily tweak to generate the
code needed to paste in to add these.

The question is, do we want to keep these in sync?

If the answer is yes I can get this fixed fairly quickly and add some
unit tests to ensure that they stay in sync.

Thoughts?

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Mime types

[Rémy Maucherat]

On Thu, Nov 1, 2018 at 2:34 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Why bother even doing that? File-extension mapping is a poor way to
> perform file-type detection (in fact, it is NOT file type detection),
> but the spec mandates it, so we are stuck.
>
> But why bother providing container-managed file-type detection? Seems
> like useless cruft to me.
>

Well, some users want it.

Anyway, I get the idea that the feature add isn't very popular, so I'll
forget about it.

Rémy

Re: Mime types

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Rémy,

On 10/31/18 12:37, Rémy Maucherat wrote:
> There are two main contraptions in Tomcat that do (badly ...)
> extension to mime type mapping: the shared web.xml and some
> hardcoded stuff in startup.Tomcat.

It would probably be good to unify all that code, regardless of what
else we do.

> While we should obviously have support for user configured mime
> types in web.xml, as it's the spec, there should be a possibility
> to use Files.probeContentType as the fallback when a mime type
> isn't found (and maybe also have an option to disable it ? -
> although I don't quite see why it would bother anyone).

Why bother even doing that? File-extension mapping is a poor way to
perform file-type detection (in fact, it is NOT file type detection),
but the spec mandates it, so we are stuck.

But why bother providing container-managed file-type detection? Seems
like useless cruft to me.

> After looking at its implementation, it looks into all mime type 
> locations we might want (the OS, a mime.types file, etc). The only 
> problem is that it uses a Path (that would be an issue since it's 
> super tied to a real filesystem), but thankfully it mostly uses 
> toString and thus can be worked around using a new fake Path 
> implementation.> The code calling Files.probeContentType could be 
> inserted here in DefaultServlet:
> 
> // Find content type. String contentType = resource.getMimeType(); 
> if (contentType == null) { contentType = 
> getServletContext().getMimeType(resource.getName()); ---> 
> resource.setMimeType(contentType); }
> 
> And then all the badly maintained content from web.xml and the
> Tomcat class can be deleted.
> 
> Comments ?

Even a benign library like libmagic has had a bunch of
vulnerabilities[1] over the years: provide a specially-crafted file
and you can do Bad Things.

If applications want to do this to themselves, why not let them do it
independently?

- -chris

[1]
https://www.cvedetails.com/product/22647/Tim-Robbins-Libmagic.html?vendo
r_id=12061
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvbAOAACgkQHPApP6U8
pFg5gw/9EfWNeXueSYS4bwHRR4IYA3COPGRuuepayAABIEOTNO1ZY1O+KZu39y/t
52GxR6tl6WvJxJ3FXkgqP9QrEAIAiDQwNTqj2OsragyFiE5jnIj5lRsZER9JWJmy
fjKRAMV3UOqSOYceU0PaXGOmnije5IU3rhO/ZQIMcZToXPPevI8RyM1QPpGDQGvt
9y1OkVfeaHcaIQuYEN/ue0u3U4ARt/s4JwkSf1LJCIqgc5on4Cj/pAtcDOhSG4cN
ILsJmcuJDsbPaaBp8VkNw6gaQsMitZ8H8cAtovkMB8RdpSpgtL/EeN4NTdDaXgoR
EzP5hrhrx20jpgLjVfLW1NI/7F+Ek+IYHwUdNcAy2oDQaEO8tVeRh9D0O8V8Dk+T
czFgoSq0MajlmWSzlRSCmhGVd7xxaYhaVH0PThTMLVKkF/QWafzrZQZXJ30oLXQv
iDnvJ5rXP87WamtSlXL+KvDTJSK8stJwCZAFRJEz67hxTSKG60yByvN134yP4E4i
4TdXGmvHnexqdPJ7/+Eymf6ESgAEW0IuH2DHnWaKQGtc2TZeLSK1Gp4m9zjS0Ojl
wsWwgiO4vDFddLZZvoa1shnE964uufES/DLe92TbnfIpRk6GoT85tlojKSMDsBhV
56ce0jWim4c7mf9QGbe25Hf/rf6ooCRwj2TU/WBUy+aYg+v2xH4=
=caWD
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Mime types

[Rainer Jung]

Am 31.10.2018 um 20:15 schrieb Konstantin Kolinko:

ср, 31 окт. 2018 г. в 19:38, Rémy Maucherat :

...

And then all the badly maintained content from web.xml and the Tomcat class
can be deleted.

Comments ?


1. "badly maintained content from web.xml"

Do not call them "bad".

AFAIK, Those are synchronized with httpd. IIRC there was python script
to check the sync. Technically, it should be possible to sync with
IANA registry.

(I do not remember the details - those should be easy to find in the
archives of this mailing list. I just remember that the last time that
the sync was checked, there was some good job done to automate and
perform the check.)


Here's what I had originally prepare (script, but not automated):

% res/scripts/check-mime.pl --help
trunk-veryclean2/res/scripts/check-mime.pl version 1.1 calling 
Getopt::Std::getopts (version 1.05),

running under Perl version 5.8.4.
Usage:: res/scripts/check-mime.pl -m MIMEFILE -i INPUTFILE -o OUTPUTFILE
   MIMEFILE:   path to mime.types from the httpd project
   INPUTFILE:  path to existing web.xml, which will be checked
   OUTPUTFILE: path to the new (generated) web.xml. Any existing
   file will be overwritten.

% res/scripts/check-mime.pl -m /shared/build/dev/httpd/svn/httpd \
/path/to/mime.types -i conf/web.xml -o conf/web.xml-new
FATAL Definition 'flac' -> 'audio/x-flac' exists in mime.types but
FATAL differs from 'flac' -> 'audio/flac' in TOMCAT_ONLY.
FATAL You must either remove 'flac' from the TOMCAT_ONLY list to keep 
the mime.types variant,
FATAL or move it to TOMCAT_KEEP to overwrite the mime.types variant - 
Aborting!


OK, so that means we originally had "flac" in our web.xml, but httpd had 
none. In the meantime httpd added it as audio/x-flac, but we had 
audio/flac, so we need to decide between the two.


in httpd the change was (svn log):

r1301894 | fielding | 2012-03-17 09:58:59 +0100 (Sat, 17 Mar 2012) | 2 lines
Unregistered media types in common use, according to Wikipedia and MS

Doing a little search does not result in any of the two variants being 
obviously correct. So for now I add audio/flac to TOMCAT_KEEP and we 
stick to our decision (done in 1845472).


And then:

% res/scripts/check-mime.pl -m /shared/build/dev/httpd/svn/httpd \
/path/to/mime.types -i conf/web.xml -o conf/web.xml-new
New file 'conf/web.xml-new' has been written.

% diff -u  conf/web.xmlconf/web.xml-new

No output, so no changes, nothing to update.

Regards,

Rainer

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Mime types

[Rémy Maucherat]

On Wed, Oct 31, 2018 at 10:30 PM Igal Sapir  wrote:

> >> Unless DefaultServlet behaviour is fixed, enabling probeContentType is
> >> likely to break my configurations.
>
> How about allowing to remove a Mime type by mapping it to an empty
> string?  That way you would only need to override that one in your
> Application's web.xml.
>
> Would that work?
>

That won't work, but I'm adding an init-param for the feature to disable it
at the same time.


>
> >> 4. I see a similarity to mod_mime_magic module of HTTPD.
> >>
> >> http://httpd.apache.org/docs/current/mod/mod_mime.html
> >> http://httpd.apache.org/docs/current/mod/mod_mime_magic.html
> >>
> >> (For some reason I though that mod_mime_magic uses the magic file from
> >> Unix OS.
> >> Actually it uses its own magic file from configuration of HTTPD,
> >> configured by directive "MimeMagicFile".
> >> So it is actually portable.)
> >>
> >> MimeMagicFile directive is disabled by default.
> >>
> >>
> http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/httpd.conf.in?view=markup#l330
> >>
> >> Performance =?
> >> The documentation of mod_mime_magic says that performance is a concern
> >> for this module.
>
> The lookup will only happen for unknown types, and as Remy said they
> will be cached so I don't think that performance is an issue here.
>
> >
> > No idea really, probeContentType checks a number of sources [on Fedora it
> > is: new GnomeFileTypeDetector(), new
> > MimeTypesFileTypeDetector(userMimeTypes), new
> > MimeTypesFileTypeDetector(etcMimeTypes), new MagicFileTypeDetector()],
> and
> > it is platform dependent as you say. The result is then cached into the
> > resource in Tomcat, so it should be "ok" performance wise. The question
> is:
> > besides the default servlet not respecting a possibly set content-type (I
> > have not looked at it, and IMO it is a separate issue, no problem if
> you'd
> > like to fix it), is the Files.probeContentType result good enough on
> > Windows ?
>
> I have both Windows and Fedora so if you want me to test something
> specific please let me know.
>

Looked at the code, Windows uses the registry only (so something that may
be incomplete and not standardized). As a result, it would be unreasonable
to remove the current default mappings.
Example: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8080369

Rémy

Re: Mime types

[Igal Sapir]

On 10/31/2018 12:50 PM, Rémy Maucherat wrote:

On Wed, Oct 31, 2018 at 8:16 PM Konstantin Kolinko 
wrote:


ср, 31 окт. 2018 г. в 19:38, Rémy Maucherat :

Hi,

There are two main contraptions in Tomcat that do (badly ...) extension tomime 
type mapping: the shared web.xml and some hardcoded stuff in
startup.Tomcat.

While we should obviously have support for user configured mime types in
web.xml, as it's the spec, there should be a possibility to use
Files.probeContentType as the fallback when a mime type isn't found (and
maybe also have an option to disable it ? - although I don't quite see why it 
would bother anyone). After looking at its implementation, it looks into all 
mime type locations we might want (the OS, a mime.types file, etc).  Theonly 
problem is that it uses a Path (that would be an issue since it's
super tied to a real filesystem), but thankfully it mostly uses toSting and 
thus can be worked around using a new fake Path implementation.

The code calling Files.probeContentType could be inserted here in
DefaultServlet:
 // Find content type.
 String contentType = resource.getMimeType();
 if (contentType == null) {
 contentType =
getServletContext().getMimeType(resource.getName());
--->
 resource.setMimeType(contentType);
 }

And then all the badly maintained content from web.xml and the Tomcat class can 
be deleted.

Comments ?

1. "badly maintained content from web.xml"

Do not call them "bad".


Ok, but they didn't look too good. My mime.types has a lot more types for
starters, and it's so big I don't feel like adding all that.


AFAIK, Those are synchronized with httpd. IIRC there was python script
to check the sync. Technically, it should be possible to sync with
IANA registry.

(I do not remember the details - those should be easy to find in the
archives of this mailing list. I just remember that the last time that
the sync was checked, there was some good job done to automate and
perform the check.)

Who maintains the mappings used Files.probeContentType and why do you
think that those are maintained any better?

I remember we had a number of BZs asking to add or fix mime mappings.  Annoying.


If an OS is an LTS one, are mime-mapping configurations in the OS
updated as the time goes?

Well, ultimately the guy can still add its new mapping in web.xml. Just the
basic usual ones don't need to be there.


It should be possible to write a JUnit test to keep the mappings in
startup.Tomcat.DEFAULT_MIME_MAPPINGS in sync with the default web.xml
file.

2. web.xml is portable between operating systems.

I would expect surprises from Files.probeContentType()

Looking at javadocs, FileTypeDetector is pluggable,
and the default behaviour is os-dependent


https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#probeContentType-java.nio.file.Path-

https://docs.oracle.com/javase/8/docs/api/java/nio/file/spi/FileTypeDetector.html

3, My own story: I had to remove the default mime-type mapping for
"gz" from conf/web.xml in my configurations:

For filenames like "filename.foo.gz" it is "foo" part that determines
the mime-type for me. This cannot be configured in web.xml. I use a
filter (urlrewrite) to set content-type for requests to those files.

Generally, configuring a Filter should have been enough. But there is
a bug in the DefaultServlet that it does not respect the content-type
that has already been set on the response and blindly overwrites it.
Unless I remove the default mapping for "gz", the content-type value
set by a filter is overwritten.

Unless DefaultServlet behaviour is fixed, enabling probeContentType is
likely to break my configurations.


How about allowing to remove a Mime type by mapping it to an empty 
string?  That way you would only need to override that one in your 
Application's web.xml.


Would that work?


4. I see a similarity to mod_mime_magic module of HTTPD.

http://httpd.apache.org/docs/current/mod/mod_mime.html
http://httpd.apache.org/docs/current/mod/mod_mime_magic.html

(For some reason I though that mod_mime_magic uses the magic file from
Unix OS.
Actually it uses its own magic file from configuration of HTTPD,
configured by directive "MimeMagicFile".
So it is actually portable.)

MimeMagicFile directive is disabled by default.

http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/httpd.conf.in?view=markup#l330

Performance =?
The documentation of mod_mime_magic says that performance is a concern
for this module.


The lookup will only happen for unknown types, and as Remy said they 
will be cached so I don't think that performance is an issue here.




No idea really, probeContentType checks a number of sources [on Fedora it
is: new GnomeFileTypeDetector(), new
MimeTypesFileTypeDetector(userMimeTypes), new
MimeTypesFileTypeDetector(etcMimeTypes), new MagicFileTypeDetector()], and
it is platform dependent as you

Re: Mime types

[Rémy Maucherat]

On Wed, Oct 31, 2018 at 8:16 PM Konstantin Kolinko 
wrote:

> ср, 31 окт. 2018 г. в 19:38, Rémy Maucherat :
> >
> > Hi,
> >
> > There are two main contraptions in Tomcat that do (badly ...) extension
> to
> > mime type mapping: the shared web.xml and some hardcoded stuff in
> > startup.Tomcat.
> >
> > While we should obviously have support for user configured mime types in
> > web.xml, as it's the spec, there should be a possibility to use
> > Files.probeContentType as the fallback when a mime type isn't found (and
> > maybe also have an option to disable it ? - although I don't quite see
> why
> > it would bother anyone). After looking at its implementation, it looks
> into
> > all mime type locations we might want (the OS, a mime.types file, etc).
> The
> > only problem is that it uses a Path (that would be an issue since it's
> > super tied to a real filesystem), but thankfully it mostly uses toSting
> and
> > thus can be worked around using a new fake Path implementation.
> >
> > The code calling Files.probeContentType could be inserted here in
> > DefaultServlet:
> > // Find content type.
> > String contentType = resource.getMimeType();
> > if (contentType == null) {
> > contentType =
> > getServletContext().getMimeType(resource.getName());
> > --->
> > resource.setMimeType(contentType);
> > }
> >
> > And then all the badly maintained content from web.xml and the Tomcat
> class
> > can be deleted.
> >
> > Comments ?
>
> 1. "badly maintained content from web.xml"
>
> Do not call them "bad".
>

Ok, but they didn't look too good. My mime.types has a lot more types for
starters, and it's so big I don't feel like adding all that.

>
> AFAIK, Those are synchronized with httpd. IIRC there was python script
> to check the sync. Technically, it should be possible to sync with
> IANA registry.
>
> (I do not remember the details - those should be easy to find in the
> archives of this mailing list. I just remember that the last time that
> the sync was checked, there was some good job done to automate and
> perform the check.)
>
> Who maintains the mappings used Files.probeContentType and why do you
> think that those are maintained any better?
>

I remember we had a number of BZs asking to add or fix mime mappings.
Annoying.

>
> If an OS is an LTS one, are mime-mapping configurations in the OS
> updated as the time goes?
>

Well, ultimately the guy can still add its new mapping in web.xml. Just the
basic usual ones don't need to be there.

>
> It should be possible to write a JUnit test to keep the mappings in
> startup.Tomcat.DEFAULT_MIME_MAPPINGS in sync with the default web.xml
> file.
>
> 2. web.xml is portable between operating systems.
>
> I would expect surprises from Files.probeContentType()
>
> Looking at javadocs, FileTypeDetector is pluggable,
> and the default behaviour is os-dependent
>
>
> https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#probeContentType-java.nio.file.Path-
>
> https://docs.oracle.com/javase/8/docs/api/java/nio/file/spi/FileTypeDetector.html
>
> 3, My own story: I had to remove the default mime-type mapping for
> "gz" from conf/web.xml in my configurations:
>
> For filenames like "filename.foo.gz" it is "foo" part that determines
> the mime-type for me. This cannot be configured in web.xml. I use a
> filter (urlrewrite) to set content-type for requests to those files.
>
> Generally, configuring a Filter should have been enough. But there is
> a bug in the DefaultServlet that it does not respect the content-type
> that has already been set on the response and blindly overwrites it.
> Unless I remove the default mapping for "gz", the content-type value
> set by a filter is overwritten.
>
> Unless DefaultServlet behaviour is fixed, enabling probeContentType is
> likely to break my configurations.
>
> 4. I see a similarity to mod_mime_magic module of HTTPD.
>
> http://httpd.apache.org/docs/current/mod/mod_mime.html
> http://httpd.apache.org/docs/current/mod/mod_mime_magic.html
>
> (For some reason I though that mod_mime_magic uses the magic file from
> Unix OS.
> Actually it uses its own magic file from configuration of HTTPD,
> configured by directive "MimeMagicFile".
> So it is actually portable.)
>
> MimeMagicFile directive is disabled by default.
>
> http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/httpd.conf.in?view=markup#l330
>
> Performance =?
> The documentation of mod_mime_magic says that per

Re: Mime types

[Konstantin Kolinko]

ср, 31 окт. 2018 г. в 19:38, Rémy Maucherat :
>
> Hi,
>
> There are two main contraptions in Tomcat that do (badly ...) extension to
> mime type mapping: the shared web.xml and some hardcoded stuff in
> startup.Tomcat.
>
> While we should obviously have support for user configured mime types in
> web.xml, as it's the spec, there should be a possibility to use
> Files.probeContentType as the fallback when a mime type isn't found (and
> maybe also have an option to disable it ? - although I don't quite see why
> it would bother anyone). After looking at its implementation, it looks into
> all mime type locations we might want (the OS, a mime.types file, etc). The
> only problem is that it uses a Path (that would be an issue since it's
> super tied to a real filesystem), but thankfully it mostly uses toSting and
> thus can be worked around using a new fake Path implementation.
>
> The code calling Files.probeContentType could be inserted here in
> DefaultServlet:
> // Find content type.
> String contentType = resource.getMimeType();
> if (contentType == null) {
> contentType =
> getServletContext().getMimeType(resource.getName());
> --->
> resource.setMimeType(contentType);
> }
>
> And then all the badly maintained content from web.xml and the Tomcat class
> can be deleted.
>
> Comments ?

1. "badly maintained content from web.xml"

Do not call them "bad".

AFAIK, Those are synchronized with httpd. IIRC there was python script
to check the sync. Technically, it should be possible to sync with
IANA registry.

(I do not remember the details - those should be easy to find in the
archives of this mailing list. I just remember that the last time that
the sync was checked, there was some good job done to automate and
perform the check.)

Who maintains the mappings used Files.probeContentType and why do you
think that those are maintained any better?

If an OS is an LTS one, are mime-mapping configurations in the OS
updated as the time goes?

It should be possible to write a JUnit test to keep the mappings in
startup.Tomcat.DEFAULT_MIME_MAPPINGS in sync with the default web.xml
file.

2. web.xml is portable between operating systems.

I would expect surprises from Files.probeContentType()

Looking at javadocs, FileTypeDetector is pluggable,
and the default behaviour is os-dependent

https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#probeContentType-java.nio.file.Path-
https://docs.oracle.com/javase/8/docs/api/java/nio/file/spi/FileTypeDetector.html

3, My own story: I had to remove the default mime-type mapping for
"gz" from conf/web.xml in my configurations:

For filenames like "filename.foo.gz" it is "foo" part that determines
the mime-type for me. This cannot be configured in web.xml. I use a
filter (urlrewrite) to set content-type for requests to those files.

Generally, configuring a Filter should have been enough. But there is
a bug in the DefaultServlet that it does not respect the content-type
that has already been set on the response and blindly overwrites it.
Unless I remove the default mapping for "gz", the content-type value
set by a filter is overwritten.

Unless DefaultServlet behaviour is fixed, enabling probeContentType is
likely to break my configurations.

4. I see a similarity to mod_mime_magic module of HTTPD.

http://httpd.apache.org/docs/current/mod/mod_mime.html
http://httpd.apache.org/docs/current/mod/mod_mime_magic.html

(For some reason I though that mod_mime_magic uses the magic file from Unix OS.
Actually it uses its own magic file from configuration of HTTPD,
configured by directive "MimeMagicFile".
So it is actually portable.)

MimeMagicFile directive is disabled by default.
http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/httpd.conf.in?view=markup#l330

Performance =?
The documentation of mod_mime_magic says that performance is a concern
for this module.

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Mime types

[Rémy Maucherat]

Hi,

There are two main contraptions in Tomcat that do (badly ...) extension to
mime type mapping: the shared web.xml and some hardcoded stuff in
startup.Tomcat.

While we should obviously have support for user configured mime types in
web.xml, as it's the spec, there should be a possibility to use
Files.probeContentType as the fallback when a mime type isn't found (and
maybe also have an option to disable it ? - although I don't quite see why
it would bother anyone). After looking at its implementation, it looks into
all mime type locations we might want (the OS, a mime.types file, etc). The
only problem is that it uses a Path (that would be an issue since it's
super tied to a real filesystem), but thankfully it mostly uses toSting and
thus can be worked around using a new fake Path implementation.

The code calling Files.probeContentType could be inserted here in
DefaultServlet:
// Find content type.
String contentType = resource.getMimeType();
if (contentType == null) {
contentType =
getServletContext().getMimeType(resource.getName());
--->
resource.setMimeType(contentType);
}

And then all the badly maintained content from web.xml and the Tomcat class
can be deleted.

Comments ?

Rémy

[Bug 61658] Update mime-mappings for font mime-types to use font/*, per RFC8081

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=61658

Mark Thomas  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED
 OS||All

--- Comment #1 from Mark Thomas  ---
Fixed in:
- trunk for 9.0.2 onwards
- 8.5.x for 8.5.24 onwards
- 8.0.x for 8.0.48 onwardse
- 7.0.x for 7.0.83 onwards

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 61658] New: Update mime-mappings for font mime-types to use font/*, per RFC8081

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=61658

Bug ID: 61658
   Summary: Update mime-mappings for font mime-types to use
font/*, per RFC8081
   Product: Tomcat 9
   Version: 9.0.1
  Hardware: PC
Status: NEW
  Severity: enhancement
  Priority: P2
 Component: Catalina
  Assignee: dev@tomcat.apache.org
  Reporter: knst.koli...@gmail.com
  Target Milestone: -

In recently released Apache HTTPD 2.4.29 the mime mappings for fonts in its
conf/mime.types file were updated from application/* to font/*.

Commit: r1810122

RFC8081, published in February 2017:
https://tools.ietf.org/html/rfc8081

Discussion on dev@httpd:
"mime type woff woff2" on 2017-09-18
http://httpd.markmail.org/thread/fvzh4r7qh55drtk6

I think that r1810122 can be backported to Apache Tomcat.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 60851] Add more default compressible MIME types

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=60851

Mark Thomas  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #1 from Mark Thomas  ---
Fixed in 9.0.x for 9.0.0.M19 only.

We generally don't change defaults in stable versions unless there is a bug
that needs to be addressed.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 60851] New: Add more default compressible MIME types

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=60851

Bug ID: 60851
   Summary: Add more default compressible MIME types
   Product: Tomcat 8
   Version: 8.5.11
  Hardware: All
OS: All
Status: NEW
  Severity: enhancement
  Priority: P2
 Component: Catalina
  Assignee: dev@tomcat.apache.org
  Reporter: 1983-01...@gmx.net
  Target Milestone: 

Created attachment 34818
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34818=edit
New MIME types

Two promiment MIME types are missing: application/xml and application/json.
Both added in the patch attached.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 51137] Please add MIME types for xslx, xslm, docx, docm since xls and doc already contained in web.xml have been replaced by these.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=51137

Rainer Jung rainer.j...@kippdata.de changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution||FIXED

--- Comment #2 from Rainer Jung rainer.j...@kippdata.de 2011-05-03 12:21:05 
UTC ---
The MIME type mappings have mostly been synced with the oned from Apache web
server. Especially the types proposed here should now be part of the Tomcat
MIME type mappings.

The new mappings have been added in r1099020 and will be part of TC 7.0.13.

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 51137] New: Please add MIME types for xslx, xslm, docx, docm since xls and doc already contained in web.xml have been replaced by these.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=51137

 Bug #: 51137
   Summary: Please add MIME types for xslx, xslm, docx, docm since
xls and doc already contained in web.xml have been
replaced by these.
   Product: Tomcat 7
   Version: 7.0.10
  Platform: PC
Status: NEW
  Severity: normal
  Priority: P2
 Component: Catalina
AssignedTo: dev@tomcat.apache.org
ReportedBy: guido.leend...@invantive.com
Classification: Unclassified


Could the following MIME types be added to conf/web.xml?

docm application/vnd.ms-word.document.macroEnabled.12
docx application/vnd.openxmlformats-officedocument.wordprocessingml.document
dotx application/vnd.openxmlformats-officedocument.wordprocessingml.template
potm application/vnd.ms-powerpoint.template.macroEnabled.12
potx application/vnd.openxmlformats-officedocument.presentationml.template
ppam application/vnd.ms-powerpoint.addin.macroEnabled.12
ppsm application/vnd.ms-powerpoint.slideshow.macroEnabled.12
ppsx application/vnd.openxmlformats-officedocument.presentationml.slideshow
pptm application/vnd.ms-powerpoint.presentation.macroEnabled.12
pptx application/vnd.openxmlformats-officedocument.presentationml.presentation
xlam application/vnd.ms-excel.addin.macroEnabled.12
xlsb application/vnd.ms-excel.sheet.binary.macroEnabled.12
xlsm application/vnd.ms-excel.sheet.macroEnabled.12
xlsx application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xltm application/vnd.ms-excel.template.macroEnabled.12
xltx application/vnd.openxmlformats-officedocument.spreadsheetml.template

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 51137] Please add MIME types for xslx, xslm, docx, docm since xls and doc already contained in web.xml have been replaced by these.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=51137

Mark Thomas ma...@apache.org changed:

   What|Removed |Added

 OS/Version||All
   Severity|normal  |enhancement

--- Comment #1 from Mark Thomas ma...@apache.org 2011-04-29 16:52:45 UTC ---
Enhancement requests where patches are provided (in diff -u format) tend to get
addressed quicker than enhancement requests without patches

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 51050] MIME types for m4a m4v files

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=51050

Mark Thomas ma...@apache.org changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution||FIXED

--- Comment #1 from Mark Thomas ma...@apache.org 2011-04-16 18:54:09 EDT ---
Fixed in 7.0.x and will be included in 7.0.13 onwards.

Thanks for the patch.

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 51050] New: MIME types for m4a m4v files

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=51050

   Summary: MIME types for m4a  m4v files
   Product: Tomcat 7
   Version: 7.0.12
  Platform: All
OS/Version: All
Status: NEW
  Severity: minor
  Priority: P2
 Component: Catalina
AssignedTo: dev@tomcat.apache.org
ReportedBy: cedrik.l...@gmail.com


Created an attachment (id=26874)
 -- (https://issues.apache.org/bugzilla/attachment.cgi?id=26874)
Patch for additional common mpeg 4 mime types

Tomcat correctly serves MPEG 4 files with a standardized file extension of
.mp4, as video/mp4.

Common but non-standard mpeg4 files also use of the extensions .m4a and .m4v,
which are unknown to Tomcat and are therefore served without any Content-Type
header. This behaviour leads Internet Explorer to display garbage when
following a link to such a file.

This patch adds the correct MIME type for MPEG 4 files with extensions m4a,
m4b, m4r and m4v. (It omits the m4p extension, which is both not widely used,
and of limited interest since the data is encrypted.)

References:
http://tools.ietf.org/html/rfc4337
http://en.wikipedia.org/wiki/MPEG-4_Part_14#.MP4_versus_.M4A_filename_extensions

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

DO NOT REPLY [Bug 41990] - several mime types missing from conf/web.xml

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
http://issues.apache.org/bugzilla/show_bug.cgi?id=41990.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=41990


[EMAIL PROTECTED] changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution||FIXED




--- Additional Comments From [EMAIL PROTECTED]  2007-04-15 13:05 ---
I have added these two to svn. They will be included in 5.5.24 and 6.0.12 
onwards.

If you want to create a patch for any other missing mime-types (I didn't check
the link you gave), I'll happily apply it.

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 41990] - several mime types missing from conf/web.xml

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
http://issues.apache.org/bugzilla/show_bug.cgi?id=41990.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=41990


[EMAIL PROTECTED] changed:

   What|Removed |Added

Summary|wmv mime type missing from  |several mime types missing
   |conf/web.xml|from conf/web.xml




--- Additional Comments From [EMAIL PROTECTED]  2007-03-29 18:00 ---
It seems several mime types are missing... here is another one...

mime-mapping
extensionmp4/extension
mime-typevideo/mp4/mime-type
/mime-mapping

Here is a nice list to compare against...

http://www.feedforall.com/mime-types.htm

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 27617] - ppt and vsd MIME types should be included in web.xml

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
http://issues.apache.org/bugzilla/show_bug.cgi?id=27617.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=27617


[EMAIL PROTECTED] changed:

   What|Removed |Added

 Status|RESOLVED|REOPENED
 Resolution|FIXED   |




--- Additional Comments From [EMAIL PROTECTED]  2006-02-27 15:41 ---
The lines in web.xml currently distributed with Tomcat 5.0.28 do not work for
Internet Exploder.  Further, the web.xml file omits an entry for PPS
(power-point show) files.

Details: I hit problems while attempting to provide access to powerpoint files
using Tomcat 5.0.28.  That version is distributed with these lines in web.xml:

mime-mapping
extensionppt/extension
mime-typeapplication/powerpoint/mime-type
/mime-mapping

I compared the mime-types with entries in Apache 2.20 mime.types file,
where I find only this:

application/vnd.ms-powerpointppt

Based on my tests, Firefox works fine with the application/powerpoint
entry, but IE does not.  IE seems to require the
application/vnd.ms-powerpoint variation, and Firefox accepts that as
well.  I used the following to replace what is shown above with good results:

mime-mapping
extensionppt/extension
mime-typeapplication/vnd.ms-powerpoint/mime-type
/mime-mapping
mime-mapping
extensionpps/extension
mime-typeapplication/vnd.ms-powerpoint/mime-type
/mime-mapping

Please consider using these entries in the web.xml distributed with Tomcat.

I do not have Visio, so could not test the behavior of IE on a Visio file, but
that probably should be checked as well.

I contacted Scott Pontillo, the original reporter of this bug, and he made the
following suggestion:

As a longer-term solution to this problem, maybe it would be good to
establish a process for importing the entries from the native Apache
server periodically? It seems they are kept more up-to-date for that
product...

I've taken it upon myself to reopen the bug since it is easily reproducible and
seems like such a minor issue.  I apologize if this was premature.  Thanks for
listening. 

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]