[
https://issues.apache.org/jira/browse/TIKA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612321#comment-14612321
]
Jeremy B. Merrill commented on TIKA-1602:
-----------------------------------------
Thank you, [~chrismattmann], [[email protected]] et al.!
[[email protected]] -- got a bunch of normal headers, but also this `Status:`
one. The only possible value in my dataset (a bunch of publicly-released emails
from Jeb Bush's tenure as FL Gov) is `RO`, so the first lines of the emails who
were treated improperly by Tika before this patch was uniformly `Status: RO`.
I'm going to check the whole dataset once I manage to download it all back down
again from storage to make sure there are no other values than `RO`.
My understanding is that some mail servers use this header internally to keep
track of read status. When emails are exported, they retain the header, and it
sometimes appears first -- even though the server would never send this header
over the wire.
> Detecting standards-non-compliant emails as message/rfc822
> ----------------------------------------------------------
>
> Key: TIKA-1602
> URL: https://issues.apache.org/jira/browse/TIKA-1602
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Reporter: Jeremy B. Merrill
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 1.10
>
> Attachments: 036491.txt.zip
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Tika does not properly detect certain emails as `message/rfc822` if they're
> slightly standards-non-compliant and begin with `Status: ` as the first
> header. I've added `Status: ` as a magic detection line in
> tika-mimetypes.xml.
> This solves my problem and does not appear to cause unit test failures. I
> have not yet run the tika-batch tests.
> As further information, the emails that are processed incorrectly come from
> dumps directly from various US public officials' mailservers. The dumps, I
> believe since they're not intended to be transmitted over the wire, sometimes
> are slightly non-compliant.
> It's important to note that Tika (and the underlying library, James Mime4J)
> do properly *parse* these emails, despite the non-compliant header. The
> problem is getting Tika to *detect* the file as an email so that Mime4J gets
> chosen to parse it.
> Pull request on Github at https://github.com/apache/tika/pull/40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
