[
https://issues.apache.org/jira/browse/TIKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17921798#comment-17921798
]
Tim Allison edited comment on TIKA-4370 at 1/28/25 4:41 PM:
------------------------------------------------------------
Hi [~subbudvk], this is long standing problem. Figuring out when a file is a
binary vs a non-ascii text file when there is no filename hint is not trivial.
Depending on your application, you could configure the text parser to parse
{{application/octet-stream}} so that you get content out. A similar hack would
be to tell Tika that the filename is "x.txt" if the detector returns
application/octet-stream and then re-detect.
A more comprehensive option would be to integrate the tika-eval module in a new
detector that runs encoding detection on application/octet stream, extracts the
text and then looks at the text statistics (language id, out of vocabulary etc)
and determines if there's enough "non-garbage" to count as a text file.
And, then there's the strings utility. :/
was (Author: [email protected]):
Hi [~subbudvk], this is long standing problem. Figuring out when a file is a
binary vs a non-ascii text file when there is no filename hint is not trivial.
Depending on your application, you could configure the text parser to parse
{{application/octet-stream}} so that you get content out. Another option would
be to integrate the tika-eval module in a new detector that runs encoding
detection on application/octet stream, extracts the text and then looks at the
text statistics (language id, out of vocabulary etc) and determines if there's
enough "non-garbage" to count as a text file.
> SJIS Encoded Files Can't be Detected
> ------------------------------------
>
> Key: TIKA-4370
> URL: https://issues.apache.org/jira/browse/TIKA-4370
> Project: Tika
> Issue Type: Bug
> Reporter: Subbu
> Priority: Major
>
> When character encoding of file is SJIS, without file name in the metadata,
> most files content-type detected as application/octet-stream. Is there zero
> support for SJIS?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
