[
https://issues.apache.org/jira/browse/TIKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922098#comment-17922098
]
Tim Allison commented on TIKA-4370:
-----------------------------------
bq. The problem is not specific to text file vs binary
Unless I misunderstand your point, under the hood in Tika, that's the problem.
This is the hack that does not work for shift-jis:
https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/detect/TextDetector.java#L131
and
https://github.com/apache/tika/blob/9a6bda5fadcc4c287926ec3361c6f398d113eab0/tika-core/src/main/java/org/apache/tika/mime/MimeTypes.java#L292
This hack also does not work for UTF 16 and other encodings. We need to do
something better.
I'm not sure I follow your proposals for a work around.
bq. Shift-JIS encoded file types
This is a text file with shift-jis encoding? Right? You don't have this problem
with x?html? Are there other file types that are shiftjis?
bq. using prefix...
There's no mime magic or BOM for shift-jis text files?
> SJIS Encoded Files Can't be Detected
> ------------------------------------
>
> Key: TIKA-4370
> URL: https://issues.apache.org/jira/browse/TIKA-4370
> Project: Tika
> Issue Type: Bug
> Reporter: Subbu
> Priority: Major
>
> When character encoding of file is SJIS, without file name in the metadata,
> most files content-type detected as application/octet-stream. Is there zero
> support for SJIS?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
