[
https://issues.apache.org/jira/browse/TIKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311065#comment-14311065
]
Giuseppe Totaro edited comment on TIKA-1541 at 2/8/15 2:24 AM:
---------------------------------------------------------------
Hi all,
you can find the updated patch and a file for testing in attachment.
[~gagravarr] I agree to not enable {{StringsParser}} by default for
{{application/octet-stream}}. However, I think it may be a good solution for
digital investigation purposes as made by many forensic tools (e.g., FTK) that
run {{strings}} (or a similar program) over undetected files in order to
extract text for building an inverted index.
I added a configuration class for {{file}} command ({{FileConfig}}). I added
also "essential" unit tests for {{StringsParser}}, {{StringsConfig}}, and
{{FileConfig}}.
[~lfcnassif] I thought a more general configuration class for strings-like
commands (using the {{setOptions}} method).
Since we are focusing on {{strings}} command, I updated the {{StringsConfig}}
class supporting {{min-len}} and {{encoding}} options. The {{StringsEncoding}}
enumeration contains all values allowed for {{strings}}. As far as I
understood, the {{-e}} option for encoding is not supported yet by the Windows
version. Now the parser does not set the Content-Type as {{octet-stream}}.
Thanks [~chrismattmann], [~gagravarr] and [~lfcnassif] for supporting me.
was (Author: gostep):
Hi all,
you can find the updated patch and a file for testing in attachment.
[~gagravarr] I agree to not enable {{StringsParser}} by default for
{{application/octet-stream}}. However, I think it may be a good solution for
digital investigation purposes as made by many forensic tools (e.g., FTK) that
run {{strings}} (or a similar program) over undetected files in order to
extract text for building an inverted index.
I added a configuration class for {{file}} command ({{FileConfig}}). I added
also "essential" unit tests for {{StringsParser}}, {{StringsConfig}}, and
{{FileConfig}}.
[~lfcnassif]] I thought a more general configuration class for strings-like
commands (using the {{setOptions}} method).
Since we are focusing on {{strings}} command, I updated the {{StringsConfig}}
class supporting {{min-len}} and {{encoding}} options. The {{StringsEncoding}}
enumeration contains all values allowed for {{strings}}. As far as I
understood, the {{-e}} option for encoding is not supported yet by the Windows
version. Now the parser does not set the Content-Type as {{octet-stream}}.
Thanks [~chrismattmann], [~gagravarr] and [~lfcnassif] for supporting me.
> StringsParser: a simple strings-based parser for Tika
> -----------------------------------------------------
>
> Key: TIKA-1541
> URL: https://issues.apache.org/jira/browse/TIKA-1541
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Giuseppe Totaro
> Assignee: Chris A. Mattmann
> Attachments: TIKA-1541.TotaroMattmann.020615.patch.txt,
> TIKA-1541.TotaroMattmann.020615.patch.txt,
> TIKA-1541.TotaroMattmannBurchNassif.020715.patch, TIKA-1541.patch,
> testOCTET_header.dbase3
>
>
> I thought to implement an extremely simple implementation of
> {{StringsParser}}, a parser based on the {{strings}} command (or
> {{strings}}-alternative command), instead of using the dummy {{EmptyParser}}
> for undetected files. It is a preliminary work (you can see a lot of todos).
> It is inspired by the work on {{TesseractOCRParser}}. You can find the patch
> in attachment.
> I created a GitHub
> [repository|https://github.com/giuseppetotaro/StringsParser] for sharing the
> code. As first test, you can clone the repo, build the code using the
> {{build.sh}} script, and then run the parser using the {{run.sh}} script on
> some [govdocs1|http://digitalcorpora.org/corpora/govdocs] files (grabbed from
> "016" subset) detected as {{application/octet-stream}}. The latter script
> launches a simple {{StringsTest}} class for testing.
> I hope you will find the {{StringsParser}} a good solution for extracting
> ASCII strings from undetected filetypes. As far as I understood, many
> "sophisticated" forensics tools work in a similar manner for indexing
> purposes. They use a sort of {{strings}} command against files that they are
> not able to detect.
> In addition to run {{strings}} on undetected files, the {{StringsParser}}
> launches the {{file}} command on undetected files and then writes the output
> in the {{strings:file_output}} property (I noticed that sometimes the
> {{file}} command is able to detect the media type for documents not detected
> by Tika).
> Finally, you can fine an old discussion about this topic
> [here|http://lucene.472066.n3.nabble.com/Default-MIME-Type-td645215.html].
> Thanks [~chrismattmann].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
