[
https://issues.apache.org/jira/browse/TIKA-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18053943#comment-18053943
]
Tim Allison edited comment on TIKA-4630 at 1/23/26 2:49 PM:
------------------------------------------------------------
Using embeddedRelationshipID as I did in the package parser was a mistake. I
shouldn't have conflated this microsoft internal designator with other file
formats, where it really is something different. I now understand why you used
that and you were right to do so.
In 3.x, I propose we introduce a "X-TIKA:internalPath" that will capture what
the container file says the internal path (and file name) is for a file. This
will duplicate embeddedRelationShipId in 3.x, and we can remove
embeddedRelationshipId in the PackageParser for 4.x.
There are two ways that we should have been able to get the test-documents.tar
correct, one from the parent file, and one from the stored filename in the gz
stream.
* One is that our unit test framework getRecursiveMetadata was not passing in
the file name. I fixed that.
* Within the Compressor parser, we were already doing some name based regexes
to remove the compressor part from the incoming file name. Weirdly, though, we
weren't handling tar.gz/.tgz. I fixed that.
* The gzipcompressorinputstream has a metadata field on it that includes the
file name for the original file name that was compressed. We were not
processing this. IIUC, gzip is the only compressor stream that stores the file
name -- let me know if I'm wrong. I added extraction for this file name from
gzip's metadata.
With the updated code, these are the relevant metadata values:
{noformat}
0: Content-Type : application/gzip
0: resourceName : test-documents.tgz
1: Content-Type : application/vnd.ms-excel
1: X-TIKA:embedded_id : 2
1: X-TIKA:embedded_id_path : /1/2
1: X-TIKA:embedded_resource_path : /test-documents.tar/testEXCEL.xls
1: X-TIKA:internalPath : test-documents/testEXCEL.xls
1: embeddedRelationshipId : test-documents/testEXCEL.xls
1: resourceName : test-documents/testEXCEL.xls
2: Content-Type : text/html; charset=windows-1252
2: X-TIKA:embedded_id : 3
2: X-TIKA:embedded_id_path : /1/3
2: X-TIKA:embedded_resource_path : /test-documents.tar/testHTML.html
2: X-TIKA:internalPath : test-documents/testHTML.html
2: embeddedRelationshipId : test-documents/testHTML.html
2: resourceName : test-documents/testHTML.html
3: Content-Type : image/png
3: X-TIKA:embedded_id : 5
3: X-TIKA:embedded_id_path : /1/4/5
3: X-TIKA:embedded_resource_path :
/test-documents.tar/testOpenOffice2.odt/thumbnail.png
3: X-TIKA:internalPath : Thumbnails/thumbnail.png
3: resourceName : Thumbnails/thumbnail.png
4: Content-Type : application/pdf
4: X-TIKA:embedded_id : 6
4: X-TIKA:embedded_id_path : /1/4/6
4: X-TIKA:embedded_resource_path :
/test-documents.tar/testOpenOffice2.odt/thumbnail.pdf
4: X-TIKA:internalPath : Thumbnails/thumbnail.pdf
4: resourceName : Thumbnails/thumbnail.pdf
5: Content-Type : application/vnd.oasis.opendocument.text
5: X-TIKA:embedded_id : 4
5: X-TIKA:embedded_id_path : /1/4
5: X-TIKA:embedded_resource_path : /test-documents.tar/testOpenOffice2.odt
5: X-TIKA:internalPath : test-documents/testOpenOffice2.odt
5: embeddedRelationshipId : test-documents/testOpenOffice2.odt
5: resourceName : test-documents/testOpenOffice2.odt
6: Content-Type : application/pdf
6: X-TIKA:embedded_id : 7
6: X-TIKA:embedded_id_path : /1/7
6: X-TIKA:embedded_resource_path : /test-documents.tar/testPDF.pdf
6: X-TIKA:internalPath : test-documents/testPDF.pdf
6: embeddedRelationshipId : test-documents/testPDF.pdf
6: resourceName : test-documents/testPDF.pdf
7: Content-Type : application/vnd.ms-powerpoint
7: X-TIKA:embedded_id : 8
7: X-TIKA:embedded_id_path : /1/8
7: X-TIKA:embedded_resource_path : /test-documents.tar/testPPT.ppt
7: X-TIKA:internalPath : test-documents/testPPT.ppt
7: embeddedRelationshipId : test-documents/testPPT.ppt
7: resourceName : test-documents/testPPT.ppt
8: Content-Type : application/rtf
8: X-TIKA:embedded_id : 9
8: X-TIKA:embedded_id_path : /1/9
8: X-TIKA:embedded_resource_path : /test-documents.tar/testRTF.rtf
8: X-TIKA:internalPath : test-documents/testRTF.rtf
8: embeddedRelationshipId : test-documents/testRTF.rtf
8: resourceName : test-documents/testRTF.rtf
9: Content-Type : text/plain; charset=windows-1252
9: X-TIKA:embedded_id : 10
9: X-TIKA:embedded_id_path : /1/10
9: X-TIKA:embedded_resource_path : /test-documents.tar/testTXT.txt
9: X-TIKA:internalPath : test-documents/testTXT.txt
9: embeddedRelationshipId : test-documents/testTXT.txt
9: resourceName : test-documents/testTXT.txt
10: Content-Type : application/msword
10: X-TIKA:embedded_id : 11
10: X-TIKA:embedded_id_path : /1/11
10: X-TIKA:embedded_resource_path : /test-documents.tar/testWORD.doc
10: X-TIKA:internalPath : test-documents/testWORD.doc
10: embeddedRelationshipId : test-documents/testWORD.doc
10: resourceName : test-documents/testWORD.doc
11: Content-Type : application/xml
11: X-TIKA:embedded_id : 12
11: X-TIKA:embedded_id_path : /1/12
11: X-TIKA:embedded_resource_path : /test-documents.tar/testXML.xml
11: X-TIKA:internalPath : test-documents/testXML.xml
11: embeddedRelationshipId : test-documents/testXML.xml
11: resourceName : test-documents/testXML.xml
12: Content-Type : application/x-gtar
12: X-TIKA:embedded_id : 1
12: X-TIKA:embedded_id_path : /1
12: X-TIKA:embedded_resource_path : /test-documents.tar
12: X-TIKA:internalPath : test-documents.tar
12: resourceName : test-documents.tar
{noformat}
was (Author: [email protected]):
Using embeddedRelationshipID as I did in the package parser was a mistake. I
shouldn't have conflated this microsoft internal designator with other file
formats, where it really is something different. I now understand why you used
that and you were right to do so.
In 3.x, I propose we introduce a "X-TIKA:internalPath" that will capture what
the container file says the internal path (and file name) is for a file. This
will duplicate embeddedRelationShipId in 3.x, and we can remove
embeddedRelationshipId in the PackageParser for 4.x.
There are two ways that we should have been able to get the test-documents.tar
correct, one from the parent file, and one from the stored filename in the gz
stream.
* One is that our unit test framework getRecursiveMetadata was not passing in
the file name. I fixed that.
* Within the Compressor parser, we were already do some name based regexes to
remove the compressor part from the incoming file name. Weirdly, though, we
weren't handling tar.gz/.tgz. I fixed that.
* The gzipcompressorinputstream has a metadata field on it that includes the
file name for the original file name that was compressed. We were not
processing this. IIUC, gzip is the only compressor stream that stores the file
name -- let me know if I'm wrong. I added extraction for this file name from
gzip's metadata.
With the updated code, these are the relevant metadata values:
{noformat}
0: Content-Type : application/gzip
0: resourceName : test-documents.tgz
1: Content-Type : application/vnd.ms-excel
1: X-TIKA:embedded_id : 2
1: X-TIKA:embedded_id_path : /1/2
1: X-TIKA:embedded_resource_path : /test-documents.tar/testEXCEL.xls
1: X-TIKA:internalPath : test-documents/testEXCEL.xls
1: embeddedRelationshipId : test-documents/testEXCEL.xls
1: resourceName : test-documents/testEXCEL.xls
2: Content-Type : text/html; charset=windows-1252
2: X-TIKA:embedded_id : 3
2: X-TIKA:embedded_id_path : /1/3
2: X-TIKA:embedded_resource_path : /test-documents.tar/testHTML.html
2: X-TIKA:internalPath : test-documents/testHTML.html
2: embeddedRelationshipId : test-documents/testHTML.html
2: resourceName : test-documents/testHTML.html
3: Content-Type : image/png
3: X-TIKA:embedded_id : 5
3: X-TIKA:embedded_id_path : /1/4/5
3: X-TIKA:embedded_resource_path :
/test-documents.tar/testOpenOffice2.odt/thumbnail.png
3: X-TIKA:internalPath : Thumbnails/thumbnail.png
3: resourceName : Thumbnails/thumbnail.png
4: Content-Type : application/pdf
4: X-TIKA:embedded_id : 6
4: X-TIKA:embedded_id_path : /1/4/6
4: X-TIKA:embedded_resource_path :
/test-documents.tar/testOpenOffice2.odt/thumbnail.pdf
4: X-TIKA:internalPath : Thumbnails/thumbnail.pdf
4: resourceName : Thumbnails/thumbnail.pdf
5: Content-Type : application/vnd.oasis.opendocument.text
5: X-TIKA:embedded_id : 4
5: X-TIKA:embedded_id_path : /1/4
5: X-TIKA:embedded_resource_path : /test-documents.tar/testOpenOffice2.odt
5: X-TIKA:internalPath : test-documents/testOpenOffice2.odt
5: embeddedRelationshipId : test-documents/testOpenOffice2.odt
5: resourceName : test-documents/testOpenOffice2.odt
6: Content-Type : application/pdf
6: X-TIKA:embedded_id : 7
6: X-TIKA:embedded_id_path : /1/7
6: X-TIKA:embedded_resource_path : /test-documents.tar/testPDF.pdf
6: X-TIKA:internalPath : test-documents/testPDF.pdf
6: embeddedRelationshipId : test-documents/testPDF.pdf
6: resourceName : test-documents/testPDF.pdf
7: Content-Type : application/vnd.ms-powerpoint
7: X-TIKA:embedded_id : 8
7: X-TIKA:embedded_id_path : /1/8
7: X-TIKA:embedded_resource_path : /test-documents.tar/testPPT.ppt
7: X-TIKA:internalPath : test-documents/testPPT.ppt
7: embeddedRelationshipId : test-documents/testPPT.ppt
7: resourceName : test-documents/testPPT.ppt
8: Content-Type : application/rtf
8: X-TIKA:embedded_id : 9
8: X-TIKA:embedded_id_path : /1/9
8: X-TIKA:embedded_resource_path : /test-documents.tar/testRTF.rtf
8: X-TIKA:internalPath : test-documents/testRTF.rtf
8: embeddedRelationshipId : test-documents/testRTF.rtf
8: resourceName : test-documents/testRTF.rtf
9: Content-Type : text/plain; charset=windows-1252
9: X-TIKA:embedded_id : 10
9: X-TIKA:embedded_id_path : /1/10
9: X-TIKA:embedded_resource_path : /test-documents.tar/testTXT.txt
9: X-TIKA:internalPath : test-documents/testTXT.txt
9: embeddedRelationshipId : test-documents/testTXT.txt
9: resourceName : test-documents/testTXT.txt
10: Content-Type : application/msword
10: X-TIKA:embedded_id : 11
10: X-TIKA:embedded_id_path : /1/11
10: X-TIKA:embedded_resource_path : /test-documents.tar/testWORD.doc
10: X-TIKA:internalPath : test-documents/testWORD.doc
10: embeddedRelationshipId : test-documents/testWORD.doc
10: resourceName : test-documents/testWORD.doc
11: Content-Type : application/xml
11: X-TIKA:embedded_id : 12
11: X-TIKA:embedded_id_path : /1/12
11: X-TIKA:embedded_resource_path : /test-documents.tar/testXML.xml
11: X-TIKA:internalPath : test-documents/testXML.xml
11: embeddedRelationshipId : test-documents/testXML.xml
11: resourceName : test-documents/testXML.xml
12: Content-Type : application/x-gtar
12: X-TIKA:embedded_id : 1
12: X-TIKA:embedded_id_path : /1
12: X-TIKA:embedded_resource_path : /test-documents.tar
12: X-TIKA:internalPath : test-documents.tar
12: resourceName : test-documents.tar
{noformat}
> embeddedRelationshipId is missing from tar files that are children of gzip
> files (i.e. tarballs)
> ------------------------------------------------------------------------------------------------
>
> Key: TIKA-4630
> URL: https://issues.apache.org/jira/browse/TIKA-4630
> Project: Tika
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.2.3
> Reporter: Iachimoe
> Assignee: Tim Allison
> Priority: Major
> Attachments: tika-embedded-reln-jira.PATCH
>
>
> For some context, my usecase requires me to be able to precisely locate
> individual files within nested archives, using only metadata from tika.
> I have made this work for zip files using a combination of
> EMBEDDED_RELATIONSHIP_ID and EMBEDDED_RESOURCE_PATH. EMBEDDED_RESOURCE_PATH
> on its own does not work because it omits information about which directory a
> file is stored in within an archive. When trying to make it work for tgz
> files, the approach breaks because, while the parser quite reasonably
> considers "archive.tar" to be a child of "archive.tar.gz", the corresponding
> entry metadata lacks an EMBEDDED_RELATIONSHIP_ID. This makes it difficult to
> construct a path of the form
> "archive.tar.gz/archive.tar/my_folder/my_file.txt".
> I attach a rough solution (patch is against branch_3x), with unit test, that
> appears to work for my use case. The approach is for the
> RecursiveParserWrapper to set EMBEDDED_RELATIONSHIP_ID once parsing has
> completed if it has not been set during the parsing process. It sets it to
> the value of RESOURCE_NAME_KEY , which is probably not ideal, but in practice
> it seems to contain the filename and parent folders within the archive.
> Although the aim was to make this work for tar files, the unit test
> demonstrates that it has a similar effect for chidren of ODT files (this was
> an accident as the test archive that was already in the codebase happened to
> have ODT files).
> I could not find descriptions of exactly what EMBEDDED_RELATIONSHIP_ID and
> EMBEDDED_RESOURCE_PATH are meant to represent, so I'm not sure whether this
> approach is heading in the right direction, though it seems to work for my
> usecase. A future enhancement could be for the recursive parser to include a
> new field with the complete path to each file (again, imagining nested
> archives it might look something like
> "archive.tar.gz/archive.tar/folder1/nested.zip/folder2/file.txt").
> Looking forward to everybody's thoughts, and happy to make refinements to the
> proof of concept attached as needed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
