[
https://issues.apache.org/jira/browse/TIKA-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18053735#comment-18053735
]
Iachimoe commented on TIKA-4630:
--------------------------------
Thank you for the swift and detailed response - it is much appreciated!
1) In my opinion the current approach of treating a tgz as a parent gz file
containing a child tar file is actually the most sensible way to approach it,
as it reflects how tarballs actually work and are used on the command line. As
you allude to when you refer to tika using "embedded-1", technically the tar
file has no name. However, based on the behaviour of the command line tool, the
least surprising approach is to follow what the command line gzip tool does,
which is to name the uncompressed version of archive.tgz as archive.tar . This
is also what Tika currently does when determining the resource name - it uses
GzipUtils.getUncompressedFileName from commons compress. This in turn is how my
patch ends up producing an EMBEDDED_RELATIONSHIP_ID of archive.tar
2) I think the most fundamental improvement would be to have a field that
reliably contains the directory path (if only within the immediate parent
archive) to the file. When I was doing my zip implementation (a couple of years
ago now), the two options were RESOURCE_NAME or EMBEDDED_RELATIONSHIP_ID . At
the time I chose to go with EMBEDDED_RELATIONSHIP_ID , on the grounds that it
seemed more likely to be designed specifically for that purpose, as opposed to
RESOURCE_NAME which I imagined could be prone to mangling by individual parsers
for different formats. Because the tarball (and it seems ODT) doesn't have
EMBEDDED_RELATIONSHIP_ID , my only option becomes RESOURCE_NAME , which is why
my patch works as it does. If I could be sure that RESOURCE_NAME name behaves
consistently, then I could just use that everywhere in place of
EMBEDDED_RELATIONSHIP_ID , but as I mentioned above I expect that RESOURCE_NAME
could be prone to getting changed by individual parsers? As you suggested,
EMBEDDED_RESOURCE_PATH could be a good place to include directory information,
but I guess at this point it would be a breaking change that could have
backwards compatibility challenges?
3) Thanks for the info on the embedded_ids, it may well be simpler and more
reliable that what I have been doing. That said (correct me if I'm wrong), they
seem to only relate to finding the (sub)archive to which the file belongs, and
we still need a solution for (2) above in order to find the directory within
the archive.
{code:java}
Secondly, perhaps we trust path based information from the container file so
that we use "test-documents/testWORD.doc". I think we do that with zip? Why
aren't we doing it with tar? {code}
Actually files within the tarball do contain an EMBEDDED_RELATIONSHIP_ID with
the folder information. My initial issue primarily just related to needing some
value in EMBEDDED_RELATIONSHIP_ID of the tar file itself, the most sensible
value being the the name of the tgz but with tgz replaced by tar, as that
corresponds to how the command line gzip tool works. It's also worth noting
that this convention already exists within tika. Running my unit test against
an unchanged version of the production code, I see the following for the first
file within the archive:
{code:java}
X-TIKA:embedded_resource_path: /test-documents.tar/testEXCEL.xls
embeddedRelationshipId: test-documents/testEXCEL.xls
{code}
So Tika already refers to the tar by name using the .tar suffix (as I mentioned
above, it gets this by calling GzipUtils.getUncompressedFileName from commons
compress). This is why I think it's reasonable for the tar to have
embeddedRelationshipId with a value of test-documents.tar (it also means that I
can re-use the path construction code that I wrote for zip files without
changes, and saves me from having to do string searches for embedded_1).
This is turning into a can of worms, and the ODT files are a can of worms
within a can of worms :) Those are not a priority for me right now, but given
that my test case threw up the issue, it might be worth considering them.
Again, IMHO the ideal would be to provide a way of working backwards from tika
metadata such that the file can be unamigously located within the original
(potentially nested) archives.
In terms of a way forward, I think the most important thing is to have a field
that can be reliably used to determine the directory that where a file exists
within an archive. Currently the options are RESOURCE_NAME (which I doubt can
be relied upon, but I could be wrong) and EMBEDDED_RELATIONSHIP_ID , which
based on your comment is only used as a backoff, though it seems to actually
contain the correct data most of the time. And given that for a tarball, the
EMBEDDED_RESOURCE_PATH already references the .tar file, I think that using
<archivename>.tar as the EMBEDDED_RELATIONSHIP_ID of the tar file itself makes
sense and is also consistent with how nested zip files are handled.
Providing metadata so that embedded content within ODTs can be located reliably
also looks like it would be challenging, but perhaps best to leave that as a
separate issue with the ODT parser?
Hopefully I haven't been wittering on too much :) While I'm not very familiar
with the tika codebase, I'm happy to help as much as I can with moving forward
on this.
> embeddedRelationshipId is missing from tar files that are children of gzip
> files (i.e. tarballs)
> ------------------------------------------------------------------------------------------------
>
> Key: TIKA-4630
> URL: https://issues.apache.org/jira/browse/TIKA-4630
> Project: Tika
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.2.3
> Reporter: Iachimoe
> Assignee: Tim Allison
> Priority: Major
> Attachments: tika-embedded-reln-jira.PATCH
>
>
> For some context, my usecase requires me to be able to precisely locate
> individual files within nested archives, using only metadata from tika.
> I have made this work for zip files using a combination of
> EMBEDDED_RELATIONSHIP_ID and EMBEDDED_RESOURCE_PATH. EMBEDDED_RESOURCE_PATH
> on its own does not work because it omits information about which directory a
> file is stored in within an archive. When trying to make it work for tgz
> files, the approach breaks because, while the parser quite reasonably
> considers "archive.tar" to be a child of "archive.tar.gz", the corresponding
> entry metadata lacks an EMBEDDED_RELATIONSHIP_ID. This makes it difficult to
> construct a path of the form
> "archive.tar.gz/archive.tar/my_folder/my_file.txt".
> I attach a rough solution (patch is against branch_3x), with unit test, that
> appears to work for my use case. The approach is for the
> RecursiveParserWrapper to set EMBEDDED_RELATIONSHIP_ID once parsing has
> completed if it has not been set during the parsing process. It sets it to
> the value of RESOURCE_NAME_KEY , which is probably not ideal, but in practice
> it seems to contain the filename and parent folders within the archive.
> Although the aim was to make this work for tar files, the unit test
> demonstrates that it has a similar effect for chidren of ODT files (this was
> an accident as the test archive that was already in the codebase happened to
> have ODT files).
> I could not find descriptions of exactly what EMBEDDED_RELATIONSHIP_ID and
> EMBEDDED_RESOURCE_PATH are meant to represent, so I'm not sure whether this
> approach is heading in the right direction, though it seems to work for my
> usecase. A future enhancement could be for the recursive parser to include a
> new field with the complete path to each file (again, imagining nested
> archives it might look something like
> "archive.tar.gz/archive.tar/folder1/nested.zip/folder2/file.txt").
> Looking forward to everybody's thoughts, and happy to make refinements to the
> proof of concept attached as needed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
