GitHub user luanmm added a comment to the discussion: Guidance on zip-like stores
I found OpenDAL recently when I was working with ZIP files, in an efficient way (from a streaming/partial reading perspective, not full decompression), and it was very adherent to my use case (as the original library I was working on was doing HTTP buffered readings and other strategies that OpenDAL already considers). So I thought OpenDAL could make a move towards this "archive" scenario, as it already has very nice examples of opening files from multiple protocols and serving as an object store, for instance... or even the "ofs" project. In that way, archive files could be read, generally, as "virtual file systems" somehow. When I open a ZIP file, what I'm doing, in practice, is listing files like I would do when I open an S3 bucket. Even Windows (which I haven't touched for a while, by the way) has some kind of "folder-like" integration in its iconic "file explorer". In terms of API, a ZIP file could be "opened" in OpenDAL as any other operator. The particular case here is that it can be thought of as "an operator inside another operator", as I may have ZIP files inside S3 buckets, or served directly via HTTP or even the filesystem. This is the only "challenge" that I see in terms of how OpenDAL could handle this kind of scenario. Example of how this could be represented as URIs: ```bash # 1) as a "path", directly (should be parsed considering the extension, which is not great) s3://<bucket>/folder-a/file.zip/inner-file-1.txt # 2) same as above, but with a "hint" from protocol, as used in Git and others s3+zip://<bucket>/folder-a/file.zip/inner-file-1.txt ``` There are other more complex scenarios, like nested "VFS": ```bash s3://<bucket>/data.iso/file-a.zip/file-b.zip/inner-file-a.7z/target-file.txt # considering "hints" in protocol (yeah, just to be a little bit crazy here) s3+iso+zip+7z://<bucket>/data.iso/file-a.zip/file-b.zip/inner-file-a.7z/target-file.txt ``` This is taking it to the limit and considering some edge cases (which aren't impossible anyway). Thinking from an architectural perspective, it seems to me that this would make some kind of "processor" entity emerge, that could be enabled when needed, in options/protocol-hints or as a layer (e.g. a middleware that can intercept processing, taking only the part of the URI that it has interest in, like `.zip`, then passes the operator that handles ZIP-like VFS to the next one, etc). Well... this is just a big "brainstorm" here, just to say that I would think that OpenDAL would be even more incredible supporting this kind of scenario, as it just rounds out what it currently supports. GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-16931651 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
