davidanthoff commented on PR #609: URL: https://github.com/apache/arrow-julia/pull/609#issuecomment-5489012163
Very cool! One question: could ArrowCore.jl be a real standalone, dependency-free package, similar to how ArrowStrings.jl is split out already? From reading the code, it looks like ArrowCore.jl could contain what it has at the moment in this branch, plus cdata.jl, and the uncompressed IPC read/write path (with the vendored FlatBuffers/Meta layer). And the Tables.jl facade, ArrowTypes lowering, and the compression codecs could then stay in Arrow.jl on top of it. As far as I can tell this split already exists at the code level: ArrowCore only imports Mmap, cdata.jl only uses ArrowCore, and neither ipc_read.jl nor ipc_write.jl references Tables at all — the codecs only appear at the buffer (de)compression step. So ArrowCore.jl would depend on Base and stdlibs only: no binary deps (Lz4_jll/Zstd_jll), no Tables.jl. The two places where the package boundary doesn't quite match the code today seem small: EnumX (but Meta is generated, so the generator could emit self-contained enum modules instead), and compression (which could be a plain dispatch seam — ArrowCore.jl defines compress/decompress generics and errors helpfully on compressed buffers, Arrow.jl keeps its hard codec deps and implements them; so using Arrow behaves exactly as it does now. The reason I'm asking: we could use this in the Julia VS Code extension. The REPL process could use ArrowCore.jl and ArrowStrings.jl to hand large tables to the extension's interactive table viewer — write an uncompressed IPC file, and the TypeScript side reads it with Arrow JS, with the file-format footer giving the viewer random access to just the visible row range. That would be really nice! But the way we ship code that loads into the user's REPL process rules out anything with a binary dependency. And we also couldn't use anything that pulls in Tables.jl, because that would pin the Tables.jl version users see in their REPL to whatever we ship. So this could only work if there is a core package without those deps. (I'd also have some uses for this in Queryverse, but the VS Code one probably the more interesting one) And, as all ideas these days, heavily Claude helped ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
