This sounds really interesting, how about adding the wasm build (C++) to the releases? I've done a lot of asm.js work (different from wasm) in the past, but my assumption would be that using Rust instead of C++ as source for wasm should result in smaller wasm binaries. Rust Arrow doesn't really use exotic solutions, eg. simd or tokio dependency can be turned off.
Having DataFusion + some performant data access in browsers or even in node.js would be useful. Not needing to build fancy HTTP/GraphQL API over the Rust/C++ impl. but moving the data processing code to the client is viable for "small" workloads. Ofc if JS Arrow lands Flight support this may become less of an issue, but AFAIK it's gRPC based which would need setting up a gRPC reverse proxy for C++/Rust Arrow. Overall both the code-duplication and feature fragmentation would decrease by using a single source (like you don't have a full Python impl. for obvious reasons) Best regards, Adam Lippai On Tue, Jul 14, 2020 at 4:27 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > Fwiw, I believe at least the core c++ library already can be compiled to > wasm. I think perspective does this [1] > > > I'm curious What are you hoping to achieve with embedded wasm in spark? > > Thanks, > Micah > > [1] https://perspective.finos.org/ > > On Tuesday, July 14, 2020, Brian Hulette <hulet...@gmail.com> wrote: > > > That sounds great! I'd like to have some support for using the rust > and/or > > C++ libraries in the browser via wasm as well. > > As long as the community is ok with your overall approach "to add > compiler > > conditionals around any I/O features and libc dependent features of these > > two libraries," I think it may be best to start with a PR and discuss > > specifics from there. > > > > Do any rust contributors have objections to this? > > > > Brian > > > > On Mon, Jul 13, 2020 at 9:42 PM RJ Atwal <atwal...@gmail.com> wrote: > > > > > Hi all, > > > > > > Looking for guidance on how to submit a design and PR to add WASM32 > > support > > > to apache arrow's rust libraries. > > > > > > I am looking to use the arrow library to pass data in arrow format > > between > > > the host spark environment and UDFs defined in WASM . > > > > > > I created the following JIRA ticket to capture the work > > > https://issues.apache.org/jira/browse/ARROW-9453 > > > > > > Thanks, > > > RJ > > > > > >
