Somewhat related, I tried to compile DataFusion to WASM and it didn’t work because of some dependencies: https://issues.apache.org/jira/projects/ARROW/issues/ARROW-11615. I wonder whether DataFusion could have a feature flag for only shipping what is WASM compatible?
On Feb 15, 2021 at 12:13:04, Andrew Lamb <al...@influxdata.com> wrote: > Also, unrelated, is there a schedule for the sync calls? Will try and > > carve out some free time for the next one :) > > It is every other Wednesday at noon EST. Here is the original announcement > with more details: > > https://lists.apache.org/thread.html/raa72e1a8a3ad5dbb8366e9609a041eccca87f85545c3bc3d85170cfc%40%3Cdev.arrow.apache.org%3E > > > On Sun, Feb 14, 2021 at 8:29 AM Ruan Pearce-Authers <r...@reservoirdb.com> > wrote: > > I'd be interested in helping spec this out, it's especially tricky atm to > > track down issues when integrating DataFusion into the same binary as other > > medium/large dependencies. > > > Recently hit a really specific issue where DataFusion depends on Parquet, > > which supports various compression algs, including Brotli, and actix-web > > also depends on a slightly different Rust implementation of Brotli. Both of > > these Brotli libs package the same underlying C lib separately, resulting > > in multiply-defined symbols compiling using msvc (and maybe on other > > platforms? didn't test in CI in the end). > > > Got a quick interim hack [1] in place for my use case which doesn't really > > use Parquet, so it's not pressing, but would be awesome to sort this > > properly upstream. > > > I guess the only major tradeoff of having a comprehensive feature setup is > > that it could make testing slightly harder, in terms of making sure no-one > > breaks the build for specific feature combinations; this can always be > > mitigated with more CI though (yay, unlimited Actions minutes for public > > repos). > > > Also, unrelated, is there a schedule for the sync calls? Will try and > > carve out some free time for the next one :) > > > [1] > > > https://github.com/reservoirdb/arrow/commit/e63e157927a552ecf1a6f63ec401f0b6157b5468 > > > -----Original Message----- > > From: Andrew Lamb <al...@influxdata.com> > > Sent: 14 February 2021 11:14 > > To: dev <dev@arrow.apache.org> > > Subject: [Rust] [DataFusion] Topic for next Rust Sync Call > > > I would like to add the following item to the agenda call for the next > > Rust sync call: > > > Dependencies > > > Background: As the dependency stack gets larger, it will be harder to use > > DataFusion as an embedded query engine and the compile / dev times will get > > higher. > > > As we expand the supported functions of DataFusion this problem is likely > > to get worse. For example > > https://github.com/apache/arrow/pull/9243#discussion_r575716759 and > > https://github.com/apache/arrow/pull/9139 > > > Proposal: Add Rust "features" to the datafusion crate and make many of the > > new dependencies optional (so that we had features like regex and unicode > > and hash which would only pull in the dependencies / have those functions > > if the features were enabled.) This approach has worked well for Arrow > > (which has only chrono and num as required dependencies) > > >
