Hi Joe,
Thank you for working on that.
The one question I have is: are you willing to help us maintain Arrow
C++ on the long term? The logic you're adding in
https://github.com/apache/arrow/pull/35672 is quite delicate; also I
don't think anyone among us is a Webassembly expert, which means that we
might break things unwillingly. So while it would be great to get Arrow
C++ to work with WASM, a dedicated expert is needed to help maintain and
debug WASM support in the future.
Regards
Antoine.
Le 03/07/2023 à 17:29, Joe Marshall a écrit :
Hi,
I'm a pyodide developer amongst other things (webassembly cpython intepreter)
and I've got some PRs in progress on arrow relating to webassembly support. I
wondered if it might be worth discussing my broader ideas for this on the list
or at the biweekly development meeting?
So far I have 35176 in, which makes arrow run on a single thread. This is
needed because in a lot of webassembly environments (browsers at least,
pyodide), threading isn't available or is heavily constrained.
With that I've aimed to make it relatively transparent to users, so that things
like datasets and acero mostly just work (but slower obviously). It's kind of
fiddly in the arrow code but working, and means users can port things easily.
Once that is in, the plan is to submit a following pr that adds cmake presets
for emscripten which can build the cpp libraries and pyarrow for pyodide. I've
hacked this together in a build already, it's a bit fiddly and needs a load of
tidying up, but I'm confident it can be done.
Essentially, I'm wanting to get this stuff in because pandas is moving towards
arrow as a pretty much required dependency, and webassembly is a pandas
platform, as well as
being an official python platform, so it would be great to get it working in
pyodide without us needing to maintain a load of patches. I guess it could also
come in handy with various container platforms that are moving to webassembly.
Basically I thought it's probably worth a bit of a heads up relating to this,
as I know the bigger picture of things is often hard to see from just pull
requests.
Thanks
Joe
This message and any attachment are intended solely for the addressee
and may contain confidential information. If you have received this
message in error, please contact the sender and delete the email and
attachment.
Any views or opinions expressed by the author of this email do not
necessarily reflect the views of the University of Nottingham. Email
communications with the University of Nottingham may be monitored
where permitted by law.
