Just wanted to note that our move from Arrow JS to Wasm arrow resulted in a pretty substantial speedup, although we solve none of the problems of exposing wasm arrow over JS since we use it directly from C++.
Initial PR: https://github.com/finos/perspective/pull/755 <https://github.com/finos/perspective/pull/755> Standalone Arrow C++ to wasm via emscripten: https://github.com/timkpaine/arrow-wasm-cpp <https://github.com/timkpaine/arrow-wasm-cpp> (Note that neither utilize all of arrow, just a carve out of IPC stuff) Tim Paine tim.paine.nyc 908.721.1185 > On Dec 9, 2021, at 14:18, Dominik Moritz <domor...@apache.org> wrote: > >> >> Arrow rust implementation is in another repository and has support for >> > Javascript/Webassembly : >> >> https://github.com/apache/arrow-rs/tree/master/arrow >> >> The release cadence for the Rust implementation is much higher than for >> the C++ implementation. Efficiencies might be gained by releasing Rust >> and Javascript point implementations together since then the process of >> creating and verifying signed software would minimize PMC workload.Arrow >> rust implementation is in another repository and has support for >> > Javascript/Webassembly : >> >> https://github.com/apache/arrow-rs/tree/master/arrow >> >> The release cadence for the Rust implementation is much higher than for >> the C++ implementation. Efficiencies might be gained by releasing Rust >> and Javascript point implementations together since then the process of >> creating and verifying signed software would minimize PMC workload. > > > The biggest challenge with making a web-library for Arrow with WASM is a > performant JS API. I think realistically, we will have a pure JS Arrow > library for a few years. Do you think we could sync the release processes > even if Arrow JS is not in the rust repo? If so, I would love to learn more > about how that process would work. > > On Dec 7, 2021 at 03:43:13, Benson Muite <benson_mu...@emailplus.org> wrote: > >> At the moment, the release is not packaged or signed. Thus one can only >> run the tests on the branch in the git repository. A script to do that >> on Linux is available at: >> >> https://github.com/bkmgit/arrow/blob/ARROW-14801/dev/release/verify-js.sh >> >> My understanding is that only PMC members can sign, at the moment not >> many seem to use Javascript extensively. Can create a script for >> generating the Javascript only release source package based on the >> current source packaging and release scripts, but a PMC member would >> need to have this signed and uploaded. >> >> @Dominik - was not aware of arrow-wasm, thanks. >> >> Arrow rust implementation is in another repository and has support for >> Javascript/Webassembly : >> >> https://github.com/apache/arrow-rs/tree/master/arrow >> >> The release cadence for the Rust implementation is much higher than for >> the C++ implementation. Efficiencies might be gained by releasing Rust >> and Javascript point implementations together since then the process of >> creating and verifying signed software would minimize PMC workload. >> >> Benson >> >> On 12/6/21 1:01 AM, Wes McKinney wrote: >> >> hi Dominik — can you provide instructions for how we should verify the >> >> release, aside from checking the GPG signature and checksums? >> >> >> On Sun, Nov 28, 2021 at 12:41 PM Dominik Moritz <domor...@apache.org> >> wrote: >> >>> >> >>> Are you talking about https://github.com/domoritz/arrow-wasm? It >> definitely >> >>> isn’t ready for prime time. The overhead of WASM, some issues with the >> Rust >> >>> implementation (some of which I think will be addressed with the Arrow2 >> >>> Rust migration), and the much larger bundle size make it not practical >> >>> right now. As the WASM ecosystem matures, we can reevaluate and maybe >> also >> >>> consider moving only some of the processing in WASM and leave the rest in >> >>> JS. I’m pretty excited about WASM and what it could bring to Arrow >> >>> especially when combined with WebGPU. >> >>> >> >>> Either way, I think we should release the 6.0.2 version soon. @PMC, could >> >>> you vote on the patch release? >> >>> >> >>> On Nov 28, 2021 at 04:33:41, Benson Muite <benson_mu...@emailplus.org> >> >>> wrote: >> >>> >> >>>> Rust implementation can be compiled to WebAssembly and is released >> >>>> biweekly. The Javascript version compiled from Rust may not satisfy all >> >>>> Javascript users, but maybe there could be some collaboration to reduce >> >>>> duplicated efforts? >> >>>> >> >>>> >> >>>> On 11/23/21 9:52 PM, Dominik Moritz wrote: >> >>>> >> >>>> Ahh, thank you for the clarification. There are no breaking changes >> in >> >>>> >> >>>> this point release, just fixes. >> >>>> >> >>>> >> >>>> @PMC, could you please vote on this point release. >> >>>> >> >>>> >> >>>> Would anyone volunteer as the release manager with me to give me a >> better >> >>>> >> >>>> understanding of the process? >> >>>> >> >>>> >> >>>> On Nov 23, 2021 at 13:09:47, Benson Muite <benson_mu...@emailplus.org> >> >>>> >> >>>> wrote: >> >>>> >> >>>> >> >>>>> https://issues.apache.org/jira/browse/ARROW-14801 >> >>>> >> >>>>> >> >>>> >> >>>>> Rust has its own repository and does frequent point releases: >> >>>> >> >>>>> https://github.com/apache/arrow-rs/tree/master/dev/release >> >>>> >> >>>>> >> >>>> >> >>>>> however, even point releases require 3 PMC binding +1 votes and API >> >>>> >> >>>>> breaking changes can only take place on major releases. >> >>>> >> >>>>> >> >>>> >> >>>>> Many of the tests for releases can be automated, possibly relieving >> some >> >>>> >> >>>>> of the PMC burden in the current process. Judgement on code quality >> and >> >>>> >> >>>>> software license is still required though[1]. Similarly, releases need >> >>>> >> >>>>> to be signed. >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> [1] https://infra.apache.org/release-publishing.html >> >>>> >> >>>>> >> >>>> >> >>>>> On 11/23/21 7:33 PM, Dominik Moritz wrote: >> >>>> >> >>>>> >> >>>> >> >>>>> I tested Node v14.18.1 and tests pass. I think we can go ahead and >> >>>> make a >> >>>> >> >>>>> >> >>>> >> >>>>> release. >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> @Benson, could you help me update the script to work off of branches. I >> >>>> >> >>>>> >> >>>> >> >>>>> don’t know what the expected process for release verification is. I’d >> be >> >>>> >> >>>>> >> >>>> >> >>>>> happy to adopt another process. >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> On Nov 20, 2021 at 09:57:53, Dominik Moritz <domor...@apache.org> >> wrote: >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>> Thanks for catching that. >> >>>> >> >>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>> Jest is used for running the tests and jest supports node 14.15. Could >> >>>> we >> >>>> >> >>>>> >> >>>> >> >>>>>> switch to node 14.15 instead of 14.0 for this test? >> >>>> >> >>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>> On Nov 20, 2021 at 05:37:00, Benson Muite <benson_mu...@emailplus.org >>> >> >>>> >> >>>>> >> >>>> >> >>>>>> wrote: >> >>>> >> >>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Hi, >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Tested this on AlmaLinux 8. Following steps: >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> export NVM_DIR="`pwd`/.nvm" >> >>>> >> >>>>> >> >>>> >> >>>>>>> mkdir -p $NVM_DIR >> >>>> >> >>>>> >> >>>> >> >>>>>>> curl -o- >> >>>> >> >>>>> >> >>>> >> >>>>>>> https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | \ >> >>>> >> >>>>> >> >>>> >> >>>>>>> PROFILE=/dev/null bash >> >>>> >> >>>>> >> >>>> >> >>>>>>> [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> nvm install --lts >> >>>> >> >>>>> >> >>>> >> >>>>>>> npm install -g yarn >> >>>> >> >>>>> >> >>>> >> >>>>>>> git clone https://github.com/apache/arrow >> >>>> >> >>>>> >> >>>> >> >>>>>>> cd arrow >> >>>> >> >>>>> >> >>>> >> >>>>>>> git checkout release-6.0.2-js >> >>>> >> >>>>> >> >>>> >> >>>>>>> cd js >> >>>> >> >>>>> >> >>>> >> >>>>>>> yarn --frozen-lockfile >> >>>> >> >>>>> >> >>>> >> >>>>>>> yarn run-s clean:all lint build >> >>>> >> >>>>> >> >>>> >> >>>>>>> yarn test >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Tests pass. >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> yarn 1.22.17 >> >>>> >> >>>>> >> >>>> >> >>>>>>> npm 8.1.0 >> >>>> >> >>>>> >> >>>> >> >>>>>>> node 16.13.0 >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Tests also pass on >> >>>> >> >>>>> >> >>>> >> >>>>>>> node 17.0.0 >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Node 14 is supported until 2023, however if one tries to use Node 14, >> >>>> >> >>>>> >> >>>> >> >>>>>>> one gets the error: >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> jest@27.0.6: The engine "node" is incompatible with this module. >> >>>> >> >>>>> >> >>>> >> >>>>>>> Expected version "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0". Got >> >>>> >> >>>>> >> >>>> >> >>>>>>> "14.0.0" >> >>>> >> >>>>> >> >>>> >> >>>>>>> error Found incompatible module. >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> The current release verification script could be update to support >> >>>> >> >>>>> >> >>>> >> >>>>>>> testing directly from a branch if this will be the point release >> >>>> process >> >>>> >> >>>>> >> >>>> >> >>>>>>> in future. >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> On 11/20/21 12:25 AM, Dominik Moritz wrote: >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Hi, >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> I would like to propose a patch release for Arrow JS. The release is >> >>>> >> >>>>> >> >>>> >> >>>>>>> forked >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> off of maint-6.0.x and available at >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> https://github.com/apache/arrow/tree/release-6.0.2-js. >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> The release contains two fixes for the js bundle: >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> ARROW-14773: [JS] Fix sourcemap paths >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> <https://github.com/apache/arrow/pull/11741> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> ARROW-14774: [JS] Correct package exports >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> <https://github.com/apache/arrow/pull/11742> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> [ ] +1 Release this as Apache Arrow JS 6.0.2 >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> [ ] +0 >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> [ ] -1 Do not release this as Apache Arrow JS 6.0.2 because... >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Thank you, >> >>>> >> >>>>> >> >>>> >> >>>>>>> >> >>>> >> >>>>> >> >>>> >> >>>>>>> Dominik >> >>>> >> >>>>> >> >>>> >> >>>> >> >> >>
