There's nothing stopping us from transmitting HTTP/2 or another binary protocol over UCX. You can think of UCX as a transport layer abstraction library which allows transparently taking advantage of things like RDMA over InfiniBand / RoCE, inter-process shared memory, TCP sockets, etc.
The other thing is if we think about something like handling GPU memory from Arrow CUDA in Flight, it would be nice to be able to be able to take advantage of the GPU RDMA features of libraries like UCX and libfabric without having to implement something like a HTTP/2 parser on the GPU. -Keith On Thu, Sep 9, 2021 at 7:48 AM Antoine Pitrou <anto...@python.org> wrote: > > Le 09/09/2021 à 12:34, Yibo Cai a écrit : > > Hi, > > > > We have some rough ideas of applying Flight in HPC (High Performance > > Computation). Would like to hear comments. > > > > HPC infrastructure normally leverages RDMA for fast data transfer among > > storage nodes and compute nodes. Computation tasks are dispatched to > > compute nodes with best fit resources. > > > > Concretely, we are investigating porting UCX as Flight transport layer. > > UCX is a communication framework for modern networks. [1] > > Besides HPC usage, many projects (spark, dask, blazingsql, etc) also > > adopt UCX to accelerate network transmission. [2][3] > > > > I see a recent discussion about decoupling Flight from gRPC. Looks this > > is also what we should do first to adapt UCX to Flight. > > Is it actually necessary? Or is it possible to transmit HTTP/2 over UCX? > > (I'm not fond of gRPC, but decoupling Flight *and* building support for > another transport layer will be quite a bit of work) > > Regards > > Antoine. >
