GitHub user WhisperN created a discussion: Arrow IPC via gRPC-Web into Typescript frontend.
Hello guys, I am currently experimenting with IPC streams and also gRPC-Web. My thought was that gRPC on h2 would reduce overhead and the parsing in the frontend for a much faster system. I started to research about ways to load data from an in memory database (in my case duckdb) onto another system. There was Apache Arrow Flight, it sounded ideal for sharing columnized data via streams to dump directly into the clients memory. So I tried to get a typescript frontend running that connects to the server and with a doGet it dumps some dataset into the clients memory. This way, at least I thought, I could build a frontend that utilizes client side processing instead of server side. Now my problem is this: The data gets prepared and shoved into the client extremely fast (~1.6s for a 134 MB dataset). If I also want to do something with the data I would have to parse it or have some sort of database in the frontend that just collects this payload and allows me to project it onto some kind of data visualization. Does anyone have an idea on how to handle the data in my typescript web client? Or alternative approaches for a super fast data vis system? If I hit a dead end here next thing would be to send a .parquet file into a Typescript Duckdb instance and then process it in the frontend. If you are interested in this project or you want to read the code. I have it all on my repo in the directories: Backend: https://github.com/WhisperN/Go-Flight-Server.git Frontend: https://github.com/WhisperN/gRPC-Web-Frontend.git I am looking forward to hear from you! GitHub link: https://github.com/apache/arrow/discussions/47603 ---- This is an automatically sent email for user@arrow.apache.org. To unsubscribe, please send an email to: user-unsubscr...@arrow.apache.org
