jayzhan211 commented on issue #7353: URL: https://github.com/apache/arrow-datafusion/issues/7353#issuecomment-1741759821
> I think the idea would be to use arrow-ipc arrow-ipc converts data to `EncodedData` (Vec<u8>) and writes them to a file, it is quite different from the current proto/ implementation, which just turns them into bytes. If I understand correctly, prost::Message is the equivalent concept of arrow-ipc's `EncodedData`. I think there are two approaches If we remain from_proto / to_proto for ScalarValue::List <ArrayRef>, we can get the ipc-message from arrow-ipc and store those Vec<u8> in prost::Message. Not sure whether we can do this but since they are all Vec<u8>, so probably doable. The other way is we introduce from_ipc / to_ipc for ScalarValue::List and no prost for ScalarValue::List, and use `write/read` to store `EncodedData` into a given filename, but I am not sure whether mixing prost and arrow-ipc makes sense while building logical plan and physical plan. @tustvold How do you think about these two approaches? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
