emkornfield commented on pull request #12571: URL: https://github.com/apache/arrow/pull/12571#issuecomment-1072015812
> The idea is we may want to preview sample data, or if the datasets is small we can get the data in one request. Do you think we can use DoAction() API to accomplish this goal? If yes, we can do that in FlightSQL protocol No, I don't think so. My understanding is DoAction implies the results will be smallish. One could potentially use DoAction to return a FlightInfo + optional inlined data. However, for queries FlightSQL mandates using GetFlightInfo, again we could change this to also pass it to also DoAction, but this seems to side step the intended use of GetFlightInfo(). This also runs into the same problem. If the data turns out to be large, then another RPC needed to made to start retrieving it (although the latency hit for larger data sets is probably generally less important). The crux of the problem is clients don't necessarily know the size of the results when calling GetFlightInfo. I don't think this is specific to SQL hence the proposal for putting it in Flight proper. One could argue that at some point we might want having a streaming RPC (a combination of DoAction and GetFlightInfo) that just returns tickets for absolute minimal latency but that is a much larger change and makes sense in less scenarios. If you where thinking something different here let me know. > The idea is we may want to preview sample data, IIUC this is about a better UX where you could show some data to the user immediately while the rest downloads in the background. This was incorporating a suggestion from the mailing list, my larger concern is complete results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
