xinyiZzz commented on issue #35777:
URL: https://github.com/apache/arrow/issues/35777#issuecomment-1719061018
> FWIW, ADBC exposes a direct `fetch_df` method so you can avoid having to
fetch and then convert. (Under the hood it will use PyArrow to convert from
Arrow to Pandas, and PyArrow has heavily optimized this path over the years.)
Cool! , this avoids the cost of `fetchall` and `to_pandas`, in my tests, the
performance increased to 5 times
`select URL from clickbench.hits where URL!='' limit 10000000`
- flightsql, 19.0s
- execute SQL cost, 3.6s
- fetchall cost, 13.1s
- to_pandas cost, 2.2s
- flightsql-fetch_df, 3.9s
- execute SQL cost, 3.6s
- fetch_df cost, 0.3s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
