Atul Dambalkar updated ARROW-1780:

Comments from Uwe Korn - 

My main plan was to make JDBC drivers accessible very fast from Python / Pandas 
programs. Currently, you either have the option for most DBs to either use 
ODBC/python-native drivers that are quite often awful or use JDBC ones but have 
a high cost of serialization between the JVM and the Python objects. By using 
Arrow, we should be able to use the good JDBC drivers from Python without the 
normal serialization overhead.

We’re looking at SQL-engines that work on distributed filesystem in general at 
the moment (Apache Drill and Presto are two best candidates at the moment) and 
the common pattern is that they have good JDBC drivers but the other connectors 
are not so well maintained or really slow. Currently, Presto is the one of 
biggest interest for me.

For me it seems that having a JDBC<->Arrow adapter already yields a significant 
performance impact in comparison to the current situation. And it will also 
give the speedup independent of the underlying DB.


> JDBC Adapter for Apache Arrow
> -----------------------------
>                 Key: ARROW-1780
>                 URL: https://issues.apache.org/jira/browse/ARROW-1780
>             Project: Apache Arrow
>          Issue Type: New Feature
>            Reporter: Atul Dambalkar
>            Priority: Major
> At a high level the JDBC Adapter will allow upstream apps to query RDBMS data 
> over JDBC and get the JDBC objects converted to Arrow objects/structures. The 
> upstream utility can then work with Arrow objects/structures with usual 
> performance benefits. The utility will be very much similar to C++ 
> implementation of "Convert a vector of row-wise data into an Arrow table" as 
> described here - 
> https://arrow.apache.org/docs/cpp/md_tutorials_row_wise_conversion.html
> The utility will read data from RDBMS and covert the data into Arrow 
> objects/structures. So from that perspective this will Read data from RDBMS, 
> If the utility can push Arrow objects to RDBMS is something need to be 
> discussed and will be out of scope for this utility for now. 

This message was sent by Atlassian JIRA

Reply via email to