Hi Julian, Thanks for your response. If I understand correctly (looking at other adapters), Calcite-Arrow adapter would provide SQL front end for in-memory Arrow data objects/structures. So from that perspective, are you suggesting building the Calcite-Arrow adapter?
In this case, what we are saying is to provide a mechanism for upstream apps to be able to get/create Arrow objects/structures from a relational database. This would also mean converting row like data from a SQL Database to columnar Arrow data structures. The utility may be, can make use of JDBC's MetaData features to figure out the underlying DB schema and define Arrow columnar schema. Also underlying database in this case would be any relational DB and hence would be persisted to the disk, but the Arrow objects being in-memory can be ephemeral. Please correct me if I am missing anything. -Atul -----Original Message----- From: Julian Hyde [mailto:jhyde.apa...@gmail.com] Sent: Monday, October 30, 2017 7:50 PM To: dev@arrow.apache.org Subject: Re: JDBC Adapter for Apache-Arrow How about writing an Arrow adapter for Calcite? I think it amounts to the same thing - you would inherit Calcite’s SQL parser and Avatica JDBC stack. Would this database be ephemeral (i.e. would the data go away when you close the connection)? If not, how would you know where to load the data from? Julian > On Oct 30, 2017, at 6:17 PM, Atul Dambalkar <atul.dambal...@xoriant.com> > wrote: > > Hi all, > > I wanted to open up a conversation here regarding developing a Java-based > JDBC Adapter for Apache Arrow. I have had a preliminary discussion with Wes > McKinney and Siddharth Teotia on this a couple weeks earlier. > > Basically at a high level (over-simplified) this adapter/API will allow > upstream apps to query RDBMS data over JDBC and get the JDBC objects > converted to Arrow in-memory (JVM) objects/structures. The upstream utility > can then work with Arrow objects/structures with usual performance benefits. > The utility will be very much similar to C++ implementation of "Convert a > vector of row-wise data into an Arrow table" as described here - > https://arrow.apache.org/docs/cpp/md_tutorials_row_wise_conversion.html. > > How useful this adapter would be and which other Apache projects would > benefit from this? Based on the usability we can open a JIRA for this > activity and start looking into the implementation details. > > Regards, > -Atul Dambalkar > >
