That's along the lines of what I was asking. What I'm hoping for, though, is a concrete proposal. As you say, we could be doing a bunch of improvements, but before we start talking about changing the ODBC driver, I would recommend we have at least one incremental improvement that end users will benefit from.
On Wed, Aug 24, 2016 at 11:50 AM, Jacques Nadeau <[email protected]> wrote: > I'm not sure what unused code you are worried about. The backend already > implements these APIs and these changes would simply expose that interface > to the C++ consumers. I'd expect the ODBC driver to be updated to take > advantage of these APIs and thus all the code would be exercised all the > time. The main difference is: once we have the new client in user's hands, > we can iterate on performance improvements without having to upgrade the > client again in the future. Since this is an additive change, the existing > clients will continue to work without issue. > > I think we already have some jiras about improving metadata performance and > BI tool experience. I'll try to dig them up. The thought was to start as > simple as possible, which is what the backend does now: do what the client > is doing but on the server. The next clear step for metadata preparation is > to do single parsing & planning (without the limit 0) so that in known > schema cases, we can avoid double parsing/planning (especially expensive in > partition pruning and metadata cache cases). Note, I'd expect that this > next version will still maintain the same behavior if the RowType is not > known on the initial DrillTable (where we don't have the limit 0 short > circuit today) > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Tue, Aug 23, 2016 at 6:30 PM, Parth Chandra <[email protected]> wrote: > > > I would think that the Java client would be sufficient for > experimentation. > > But what I'm looking for is an actual proposal for the backend changes. > If > > we don't have one (not sure if there is a JIRA open for it), then we > should > > start that now. > > Otherwise I'm afraid we will end up with a bunch of large amount of code > > that is not used. In particular, for the C++ client, I would like to > avoid > > that. > > > > > > > > > > On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <[email protected]> > > wrote: > > > > > The clear quick win would be caching parsing/planning/pruning on the > > server > > > and reusing (if executed within time t or until statement/connection > are > > > closed). > > > > > > My thinking is to get an implementation of the client that is opaque to > > the > > > server implementation so that we can iterate on preparation without > > having > > > to constantly update the client. From there, developers can easily > > > experiment with different mechanisms to find what works best for BI > > tools. > > > > > > -- > > > Jacques Nadeau > > > CTO and Co-Founder, Dremio > > > > > > On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <[email protected]> > > > wrote: > > > > > > > I'm currently focusing on the client work, and making sure the C++ > > > client > > > > is not lagging behind the Java one. I personally haven't worked on > > > backend > > > > changes for prepared statements. > > > > > > > > Laurent > > > > > > > > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <[email protected]> > > > wrote: > > > > > > > > > Hi Laurent, > > > > > > > > > > I'll take a look at this in the next few days. > > > > > > > > > > On a related note, do you or Venki have a proposal for the > backend > > > > > changes (i.e actual implementation of prepare)? It would be a good > > idea > > > > to > > > > > start a discussion on that. > > > > > > > > > > Parth > > > > > > > > > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon < > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I just started working on adding support for prepared statements > > and > > > > > > metadata querying in the C++ Drill client. Hopefully, nobody else > > has > > > > > > started working on this (The Drill jiras don't mention any > activity > > > on > > > > > > this), but if it is not the case, let me know. > > > > > > > > > > > > My working branch is > > > > > > https://github.com/laurentgo/drill/tree/laurent/improve- > > > native-client. > > > > > > > > > > > > For now, I just have a basic interface API ( > > > > > > https://github.com/laurentgo/drill/commit/ > > > > 1f55a3e631cd97016b113b9d4bca07 > > > > > > b5e016a25e), > > > > > > and it would be nice if people knowledgeable about the C++ client > > > could > > > > > > review it and give me some feedback. I'm also adding an actual > > > initial > > > > > > implementation in the coming days. > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Laurent > > > > > > > > > > > > > > > > > > > > >
