Hey James - thanks for the reminder, sorry for dropping the ball here. I think you can submit a PR to arrow-site: https://github.com/apache/arrow-site and we can iterate/review it from there.
Thanks, David On Tue, Jan 4, 2022, at 12:25, James Duong wrote: > If I recall correctly, there was interest in providing a blog post to > accompany the announcement of Flight SQL's release. Where should this be > submitted and what other steps would be needed for this? > > On Sat, Dec 25, 2021 at 8:07 PM Jacques Nadeau <jacq...@apache.org> wrote: > > > That's great news. Congrats and thanks to the team who worked on it. This > > is a great addition to Arrow! > > > > On Thu, Dec 23, 2021, 11:26 AM David Li <lidav...@apache.org> wrote: > > > > > The integration tests and existing PRs were merged into a separate > > branch. > > > We also merged in a few build fixes during final review. Just in time for > > > the holidays, Flight SQL has now been merged into the main branch, thanks > > > again to everyone who participated! > > > > > > I will begin reviewing the additional proposals next. > > > > > > -David > > > > > > On Fri, Dec 17, 2021, at 17:07, James Duong wrote: > > > > Yes, additional metadata would just be using the Field metadata map. > > The > > > > protocol is the same, we have just pre-defined keys for some fields > > that > > > > would be used for JDBC column attributes. > > > > > > > > Our preference would be that we get the currently approved protocol > > > merged > > > > into master first (after completing the integration tests) and then > > have > > > a > > > > separate vote on the TypeInfo changes. There's significant value in > > > adding > > > > Flight-SQL already and it'd be great to make that available. It's > > natural > > > > that there will be an ongoing need to add extensions to the protocol as > > > it > > > > gets used in more scenarios. Now that we have a solid foundation, we > > can > > > > examine further changes on a case-by-case basis. > > > > > > > > On Thu, Dec 16, 2021 at 2:42 PM David Li <lidav...@apache.org> wrote: > > > > > > > > > Strictly speaking we should have a vote since it is updating the > > format > > > > > definition files we already voted on. > > > > > > > > > > I am curious about what exactly you mean by additional column > > metadata, > > > > > but if it's just going to be encoded into the key-value metadata > > then I > > > > > don't see a problem there. (As in: it sounds like it fits in the > > Field > > > > > class given it's encoded in the Field metadata!) > > > > > > > > > > -David > > > > > > > > > > On Thu, Dec 16, 2021, at 16:14, James Duong wrote: > > > > > > Hi David, > > > > > > > > > > > > While working on the JDBC driver on top of Flight SQL and on > > > integration > > > > > > tests, we identified a couple of enhancements that were needed. > > > > > > 1. The ability to report data type information, as done in this PR: > > > > > > https://github.com/apache/arrow/pull/11982. This PR adds another > > RPC > > > > > > request for this information. > > > > > > 2. Additional column metadata that's outside of the Schema/Field > > > classes > > > > > in > > > > > > Arrow (PR pending) when returning Arrow schemas. The planned PR > > uses > > > the > > > > > > Arrow Field's MetadataMap to encode extra metadata rather than > > > altering > > > > > any > > > > > > protobuf definitions. > > > > > > > > > > > > Should these additional changes go in together with the rest of > > > > > Flight-SQL, > > > > > > or be approved separately? > > > > > > > > > > > > On Thu, Dec 16, 2021 at 7:54 AM Kyle Porter < > > ky...@bitquilltech.com > > > > > .invalid> > > > > > > wrote: > > > > > > > > > > > > > Thanks All - we'll look to get the tests merged into this branch > > > so we > > > > > can > > > > > > > close ASAP. > > > > > > > > > > > > > > *Kyle Porter* > > > > > > > CEO > > > > > > > Bit Quill Technologies Inc. > > > > > > > Office: +1.778.331.3355 | Direct: +1.604.441.7318 | > > > > > ky...@bitquilltech.com > > > > > > > https://www.bitquill.com > > > > > > > > > > > > > > This email message is for the sole use of the intended > > recipient(s) > > > > > and may > > > > > > > contain confidential and privileged information. Any > > unauthorized > > > > > review, > > > > > > > use, disclosure, or distribution is prohibited. If you are not > > the > > > > > > > intended recipient, please contact the sender by reply email and > > > > > destroy > > > > > > > all copies of the original message. Thank you. > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 15, 2021 at 9:11 AM David Li <lidav...@apache.org> > > > wrote: > > > > > > > > > > > > > > > My vote: +1 > > > > > > > > > > > > > > > > The vote passes with three +1 (binding) votes, one +1 (non > > > binding) > > > > > vote, > > > > > > > > and one -0.5 (binding) vote. > > > > > > > > > > > > > > > > However, we will first merge into a separate branch and > > implement > > > > > > > > integration tests before merging into the main branch. JIRA for > > > > > > > integration > > > > > > > > tests: https://issues.apache.org/jira/browse/ARROW-15112 > > > > > > > > > > > > > > > > @Kyle I've created the branch flight-sql[1], would you prefer I > > > > > merge in > > > > > > > > your existing PRs, or would you prefer to create new PRs > > against > > > that > > > > > > > > branch (given you've already started on things)? > > > > > > > > > > > > > > > > On a side note - do we document the requirements for proposed > > > > > additions > > > > > > > > somewhere? (multiple implementations, integration tests) It > > > would be > > > > > nice > > > > > > > > to have it on hand for reference. > > > > > > > > > > > > > > > > [1]: https://github.com/apache/arrow/tree/flight-sql > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > On Mon, Dec 13, 2021, at 11:25, Kyle Porter wrote: > > > > > > > > > Thanks David, > > > > > > > > > > > > > > > > > > Yes, the team is actually already looking at adding the cross > > > > > language > > > > > > > > > tests apologies for not communicating that earlier > > > > > > > > > > > > > > > > > > On Mon., Dec. 13, 2021, 12:18 p.m. David Li, < > > > lidav...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Are any other PMC members able to look at this? > > > > > > > > > > > > > > > > > > > > > > > OK by me. We could also create a branch to merge the > > > PRs > > > > > add > > > > > > > the > > > > > > > > > > > > > integration tests, and then merge all at once. > > > > > > > > > > > > > > > > > > > > Kyle, is this an ok solution? Would you & your team be able > > > to > > > > > get > > > > > > > > > > integration tests done reasonably soon? > > > > > > > > > > > > > > > > > > > > There's some setup for Flight integration tests already: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/arrow/blob/11be9c542b9699b7eb4ae1656775c9b30639e415/dev/archery/archery/integration/runner.py#L375-L385 > > > > > > > > > > > > > > > > > > > > So what would be needed are: > > > > > > > > > > > > > > > > > > > > 1. Enable Flight SQL for the integration test container > > > > > > > > > > 2. Link the integration test client/server to Flight SQL > > > > > > > > > > 3. Add one or more test scenarios in the integration test > > > > > runner, and > > > > > > > > in > > > > > > > > > > the integration test client/server > > > > > > > > > > > > > > > > > > > > It might be acceptable to just hardcode expected > > > > > requests/responses > > > > > > > > > > instead of integrating SQLite/Derby (as was done for the > > > > > individual > > > > > > > > > > language tests) since the focus should be on just the > > > protocol > > > > > and > > > > > > > not > > > > > > > > > > particular implementations. > > > > > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > On Sun, Dec 12, 2021, at 16:21, Wes McKinney wrote: > > > > > > > > > > > +1. Agree re: adding integration tests as soon as > > practical > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 9, 2021 at 5:21 AM Ravindra Pindikura < > > > > > > > > ravin...@dremio.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 8, 2021 at 11:42 PM Micah Kornfield < > > > > > > > > emkornfi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given that the C++ and Java components are in > > > separate > > > > > PRs, > > > > > > > > would > > > > > > > > > > it be > > > > > > > > > > > > > > acceptable to add after the initial merge? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > OK by me. We could also create a branch to merge the > > > PRs > > > > > add > > > > > > > the > > > > > > > > > > > > > integration tests, and then merge all at once. > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 8, 2021 at 10:07 AM Kyle Porter < > > > > > > > > ky...@bitquilltech.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Given that the C++ and Java components are in > > > separate > > > > > PRs, > > > > > > > > would > > > > > > > > > > it be > > > > > > > > > > > > > > acceptable to add after the initial merge? > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kyle Porter* > > > > > > > > > > > > > > CEO > > > > > > > > > > > > > > Bit Quill Technologies Inc. > > > > > > > > > > > > > > Office: +1.778.331.3355 | Direct: +1.604.441.7318 | > > > > > > > > > > > > > ky...@bitquilltech.com > > > > > > > > > > > > > > https://www.bitquill.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > This email message is for the sole use of the > > > intended > > > > > > > > > > recipient(s) and > > > > > > > > > > > > > > may contain confidential and privileged > > > information. Any > > > > > > > > > > unauthorized > > > > > > > > > > > > > > review, use, disclosure, or distribution is > > > prohibited. > > > > > If > > > > > > > you > > > > > > > > > > are not > > > > > > > > > > > > > the > > > > > > > > > > > > > > intended recipient, please contact the sender by > > > reply > > > > > email > > > > > > > > and > > > > > > > > > > destroy > > > > > > > > > > > > > > all copies of the original message. Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 8, 2021 at 2:03 PM Micah Kornfield < > > > > > > > > > > emkornfi...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > There is not an integration test. Do we want to > > > > > require > > > > > > > > this? > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> It would be nice, I'm -0.5 vote without one. So > > if > > > > > enough > > > > > > > > PMC > > > > > > > > > > members > > > > > > > > > > > > > >> want to forgo the integration test the vote can > > > still > > > > > pass. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Is cross language testing something that's > > usually > > > > > done? > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Yes. One of the value propositions of Arrow is > > the > > > > > > > > cross-language > > > > > > > > > > > > > >> support. The community agreed to specification > > > changes > > > > > > > (and I > > > > > > > > > > assumed > > > > > > > > > > > > > >> this > > > > > > > > > > > > > >> covers new specifications) need to have reference > > > > > > > > implementations > > > > > > > > > > in > > > > > > > > > > > > > >> C++/Java with integration testing between the two. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> On Wed, Dec 8, 2021 at 5:21 AM Kyle Porter < > > > > > > > > > > ky...@bitquilltech.com > > > > > > > > > > > > > >> .invalid> > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > The team initially developed the C++ client > > > against > > > > > the > > > > > > > Java > > > > > > > > > > server, > > > > > > > > > > > > > and > > > > > > > > > > > > > >> > have done some cross language testing. It wasn't > > > > > > > exhaustive > > > > > > > > or > > > > > > > > > > > > > >> methodical > > > > > > > > > > > > > >> > in nature, however. Is cross language testing > > > > > something > > > > > > > > that's > > > > > > > > > > usually > > > > > > > > > > > > > >> > done? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Wed., Dec. 8, 2021, 9:18 a.m. David Li, < > > > > > > > > lidav...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > There is not an integration test. Do we want > > to > > > > > require > > > > > > > > this? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Also CC @Kyle, in case your team has done such > > > > > testing. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > It looks like Flight itself did not have a > > test > > > for > > > > > a > > > > > > > few > > > > > > > > > > versions > > > > > > > > > > > > > >> after > > > > > > > > > > > > > >> > > it was initially implemented. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > -David > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > On Tue, Dec 7, 2021, at 23:19, Micah Kornfield > > > > > wrote: > > > > > > > > > > > > > >> > > > Is there an integration test between the two > > > > > > > languages? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > On Tue, Dec 7, 2021 at 1:35 PM David Li < > > > > > > > > > > lidav...@apache.org> > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > Hello, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Kyle Porter, Rafael Telles, Ryan > > Nicholson, > > > et. > > > > > al. > > > > > > > > have > > > > > > > > > > > > > proposed > > > > > > > > > > > > > >> > > adding > > > > > > > > > > > > > >> > > > > Arrow Flight SQL, an experimental protocol > > > for > > > > > > > > > > interacting with > > > > > > > > > > > > > >> SQL > > > > > > > > > > > > > >> > > > > databases over Arrow Flight [1], as > > > explained > > > > > in a > > > > > > > > > > previous ML > > > > > > > > > > > > > >> > > discussion > > > > > > > > > > > > > >> > > > > [2] and in a design document [3]. The > > > purpose of > > > > > > > > Flight > > > > > > > > > > SQL is > > > > > > > > > > > > > to > > > > > > > > > > > > > >> > allow > > > > > > > > > > > > > >> > > > > clients and SQL database servers to > > > communicate > > > > > > > > (execute > > > > > > > > > > > > > queries, > > > > > > > > > > > > > >> > list > > > > > > > > > > > > > >> > > > > tables, create prepared statements, etc.) > > > using > > > > > > > Arrow > > > > > > > > and > > > > > > > > > > Arrow > > > > > > > > > > > > > >> > > Flight, by > > > > > > > > > > > > > >> > > > > defining how to use Flight RPC methods, as > > > well > > > > > as > > > > > > > > message > > > > > > > > > > > > > >> payloads > > > > > > > > > > > > > >> > to > > > > > > > > > > > > > >> > > use > > > > > > > > > > > > > >> > > > > with those methods. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > The new protocol definitions can be found > > at > > > > > [4]. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > They have provided pull requests > > > implementing > > > > > the > > > > > > > > server > > > > > > > > > > and > > > > > > > > > > > > > >> client > > > > > > > > > > > > > >> > > > > protocol in C++ [5] and Java [6] which can > > > be > > > > > merged > > > > > > > > > > after this > > > > > > > > > > > > > >> > > addition is > > > > > > > > > > > > > >> > > > > approved. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Please vote whether to accept this > > > addition. The > > > > > > > vote > > > > > > > > > > will be > > > > > > > > > > > > > open > > > > > > > > > > > > > >> > for > > > > > > > > > > > > > >> > > at > > > > > > > > > > > > > >> > > > > least 72 hours. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > [1]: > > > > > > > https://arrow.apache.org/docs/format/Flight.html > > > > > > > > > > > > > >> > > > > [2]: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > https://lists.apache.org/thread/s08b20ty756qq10zybd9qr0mm4jhmz93 > > > > > > > > > > > > > >> > > > > [3]: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing > > > > > > > > > > > > > >> > > > > Note that the protocol definitions in the > > > design > > > > > > > > document > > > > > > > > > > are > > > > > > > > > > > > > out > > > > > > > > > > > > > >> of > > > > > > > > > > > > > >> > > date; > > > > > > > > > > > > > >> > > > > the canonical reference is in the pull > > > requests. > > > > > > > > > > > > > >> > > > > [4]: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/arrow/blob/72ce72ba855909052f7dfb898105b419697157c8/format/FlightSql.proto > > > > > > > > > > > > > >> > > > > [5]: > > > https://github.com/apache/arrow/pull/11507 > > > > > > > > > > > > > >> > > > > [6]: > > > https://github.com/apache/arrow/pull/10906 > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > > > > > >> > > > > David > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Thanks and regards, > > > > > > > > > > > > Ravindra. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *James Duong* > > > > > > Lead Software Developer > > > > > > Bit Quill Technologies Inc. > > > > > > Direct: +1.604.562.6082 | jam...@bitquilltech.com > > > > > > https://www.bitquilltech.com > > > > > > > > > > > > This email message is for the sole use of the intended recipient(s) > > > and > > > > > may > > > > > > contain confidential and privileged information. Any unauthorized > > > > > review, > > > > > > use, disclosure, or distribution is prohibited. If you are not the > > > > > > intended recipient, please contact the sender by reply email and > > > destroy > > > > > > all copies of the original message. Thank you. > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *James Duong* > > > > Lead Software Developer > > > > Bit Quill Technologies Inc. > > > > Direct: +1.604.562.6082 | jam...@bitquilltech.com > > > > https://www.bitquilltech.com > > > > > > > > This email message is for the sole use of the intended recipient(s) and > > > may > > > > contain confidential and privileged information. Any unauthorized > > > review, > > > > use, disclosure, or distribution is prohibited. If you are not the > > > > intended recipient, please contact the sender by reply email and > > destroy > > > > all copies of the original message. Thank you. > > > > > > > > > > > > -- > > *James Duong* > Lead Software Developer > Bit Quill Technologies Inc. > Direct: +1.604.562.6082 | jam...@bitquilltech.com > https://www.bitquilltech.com > > This email message is for the sole use of the intended recipient(s) and may > contain confidential and privileged information. Any unauthorized review, > use, disclosure, or distribution is prohibited. If you are not the > intended recipient, please contact the sender by reply email and destroy > all copies of the original message. Thank you. >
