I've updated the PR [1] and I believe everything is resolved. (I've fixed
ARROW-17254, and changed the Protobuf definition to work around Protobuf's
issues.) If there's no further comments, I'll start a vote in the coming days.
[1]: https://github.com/apache/arrow/pull/13492
Thanks,
David
On
I've added implementations for Java and C++ to the draft [1], including
integration tests, after addressing comments on the proposal itself (thanks all
for the comments).
One thing is, I might suggest punting on CancelQuery for now, or changing how
it's implemented, since embedding a message
I quickly drafted these out (sans implementation so far):
https://github.com/apache/arrow/pull/13492
On Thu, Jun 30, 2022, at 21:20, David Li wrote:
> Ah - somehow I didn't think of that. Yes, we should just implement it
> in the same way prepared statements are already implemented.
>
> On Thu,
Ah - somehow I didn't think of that. Yes, we should just implement it in the
same way prepared statements are already implemented.
On Thu, Jun 30, 2022, at 19:42, Micah Kornfield wrote:
>>
>> It would also then be good to make explicit the statefulness of
>> connections in Flight SQL. While that
>
> It would also then be good to make explicit the statefulness of
> connections in Flight SQL. While that is sort of an obvious constraint, it
> is at odds with how gRPC is usually used (especially in the presence of
> load balancing).
I'm not sure I understand where the statefulness
This is a bit of a tangent from the original discussion about
Substrait integration.
Flight SQL would definitely benefit from transaction RPC commands for
building bridge drivers. I'm also wondering if there should be an RPC call
to cancel a running query, as opposed to just having the client
Reviving this discussion: would people be interested in seeing a sketched-out
CommandSubstraitQuery et. al.?
Additionally, while working on ADBC, I realized: does Flight SQL need explicit
Commit/Rollback commands? This would presumably be necessary if we want to
build ODBC/JDBC drivers on top,
Got it, thank you David!
I started prototyping the implementation last night, hopefully I will make
some good progress and have something basic functioning soon.
RE: The metadata thing -- I think both Calcite and Teiid have solid
interfaces for defining what capabilities a datasource has.
Yes, we should, where possible, avoid any one of metadata. This is where
other standards fail in that applications must be custom built for each
data source, if we standardize the metadata then applications can at least
be built to adapt.
On Sat., Mar. 5, 2022, 6:54 p.m. David Li, wrote:
> Yes,
Yes, GetSqlInfo reserves a range of metadata IDs for Flight SQL's use, so the
application can use others for its own purposes. That said if they seem
commonly applicable maybe we should try to standardize them.
I think what you are doing should be reasonable. You may not need _all_ of the
To touch on the question about supported features -- is it possible to
advertise arbitrary/custom "capabilites" in GetSqlInfo?
Say that you want to represent some set of behaviors that FlightSQL
services can support.
Stuff like "Supports grouping by multiple distinct aggregates", "Supports
We could also add say CommandSubstraitQuery as a distinct message, and older
servers would just reject it as an unknown request type.
-David
On Fri, Mar 4, 2022, at 17:01, Micah Kornfield wrote:
>>
>> 1. How does a server report that it supports each command type? Initial
>> thought is a
>
> 1. How does a server report that it supports each command type? Initial
> thought is a property in GetSqlInfo.
This sounds reasonable.
> What happens to client code written prior to changing the command type
> to be a oneOf field? Same for servers.
It is transparent from older clients
It sounds like an interesting and useful project to use Subtstrait as an
alternative to SQL strings.
Important aspects to spec out are:
1. How does a server report that it supports each command type? Initial
thought is a property in GetSqlInfo.
2. What happens to client code written prior to
@James Duong
You are absolutely right, I realized this and confirmed whether this
would be possible with Jacques to double-check.
It would amount to what I might call "dollar-store Substrait." It's not
elegant or a good solution, but definitely presents a good duct-tape hack
and is a crafty
Gavin, thanks for sharing. I'm not so sure you'll find an alternative to
Substrait, at least one that isn't even more nascent or one that's very tied to
a particular language, so perhaps it might be better to get involved in
Substrait and see if it suits your needs? Convincing a team to try
James, I agree that you could use JSON but that feels a bit hacky (mis-use
of the paradigm). Instead, I'd really like to do something like David is
suggesting: support Substrait as an alternative to a SQL string.
Something like this:
In the same way that you could write an ODBC driver that takes in text
that's not SQL, you could write a Flight SQL server that takes in text
that's JSON.
Flight SQL doesn't parse the query, so you could create commands that are
just JSON text.
Is that the only bit you need, Gavin?
On Thu, Mar
I am enthusiastic about Substrait and have followed it's progress eagerly =D
When I presented it as a tentative option, there were reservations because
of the project/spec being young and the functionality still being
fleshed out.
I think if I were having this conversation in say, 8-16 months, it
You probably want Substrait: https://substrait.io/
Which is being worked on by several people, including Arrow community members.
It might be interesting to generalize Flight SQL to include support for
Substrait. I'm curious what your application, if you're able to share more.
-David
On Thu,
Hiya,
I am drafting a proposal for a way to enable services to express data
compute operations to each other.
However I think it'll be difficult to get buy-in if the only representation
for queries is as SQL strings.
Is there any kind of lower-level API that can be used to express operations?
21 matches
Mail list logo
