[DISCUSS][Format] Starting to do some concrete work on the new "StringView" columnar data type

2022-07-30 Thread Wes McKinney
hi folks, I'm interested to start doing some work to implement the "StringView" memory layout that we previously discussed late last year [1] with supporting document [2]. Since there's quite a few details to work out, my objective would be to do the work in a feature branch focused on a few

Re: [DISCUSS][Format] Dynamic data encodings in the IPC format and C ABI

2022-07-30 Thread Wes McKinney
On Sat, Jul 30, 2022 at 7:10 AM Andrew Lamb wrote: > > > but the decision must be declared up front in the schema. > > I am probably missing something obvious but given the description of the > IPC format [1] perhaps we could support "on the fly changes" by sending > additional `SCHEMA` messages

Re: [ARROW-17255] Logical JSON type in Arrow

2022-07-30 Thread Antoine Pitrou
Le 30/07/2022 à 01:02, Wes McKinney a écrit : I think either path: * Canonical extension type * First-class type in the Type union in Flatbuffers would be OK. The canonical extension type option is the preferable path here, I think, because it allows Arrow implementations without any special

Re: [ARROW-17255] Logical JSON type in Arrow

2022-07-30 Thread David Li
I filed ARROW-17268 [1] for the JSON parse/extract/serialize kernels. (Though probably this would get broken up across multiple tickets.) [1]: https://issues.apache.org/jira/browse/ARROW-17268 -David On Sat, Jul 30, 2022, at 11:06, Neal Richardson wrote: > Sounds good to me too. +1 on the

Re: [ARROW-17255] Logical JSON type in Arrow

2022-07-30 Thread Neal Richardson
Sounds good to me too. +1 on the canonical extension type option; maybe it should end up as a first-class type, but I'd like to see us try it without first and see what that tells us about the path for having an extension type get promoted to being a first-class type. This is something that has

Re: [Flight][Java][JDBC] IP clearance of Flight JDBC Driver

2022-07-30 Thread David Li
All credit goes to the contributors of course - and thanks to the team at Bit Quill/Dremio, especially James, for coordinating everything on their end. I have seen interest as well - hopefully this sparks more interest in Arrow, even if JDBC/ODBC aren't ideal client APIs for Arrow. On Sat, Jul

Re: [DISCUSS][Format] Dynamic data encodings in the IPC format and C ABI

2022-07-30 Thread Andrew Lamb
> but the decision must be declared up front in the schema. I am probably missing something obvious but given the description of the IPC format [1] perhaps we could support "on the fly changes" by sending additional `SCHEMA` messages in the stream, rather than just a single message at the

Re: [Flight][Java][JDBC] IP clearance of Flight JDBC Driver

2022-07-30 Thread Andrew Lamb
Thank you for working to make this happen David -- I don't know how much feedback you get from the community about JDBC/ODBC drivers for Arrow Flight SQL but I am excited, and I have encountered many other people who are similarly excited. I predict these technologies will be widely used and