Re: [DISCUSS][Format] Starting to do some concrete work on the new "StringView" columnar data type

Hi team! 2cents(maybe less): if I get the idea right, StringView data type might be very handy/optimal for cases where users already have string data in some other formats available (e.g. std::unordered_map, flat buffer structures etc.) Off which record batches are created and shipped to the

Re: June 23 virtual conference to highlight work in the Arrow ecosystem

Great news! Actually I wonder if it would be also possible to organize some non-virtual events later in the summer? On Fri, May 13, 2022, 12:02 PM Andrew Lamb wrote: > > If folks would find it interesting, I could do a short talk on a > use-case for FlightSQL (and Substrait) > > I would

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

Hi guys, 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this feature starting from the IPC(I moved initial write up steps to the bottom of the doc). Afterwards David suggested focusing on Flight and that's how we ended up with the protobufs change in the proposal. This being

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

this wouldn't be sufficient (either here or on the doc). > > Thanks, > -Micah > > On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan wrote: > > > Hi guys, > > > > Thanks for sharing your insights/concerns! I also left some comments > based > > on the discus

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

plication imperatively writing batches to > the > > client. (This is different than how Flight is implemented in Java.) You > > would normally not implement FlightDataStream - you would return a > > RecordBatchStream. > > >> > > >> DoGet could not have FlightMessage

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

not have FlightMessageWriter as a return type as that wouldn't > make sense, but it could accept an instance of that as a parameter instead, > much like DoExchange. That would be a breaking change. > > Best, > David > > On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote: >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

. Is there any reason why the DoGet can't have FlightMessageWriter as a return type? Cheers, Gosh On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan wrote: > Thanks David! > > I also responded/added more suggestions/questions to the doc. I think it > makes sense to have two sections

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

good to address why the > proposed workaround there (union-of-structs) is insufficient for the use > cases here (and in FlightSQL). > > -David > > On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote: > > Ah sorry, comments should work now. > > > > Cheers, > >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

Ah sorry, comments should work now. Cheers, Gosh On Mon., 21 Jun. 2021, 14:18 David Li, wrote: > Thanks! Will give it a look. > > Would you mind opening it up for comments? > > -David > > On 2021/06/21 11:56:24, Gosh Arzumanyan wrote: > > Hi folks, > >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

Hi folks, Started putting some thoughts together here: https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing Any feedback is welcome! Cheers, Gosh

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

; > On Fri, Apr 16, 2021, at 10:59, Gosh Arzumanyan wrote: > > Hi guys! > > > > Thanks for the feedback/info. > > Let me try to put a proposal together. Though I guess I'll need some > > assistance on crafting it both in terms of the structure of a proposal > >

Re: Discuss a very fast way to serialize a large in-memory Arrow IPC table to a void* buffer for sending over the network

ode path in the last > > > couple of years, so there may be some low hanging fruit to improve the > > > performance. Changing the in-memory data layout (the chunking) is one > > > of the most likely things to help. > > > > > > On Thu, Jun 10, 2021 at 2:14

Re: Discuss a very fast way to serialize a large in-memory Arrow IPC table to a void* buffer for sending over the network

Hi Jayjeet, I wonder if you really need to serialize the whole table into a single buffer as you will end up with twice the memory while you could be sending chunks as they are generated by the RecordBatchStreamWriter. Also is the buffer resized beforehand? I'd suspect there might be relocations

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

ast, and it was suggested to use > > Arrow's Union type which handles this directly. A Union of Struct > > types essentially lets you have multiple distinct schemas all encoded > > in the same overall table, with explicit information about which > > schema is currently in use. But

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

> I am curious though, what is your use case here? > > Best, > David > > On 2021/04/12 10:49:00, Gosh Arzumanyan wrote: > > Hi guys, hope you are well! > > > > Judging from the Flight API > > < > https://github.com/apache/arrow/blob/5b082

[INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

Hi guys, hope you are well! Judging from the Flight API and from the documentation/examples out there, it seems like data schema is supposed to be fixed per stream in