Re: [DISCUSS][Format][Flight] Ordered data support

2023-05-01 Thread Sutou Kouhei
; >> > defined", we can't implement a general purpose Flight based >> >> > client library (Flight SQL based client library, Flight SQL >> >> > based ADBC driver and so on). The client library will have >> >> > the following code: >> >> &

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-28 Thread Andrew Lamb
d) > >> > end > >> > else > >> > # Other DBs doesn't return ordered result. > >> > # So, we read data in parallel for performance. > >> > threads = endpoints.collect do |endpoint| > >> > Thre

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread David Li
; > endpoints.each do |endpoint| >> > yield(endpoints.read) >> > end >> > else >> > # Other DBs doesn't return ordered result. >> > # So, we read data in parallel for performance. >> > threads = endpoints.colle

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread Weston Pace
; > end > > end > > > > The client library needs to add 'or server_type == "DB2"' to > > 'if server_type == "DB1"' when DB2 also adds support for > > ordered result. If DB2 2.0 or later is only ordered result > > ready, the cli

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread David Li
t; end > > The client library needs to add 'or server_type == "DB2"' to > 'if server_type == "DB1"' when DB2 also adds support for > ordered result. If DB2 2.0 or later is only ordered result > ready, the client library needs more condition 'or > (server_type == "DB2

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread Sutou Kouhei
"DB1"' when DB2 also adds support for ordered result. If DB2 2.0 or later is only ordered result ready, the client library needs more condition 'or (server_type == "DB2" and server_version > 2.0)'. So I think that the "ordered" flag is useful. Thanks, -- kou In &q

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread Weston Pace
So this would be a case where multiple "endpoints" are acting as a single "stream of batches"? Or am I misunderstanding? What're some scenarios where that would be done? When would it be preferred for the client to merge the endpoints instead of the client's user? On Thu, Apr 27, 2023, 3:22 PM

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread David Li
The server would have to report these as multiple endpoints in all your examples. (There's nothing saying a particular location can only appear once, or that "Endpoint 2" has to come after "Endpoint 1" for the DESC example.) The flag tells the client if it can fetch data in parallel without

Re: [DISCUSS][Format][Flight] Ordered data support

2023-04-27 Thread Andrew Lamb
I wonder if we have considered simply removing the statement "There is no ordering defined on endpoints. Hence, if the returned data has an ordering, it should be returned in a single endpoint." and replacing it with something that says "the relative ordering of data from different endpoints is