[
https://issues.apache.org/jira/browse/ARROW-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lee updated ARROW-16629:
------------------------------
Description:
I just started testing using Arrow Flight to send results from a GraphQL server
with FlightServer() running on i.
GraphQL defines a schema for your data output which can be mapped to an Arrow
schema so I thought it would make sense to try using Arrow Flight to transport
results instead of using REST style JSON records.
Arrow Flight was 66% faster in all case, but it didn't scale as the number of
child records increased. I suspect that serializing structs or lists needs some
improvement..
Here is the discussion I opened including links to test scripts.
[https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_server.py]
Standard ASGI server. Start it up with uvicorn --host=0.0.0.0
test_asgi_server:app
[https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_arrow_flight_server.py]
New Arrow Flight server. Start it up with python test_arrow_flight_server.py
[https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_arrow_client.py]
Benchmarking script. Pass in length of lists to test and server host.
Discussion from Ariadne GraphQL package.
[https://github.com/mirumee/ariadne/discussions/867]
10 records it was 0.049 seconds faster or 80% faster
10000 records it was 0.109 seconds faster or 66% faster
10 million records it was 54 seconds faster or 66% faster.
Also here is the data structure that is sent across the wire..
pyarrow.Table
data: struct<test_lists: struct<float_list: list<item: double>, int_list:
list<item: int64>, length: int64, string_list: list<item: string>, time_spent:
double>>
child 0, test_lists: struct<float_list: list<item: double>, int_list:
list<item: int64>, length: int64, string_list: list<item: string>, time_spent:
double>
child 0, float_list: list<item: double>
child 0, item: double
child 1, int_list: list<item: int64>
child 0, item: int64
child 2, length: int64
child 3, string_list: list<item: string>
child 0, item: string
child 4, time_spent: double
data: [
– is_valid: all not null
– child 0 type: struct<float_list: list<item: double>, int_list: list<item:
int64>, length: int64, string_list: list<item: string>, time_spent: double>
– is_valid: all not null
– child 0 type: list<item: double>
[[13.500371672273381,17.747395152140353,28.973205439157457,1.361443415643098,19.029191125636135,14.62284718057391,18.44333922481529,7.906278860251386,14.402464768126993,5.826040531772251]]
– child 1 type: list<item: int64>
[[23,3,21,15,20,4,10,16,23,25]]
– child 2 type: int64
[10]
– child 3 type: list<item: string>
[["qypsupwtxy","vrxptpspyt","qpvruwsuqq","ywwpyxrvrt","wswutpxxqv","tsyypstxvv","ytprpqsxsx","wtwsxvprvu","suwtrvqvwp","wtsrwywwty"]]
– child 4 type: double
[0]]
was:
I just started testing using Arrow Flight to send results from a GraphQL server
with FlightServer() running on i.
GraphQL defines a schema for your data output which can be mapped to an Arrow
schema so I thought it would make sense to try using Arrow Flight to transport
results instead of using REST style JSON records.
Arrow Flight was 66% faster in all case, but it didn't scale as the number of
child records increased. I suspect that serializing structs or lists needs some
improvement..
Here is the discussion I opened including links to test scripts.
[https://github.com/mirumee/ariadne/discussions/867]
10 records it was 0.049 seconds faster or 80% faster
10000 records it was 0.109 seconds faster or 66% faster
10 million records it was 54 seconds faster or 66% faster.
Also here is the data structure that is sent across the wire..
pyarrow.Table
data: struct<test_lists: struct<float_list: list<item: double>, int_list:
list<item: int64>, length: int64, string_list: list<item: string>, time_spent:
double>>
child 0, test_lists: struct<float_list: list<item: double>, int_list:
list<item: int64>, length: int64, string_list: list<item: string>, time_spent:
double>
child 0, float_list: list<item: double>
child 0, item: double
child 1, int_list: list<item: int64>
child 0, item: int64
child 2, length: int64
child 3, string_list: list<item: string>
child 0, item: string
child 4, time_spent: double
data: [
-- is_valid: all not null
-- child 0 type: struct<float_list: list<item: double>, int_list: list<item:
int64>, length: int64, string_list: list<item: string>, time_spent: double>
-- is_valid: all not null
-- child 0 type: list<item: double>
[[13.500371672273381,17.747395152140353,28.973205439157457,1.361443415643098,19.029191125636135,14.62284718057391,18.44333922481529,7.906278860251386,14.402464768126993,5.826040531772251]]
-- child 1 type: list<item: int64>
[[23,3,21,15,20,4,10,16,23,25]]
-- child 2 type: int64
[10]
-- child 3 type: list<item: string>
[["qypsupwtxy","vrxptpspyt","qpvruwsuqq","ywwpyxrvrt","wswutpxxqv","tsyypstxvv","ytprpqsxsx","wtwsxvprvu","suwtrvqvwp","wtsrwywwty"]]
-- child 4 type: double
[0]]
> Apache Arrow Flight transport speed improvement for list structures
> -------------------------------------------------------------------
>
> Key: ARROW-16629
> URL: https://issues.apache.org/jira/browse/ARROW-16629
> Project: Apache Arrow
> Issue Type: Improvement
> Components: FlightRPC
> Affects Versions: 8.0.0
> Reporter: David Lee
> Priority: Minor
>
> I just started testing using Arrow Flight to send results from a GraphQL
> server with FlightServer() running on i.
> GraphQL defines a schema for your data output which can be mapped to an Arrow
> schema so I thought it would make sense to try using Arrow Flight to
> transport results instead of using REST style JSON records.
> Arrow Flight was 66% faster in all case, but it didn't scale as the number of
> child records increased. I suspect that serializing structs or lists needs
> some improvement..
> Here is the discussion I opened including links to test scripts.
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_server.py]
> Standard ASGI server. Start it up with uvicorn --host=0.0.0.0
> test_asgi_server:app
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_arrow_flight_server.py]
> New Arrow Flight server. Start it up with python test_arrow_flight_server.py
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_arrow_client.py]
> Benchmarking script. Pass in length of lists to test and server host.
>
> Discussion from Ariadne GraphQL package.
> [https://github.com/mirumee/ariadne/discussions/867]
> 10 records it was 0.049 seconds faster or 80% faster
> 10000 records it was 0.109 seconds faster or 66% faster
> 10 million records it was 54 seconds faster or 66% faster.
> Also here is the data structure that is sent across the wire..
> pyarrow.Table
> data: struct<test_lists: struct<float_list: list<item: double>, int_list:
> list<item: int64>, length: int64, string_list: list<item: string>,
> time_spent: double>>
> child 0, test_lists: struct<float_list: list<item: double>, int_list:
> list<item: int64>, length: int64, string_list: list<item: string>,
> time_spent: double>
> child 0, float_list: list<item: double>
> child 0, item: double
> child 1, int_list: list<item: int64>
> child 0, item: int64
> child 2, length: int64
> child 3, string_list: list<item: string>
> child 0, item: string
> child 4, time_spent: double
> data: [
> – is_valid: all not null
> – child 0 type: struct<float_list: list<item: double>, int_list: list<item:
> int64>, length: int64, string_list: list<item: string>, time_spent: double>
> – is_valid: all not null
> – child 0 type: list<item: double>
> [[13.500371672273381,17.747395152140353,28.973205439157457,1.361443415643098,19.029191125636135,14.62284718057391,18.44333922481529,7.906278860251386,14.402464768126993,5.826040531772251]]
> – child 1 type: list<item: int64>
> [[23,3,21,15,20,4,10,16,23,25]]
> – child 2 type: int64
> [10]
> – child 3 type: list<item: string>
> [["qypsupwtxy","vrxptpspyt","qpvruwsuqq","ywwpyxrvrt","wswutpxxqv","tsyypstxvv","ytprpqsxsx","wtwsxvprvu","suwtrvqvwp","wtsrwywwty"]]
> – child 4 type: double
> [0]]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
