[
https://issues.apache.org/jira/browse/ARROW-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17543685#comment-17543685
]
Antoine Pitrou commented on ARROW-16629:
----------------------------------------
If I understand the benchmark correctly, most likely the Python-Arrow back and
forth conversion is the main cost factor. Could you profile to get a bit more
insight?
> Apache Arrow Flight transport speed improvement for list structures
> -------------------------------------------------------------------
>
> Key: ARROW-16629
> URL: https://issues.apache.org/jira/browse/ARROW-16629
> Project: Apache Arrow
> Issue Type: Improvement
> Components: FlightRPC
> Affects Versions: 8.0.0
> Reporter: David Lee
> Priority: Minor
>
> I just started testing using Arrow Flight to send results from a GraphQL
> server with FlightServer() running on i.
> GraphQL defines a schema for your data output which can be mapped to an Arrow
> schema so I thought it would make sense to try using Arrow Flight to
> transport results instead of using REST style JSON records.
> Arrow Flight was 66% faster in all case, but it didn't scale as the number of
> child records increased. I suspect that serializing structs or lists needs
> some improvement..
> Here is the discussion I opened including links to test scripts.
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_server.py]
> Standard ASGI server. Start it up with uvicorn --host=0.0.0.0
> test_asgi_server:app
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_arrow_flight_server.py]
> New Arrow Flight server. Start it up with python test_arrow_flight_server.py
> [https://github.com/davlee1972/ariadne_arrow/blob/arrow_flight/benchmark/test_asgi_arrow_client.py]
> Benchmarking script. Pass in length of lists to test and server host.
>
> Discussion from Ariadne GraphQL package.
> [https://github.com/mirumee/ariadne/discussions/867]
> 10 records it was 0.049 seconds faster or 80% faster
> 10000 records it was 0.109 seconds faster or 66% faster
> 10 million records it was 54 seconds faster or 66% faster.
> Also here is the data structure that is sent across the wire..
> pyarrow.Table
> data: struct<test_lists: struct<float_list: list<item: double>, int_list:
> list<item: int64>, length: int64, string_list: list<item: string>,
> time_spent: double>>
> child 0, test_lists: struct<float_list: list<item: double>, int_list:
> list<item: int64>, length: int64, string_list: list<item: string>,
> time_spent: double>
> child 0, float_list: list<item: double>
> child 0, item: double
> child 1, int_list: list<item: int64>
> child 0, item: int64
> child 2, length: int64
> child 3, string_list: list<item: string>
> child 0, item: string
> child 4, time_spent: double
> data: [
> – is_valid: all not null
> – child 0 type: struct<float_list: list<item: double>, int_list: list<item:
> int64>, length: int64, string_list: list<item: string>, time_spent: double>
> – is_valid: all not null
> – child 0 type: list<item: double>
> [[13.500371672273381,17.747395152140353,28.973205439157457,1.361443415643098,19.029191125636135,14.62284718057391,18.44333922481529,7.906278860251386,14.402464768126993,5.826040531772251]]
> – child 1 type: list<item: int64>
> [[23,3,21,15,20,4,10,16,23,25]]
> – child 2 type: int64
> [10]
> – child 3 type: list<item: string>
> [["qypsupwtxy","vrxptpspyt","qpvruwsuqq","ywwpyxrvrt","wswutpxxqv","tsyypstxvv","ytprpqsxsx","wtwsxvprvu","suwtrvqvwp","wtsrwywwty"]]
> – child 4 type: double
> [0]]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
