beliefer opened a new pull request, #39461:
URL: https://github.com/apache/spark/pull/39461
### What changes were proposed in this pull request?
Python: connect client should not use pyarrow.Table.to_pylist to transform
fetched data.
For example:
the data in pyarrow.Table show below.
```
pyarrow.Table
key: string
order: int64
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
----
key: [["a","a","a","a","a","b","b"]]
order: [[0,1,2,3,4,1,2]]
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):
[[null,"x","x","x","x",null,null]]
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):
[[null,"x","x","x","x",null,null]]
nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):
[[null,null,"y","y","y",null,null]]
```
The table have five columns show above.
But the data after call pyarrow.Table.to_pylist() show below.
`[{'key': 'a', 'order': 0, 'nth_value(value, 2) OVER (PARTITION BY key ORDER
BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)':
None, 'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order
ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None},
{'key': 'a', 'order': 1, 'nth_value(value, 2) OVER (PARTITION BY key ORDER BY
order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None}, {'key':
'a', 'order': 2, 'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'}, {'key':
'a', 'order': 3, 'nth_value(value, 2) OVER (PARTITION BY key ORDER
BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)':
'x', 'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order
ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'},
{'key': 'a', 'order': 4, 'nth_value(value, 2) OVER (PARTITION BY key ORDER BY
order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'}, {'key':
'b', 'order': 1, 'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None,
'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None}, {'key':
'b', 'order': 2, 'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC
NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': No
ne, 'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order
ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None}]`
There are only four columns left.
### Why are the changes needed?
Fix the bug that connect client lost column data with
`pyarrow.Table.to_pylist`.
### Does this PR introduce _any_ user-facing change?
'No'.
New feature.
### How was this patch tested?
Manual tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
