gaogaotiantian commented on PR #56943:
URL: https://github.com/apache/spark/pull/56943#issuecomment-4860335076
A few reasons I'm against this:
1. This makes `pandas` a hard requirement for UDF servers, where it wasn't.
2. The code is significantly more difficult to read and understand (for
humans at least).
3. We introduced another type layer which could potentially cause all kinds
of coercion issue between pandas/arrow/python. It might work for simple cases,
but I'm pretty sure we'll hit corner cases all the time.
To me, this is an arrow issue. `to_pylist()` should not be significantly
slower than converting the arrow column to pandas then back to Python objects.
```python
result = []
for i in range(self.num_chunks):
result +=
self.chunk(i).to_pylist(maps_as_pydicts=maps_as_pydicts)
return result
```
This is probably where they did wrong. For large chunks, extending a list
might cause memory-reallocation all the time and could potentially slow down
the code. I don't have a solid proof for that but we could fix pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
