Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/22807#discussion_r227571481
--- Diff: python/pyspark/serializers.py ---
@@ -248,7 +248,14 @@ def create_array(s, t):
# TODO: see ARROW-2432. Remove when the minimum PyArrow
version becomes 0.10.0.
return pa.Array.from_pandas(s.apply(
lambda v: decimal.Decimal('NaN') if v is None else v),
mask=mask, type=t)
- return pa.Array.from_pandas(s, mask=mask, type=t)
+ try:
+ array = pa.Array.from_pandas(s, mask=mask, type=t)
+ except pa.ArrowInvalid as e:
--- End diff --
I'm not sure if there are other issues that could case an `ArrowInvalid`
error. What do you think about catching an `ArrowException` and making the
message a little more generic?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
