Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r144829187
--- Diff: python/pyspark/sql/types.py ---
@@ -1624,6 +1624,50 @@ def to_arrow_type(dt):
return arrow_type
+def to_arrow_schema(schema):
+ """ Convert a schema from Spark to Arrow
+ """
+ import pyarrow as pa
+ fields = [pa.field(field.name, to_arrow_type(field.dataType)) for
field in schema]
--- End diff --
We should add `nullable=field.nullable` just in case?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
