Douglas Moore created SPARK-37697:
-------------------------------------
Summary: Make it easier to convert numpy arrays to Spark Dataframes
Key: SPARK-37697
URL: https://issues.apache.org/jira/browse/SPARK-37697
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.1.2
Reporter: Douglas Moore
Make it easier to convert numpy arrays to dataframes.
Often we receive errors:
{code:java}
df = spark.createDataFrame(numpy.arange(10))
Can not infer schema for type: <class 'numpy.int64'>
{code}
OR
{code:java}
df = spark.createDataFrame(numpy.arange(10.))
Can not infer schema for type: <class 'numpy.float64'>
{code}
Today (Spark 3.x) we have to:
{code:java}
spark.createDataFrame(pd.DataFrame(numpy.arange(10.))) {code}
Make this easier with a direct conversion from Numpy arrays to Spark Dataframes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
