Hi, I'd like to "pickle" a Spark DataFrame object and have tried the
following:
import pickle
data = sparkContext.jsonFile(data_file) #load file
with open('out.pickle', 'wb') as handle:
pickle.dump(data, handle)
If I convert "data" to a Pandas DataFrame (e.g., using data.toPandas()), the
above code works. Does anybody have any idea how to do this?
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Pickle-Spark-DataFrame-tp14803.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
