Hi, I'd like to "pickle" a Spark DataFrame object and have tried the following:
import pickle data = sparkContext.jsonFile(data_file) #load file with open('out.pickle', 'wb') as handle: pickle.dump(data, handle) If I convert "data" to a Pandas DataFrame (e.g., using data.toPandas()), the above code works. Does anybody have any idea how to do this? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Pickle-Spark-DataFrame-tp14803.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org