Have you tried the spark-csv package?
https://spark-packages.org/package/databricks/spark-csv
________________________________
From: Raymond Xie <xie3208...@gmail.com>
Sent: Friday, December 30, 2016 6:46:11 PM
To: user@spark.apache.org
Subject: How to load a big csv to dataframe in Spark 1.6
Hello,
I see there is usually this way to load a csv to dataframe:
sqlContext = SQLContext(sc)
Employee_rdd = sc.textFile("\..\Employee.csv")
.map(lambda line: line.split(","))
Employee_df = Employee_rdd.toDF(['Employee_ID','Employee_name'])
Employee_df.show()
However in my case my csv has 100+ fields, which means toDF() will be very
lengthy.
Can anyone tell me a practical method to load the data?
Thank you very much.
Raymond
