Hi All,
I have Spark-1.3.0 and Tachyon-0.5.0. When I am trying to save RDD in
tachyon, it success. But for saving a DataFrame it fails with the following
error:
java.lang.IllegalArgumentException: Wrong FS:
tachyon://localhost:19998/myres, expected: hdfs://localhost:54310
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
Here is my code:
sc.hadoopConfiguration.set("fs.tachyon.impl", "tachyon.hadoop.TFS")
val region = sc.textFile("tachyon://localhost:19998/user/hduser/region.tbl")
val nation = sc.textFile("tachyon://localhost:19998/user/hduser/nation.tbl")
val regionSchemaString = "R_REGIONKEY R_NAME R_COMMENT"
val nationSchemaString = "N_NATIONKEY N_NAME N_REGIONKEY N_COMMENT"
val regionSchema =
StructType(
regionSchemaString.split(" ").map(fieldName => StructField(fieldName,
StringType, true)))
val nationSchema =
StructType(
nationSchemaString.split(" ").map(fieldName => StructField(fieldName,
StringType, true)))
val regionRowRDD = region.map(_.split("\\|")).map(r => Row(r(0), r(1),
r(2)))
val nationRowRDD = nation.map(_.split("\\|")).map(r => Row(r(0), r(1), r(2),
r(3)))
val regionDataFrame = sqlContext.createDataFrame(regionRowRDD, regionSchema)
val nationDataFrame = sqlContext.createDataFrame(nationRowRDD, nationSchema)
regionDataFrame.registerTempTable("REGION")
nationDataFrame.registerTempTable("NATION")
val df = sqlContext.sql("select * from REGION r, NATION n where
r.R_REGIONKEY = n.N_REGIONKEY ")
df.count()
df.save("tachyon://localhost:19998/myres")
Could anyone help me please?
Thanks,
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Saving-DataFrame-in-Tachyon-tp12498.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
