0
down vote
favorite
Is there a nice way of going from a Spark DataFrame to an EdgeRDD without
hardcoding types in the Scala code? The examples I've seen use case classes
to define the type of the EdgeRDD.
Let's assume that our Spark DataFrame has StructField ("dstID", LongType,
false) and ("srcID", LongType, false) and between 0 and 22 additional
StructField (We are constraining this so that we can use a TupleN to
represent them). Is there a clean way to define an EdgeRdd[TupleN] by
grabbing the types from the DataFrame? As motivation, consider that we are
loading a Parquet file that contains type information.
I'm very new to Spark and Scala, so I realize the question may be misguided.
In this case, I'd appreciate learning the "correct" way of thinking about
this problem.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Dataframes-to-EdgeRDD-GraphX-using-Scala-api-to-Spark-tp23548.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
