Hi,
I have the following pair RDD created in java.
JavaPairRDD<Integer, ProgramDataRef> progRef =
sc.textFile(programReferenceDataFile, 12).filter(
(String s) -> !s.startsWith("#")).mapToPair(
(String s) -> {
ProgramDataRef ref = new
ProgramDataRef(s);
return new Tuple2<Integer,
ProgramDataRef>(ref.startTime, ref);
}
);
I need to partition this RDD using the provided RangePartitioner in the
framework. However, I am unable to determine what the input parameters are
in order to create the partitioner.
new RangePartitioner(12, ?, true, ?, ?);
I am specifically looking for info on the 2nd parameter - RDD<? extends
scala.Product2<K,V>> rdd - which doesn't match my pair RDD. How do I create
an RDD of this type from the RDD I have (or the input text file).
Also, the 4th and 5th parameter - scala.math.Ordering<K> evidence$1,
scala.reflect.ClassTag<K> evidence$2. How do I create these classes based on
my key type of Integer.
Finally once I have my RDD partitioned I want to call
OrderedRDDFunctions.filterByRange(...). I believe once I have the answers to
the above question I will have the input parameters in order to create an
instance of OrderedRDDFunctions.
Thanks,
Dave.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/RangePartitioning-tp26010.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
