Hi all,
I am trying to work with spark-redis connector (redislabs) which
requires all transactions between redis and spark be in RDD's. The language
I am using is Java but the connector does not accept JavaRDD's .So I tried
using Spark context in my code instead of JavaSparkContext. But when I
wanted to create a RDD using sc.parallelize , it asks for some scala related
parameters as opposed to lists in java.... when I tries to have both
javaSparkContext and sparkcontext(for connector) then Multiple contexts
cannot be opened was the error....
The code that I have been trying ....
// initialize spark context
private static RedisContext config() {
conf = new SparkConf().setAppName("redis-jedis");
sc2=new SparkContext(conf);
RedisContext rc=new RedisContext(sc2);
return rc;
}
//write to redis which requires the data to be in RDD
private static void WriteUserTacticData(RedisContext rc, String userid,
String tacticsId, String value) {
hostTup= calling(redisHost,redisPort);
String key=userid+"-"+tacticsId;
RDD<Tuple2<String, String>> newTup=createTuple(key,value);
rc.toRedisKV(newTup,hostTup);
// the createTuple where the RDD is to be created which will be inserted
into redis
private static RDD<Tuple2<String, String>> createTuple(String key,
String value) {
sc=new JavaSparkContext(conf);
ArrayList<Tuple2<String,String>> list= new
ArrayList<Tuple2<String,String>>();
Tuple2<String,String> e= new Tuple2<String, String>(key,value);
list.add(e);
JavaRDD<Tuple2<String,String>> javardd= sc.parallelize(list);
RDD<Tuple2<String,String>> newTupRdd=JavaRDD.toRDD(javardd);
sc.close();
return newTupRdd;
}
How would I create an RDD(not javaRDD) in java which will be accepted by
redis connector... Any kind of related to the topic would be
appretiated......
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/using-JavaRDD-in-spark-redis-connector-tp14391.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
