(Adding back the user list) Boromir says:
Thanks much Sean, verified 1.1.0 does not have this issue. On Mon, Sep 15, 2014 at 4:47 PM, Sean Owen <so...@cloudera.com> wrote: > Looks like another instance of > https://issues.apache.org/jira/browse/SPARK-1199 which was intended to > be fixed in 1.1.0. > > I'm not clear whether https://issues.apache.org/jira/browse/SPARK-2620 > is the same issue and therefore whether it too is resolved in 1.1? > > On Mon, Sep 15, 2014 at 4:37 PM, Boromir Widas <vcsub...@gmail.com> wrote: >> Hello Folks, >> >> I am trying to chain a couple of map operations and it seems the second map >> fails with a mismatch in arguments(event though the compiler prints them to >> be the same.) I checked the function and variable types using :t and they >> look ok to me. >> >> Have you seen this earlier? I am posting the code, data and output below. >> >> Any pointers will be greatly appreciated. >> >> Thanks, >> Boromir. >> >> /////////////////////// SCRIPT >> val data = sc.textFile("data/testpv.csv") >> >> case class KVV(key: String, valvec: Array[Double]) >> >> def mapToKV(line: String) : KVV = { >> val splits = line.split(",") >> val key = splits(0).trim >> val valvec = splits.drop(1).map(x => x.trim.toDouble) >> >> val kvv = KVV(key, valvec) >> return kvv >> } >> >> val kvs = data.map(line => mapToKV(line)) >> >> def mapKVtoKVL(kvv: KVV) : KVV = { >> return kvv >> } >> val tvar = kvs.map(x => mapKVtoKVL(x)) >> >> /////////////////////// SAMPLE DATA in testpv.csv >> x,1.1,1.2,1.3 >> y,2.1,2.2,2.3 >> >> /////////////////////// REPL OUTPUT >> scala> val data = sc.textFile("data/testpv.csv") >> 14/09/15 10:53:23 INFO MemoryStore: ensureFreeSpace(146579) called with >> curMem=0, maxMem=308713881 >> 14/09/15 10:53:23 INFO MemoryStore: Block broadcast_0 stored as values to >> memory (estimated size 143.1 KB, free 294.3 MB) >> data: org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at >> <console>:12 >> >> scala> case class KVV(key: String, valvec: Array[Double]) >> defined class KVV >> >> scala> >> >> scala> def mapToKV(line: String) : KVV = { >> | val splits = line.split(",") >> | val key = splits(0).trim >> | val valvec = splits.drop(1).map(x => x.trim.toDouble) >> | >> | val kvv = KVV(key, valvec) >> | return kvv >> | } >> mapToKV: (line: String)KVV >> >> scala> val kvs = data.map(line => mapToKV(line)) >> kvs: org.apache.spark.rdd.RDD[KVV] = MappedRDD[2] at map at <console>:18 >> >> scala> >> >> scala> def mapKVtoKVL(kvv: KVV) : KVV = { >> | return kvv >> | } >> mapKVtoKVL: (kvv: KVV)KVV >> >> scala> val tvar = kvs.map(x => mapKVtoKVL(x)) >> <console>:22: error: type mismatch; >> found : KVV >> required: KVV >> val tvar = kvs.map(x => mapKVtoKVL(x)) >> ^ --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
