The output of lapply and lapplyPartition should the same by design -- The only difference is that in lapply the user-defined function returns a row, while it returns a list in lapplyPartition.
Could you given an example of a small input and output that you expect to see for the above program ? Shivaram On Wed, Aug 6, 2014 at 5:47 AM, Pranay Dave <pranay.da...@gmail.com> wrote: > Hello > As per documentation, lapply works on single records and lapplyPartition > works on partition > However the format of output does not change > > When I use lapplypartition, the data is converted to vertical format > > Here is my code > library(SparkR) > > > sc <- sparkR.init("local") > lines <- textFile(sc,"/sparkdev/datafiles/covariance.txt") > > totals <- lapply(lines, function(lines) > { > > > sumx <- 0 > sumy <- 0 > totaln <- 0 > for (i in 1:length(lines)){ > dataxy <- unlist(strsplit(lines[i], ",")) > sumx <- sumx + as.numeric(dataxy[1]) > sumy <- sumy + as.numeric(dataxy[2]) > > } > > ##list(as.numeric(sumx), as.numeric(sumy), as.numeric(sumxy), > as.numeric(totaln)) > ##list does same as below > c(sumx,sumy) > > } > > ) > > output <- collect(totals) > for (element in output) { > cat(as.character(element[1]),as.character(element[2]), "\n") > } > > I am expecting output as 55, 55 > However it is giving > 55,NA > 55,NA > > Where am I going wrong ? > Thanks > Pranay > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-lapplyPartition-transforms-the-data-in-vertical-format-tp11540.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >