It shouldn't, as lots of the streaming operations delegate to transform under the hood. Easiest way to make sure is to look at the source code - with a decent IDE navigating around should be a breeze.
As a matter of fact, for more advanced operations where you may want to control the partitioning (e.g. unioning 2 DStreams or a simple flatMap) you will be forced to use transform as the DStreams hide away some of the control. -adrian Sent from my iPhone > On 05 Oct 2015, at 03:59, swetha <swethakasire...@gmail.com> wrote: > > Hi, > > I have the following code for code reuse between the batch and the streaming > job > > * val groupedAndSortedSessions = > sessions.transform(rdd=>JobCommon.getGroupedAndSortedSessions(rdd))* > > The same code without code reuse between the batch and the streaming has the > following. > > * val groupedSessions = sessions.groupByKey(); > > val sortedSessions = groupedSessions.mapValues[(List[(Long, > String)])](iter => iter.toList.sortBy(_._1)) > * > > Does use of transform for code reuse affect groupByKey performance? > > > Thanks, > Swetha > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Usage-of-transform-for-code-reuse-between-Streaming-and-Batch-job-affects-the-performance-tp24920.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
