Thank you! Best, Wenlei
On Tue, Nov 19, 2013 at 7:20 AM, Mark Hamstra <[email protected]>wrote: > mapWith can make this use case even simpler. > > > > On Nov 19, 2013, at 1:29 AM, Sebastian Schelter <[email protected]> > wrote: > > You can use mapPartition, which allows you to apply the map function > elementwise to all elements of a partition. Here you can place custom code > around your function invocation that lets you reuse the array. > > --sebastian > Am 19.11.2013 10:02 schrieb "Wenlei Xie" <[email protected]>: > >> Hi, >> >> I am trying to do some tasks with the following style map function: >> >> rdd.map { e => >> val a = new Array[Int](100) >> ...Some calculation... >> } >> >> But here the array a is really just used as a temporary buffer and can be >> reused. I am wondering if I can avoid constructing it everytime? (As it >> might incur some overhead for JVM?) Would use an array outside the closure >> work? >> >> Thank you! >> >> Best, >> Wenlei >> >> -- Wenlei Xie (谢文磊) Department of Computer Science 5132 Upson Hall, Cornell University Ithaca, NY 14853, USA Phone: (607) 255-5577 Email: [email protected]
