I was having query regarding narrow transformation that is it ok to write adhoc logic into single transformation or we should split it into multiple tranfromations for ex: if i would like to explode one of the list and then using value of list again explode it and then using its values again attach something to it. So one of the way is to write complete logic inside one flatmap or split into multiple transformations like flatmap -> flatmap -> map i.e small units . I feel 2nd way is better than writing everything into single transformations Asking because i saw many developer writing single flatmap and write complete logic into single flat map i.e some times they use multiple maps and flatmap inside single flatmap , i think it will affect performance as it may end up producing lot of objects and will pressure GC. Would be thankful if anyone can share their thoughts.
-- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
