It sounds like mapPartitionsWithIndex will give you the information you
want over flatMap.
On Thursday, October 13, 2016, Shushant Arora <shushantaror...@gmail.com>
wrote:
> Hi
>
> I have a transformation on a pair rdd using flatmap function.
>
> 1.Can I detect in flatmap whether the current record is last record of
> partition being processed and
> 2. what is the partition index of this partition.
>
> public Iterable<Tuple2<String, String>> call(Tuple2<String, String> t)
> throws Exception {
> //whether element is last elemet of current partiton or not ??
> //partition index to which element belongs
> to
> }
> });
> Thanks
>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
