I see.
The reason we can use flatmap to map to null but not using map to map to
null is because
flatmap supports map to zero and more but map only support 1-1 mapping?
It seems Flatmap is more equivalent to haddop's map.
Thanks,
Zheng zhen
On Thu, Feb 26, 2015 at 10:44 AM, Sean Owen <so...@cloudera.com> wrote:
> You can flatMap:
>
> rdd.flatMap { in =>
> if (condition(in)) {
> Some(transformation(in))
> } else {
> None
> }
> }
>
> On Thu, Feb 26, 2015 at 6:39 PM, Crystal Xing <crystalxin...@gmail.com>
> wrote:
> > Hi,
> > I have a text file input and I want to parse line by line and map each
> line
> > to another format. But at the same time, I want to filter out some lines
> I
> > do not need.
> >
> > I wonder if there is a way to filter out those lines in the map function.
> >
> > Do I have to do two steps filter and map? In that way, I have to scan
> and
> > parse the lines twice in order to filter and map.
> >
> > If I map those unwanted line to null and filter out null, will that work?
> > never tried yet.
> >
> > Thanks,
> >
> > Zheng zheng
>
