I am on the move at the moment so i cant try it immediately but from previous
memory / experience i think if you return plain null you will get a spark
exception
Anyway yiu can try it and see what happens and then ask the question
If you do get exception try Optional instead of plain null
Sent from Samsung Mobile
<div>-------- Original message --------</div><div>From: Olivier Girardot
<ssab...@gmail.com> </div><div>Date:2015/04/18 22:04 (GMT+00:00)
</div><div>To: Steve Lewis <lordjoe2...@gmail.com>,user@spark.apache.org
</div><div>Subject: Re: Can a map function return null </div><div>
</div>You can return an RDD with null values inside, and afterwards filter on
"item != null"
In scala (or even in Java 8) you'd rather use Option/Optional, and in Scala
they're directly usable from Spark.
Exemple :
sc.parallelize(1 to 1000).flatMap(item => if (item % 2 ==0) Some(item) else
None).collect()
res0: Array[Int] = Array(2, 4, 6, ....)
Regards,
Olivier.
Le sam. 18 avr. 2015 à 20:44, Steve Lewis <lordjoe2...@gmail.com> a écrit :
I find a number of cases where I have an JavaRDD and I wish to transform the
data and depending on a test return 0 or one item (don't suggest a filter - the
real case is more complex). So I currently do something like the following -
perform a flatmap returning a list with 0 or 1 entry depending on the isUsed
function.
JavaRDD<Foo> original = ...
JavaRDD<Foo> words = original.flatMap(new FlatMapFunction<Foo, Foo>() {
@Override
public Iterable<Foo> call(final Foo s) throws Exception {
List<Foo> ret = new ArrayList<Foo>();
if(isUsed(s))
ret.add(transform(s));
return ret; // contains 0 items if isUsed is false
}
});
My question is can I do a map returning the transformed data and null if
nothing is to be returned. as shown below - what does a Spark do with a map
function returning null
JavaRDD<Foo> words = original.map(new MapFunction<String, String>() {
@Override
Foo call(final Foo s) throws Exception {
List<Foo> ret = new ArrayList<Foo>();
if(isUsed(s))
return transform(s);
return null; // not used - what happens now
}
});
