I imagine it is because parallelize() inherently only makes sense for smallish data to begin with, since it will have to be broadcast from the driver. Large enough data should probably live in distributed storage to begin with.
The Scala equivalent wants a Seq, so I assume there is some need or value in knowing the size of the input, which Iterable does not give. (I am guessing Java's version could have take a Collection, but hey.) I don't know that it makes sense to contemplate infinite distributed collections. You could do something like an RDD with 10 elements, each of which was an infinite Range, each generating every 10th Fibonacci number, and at least grapple with it in 10-way parallelism, but I don't know if that's of much practical use? what save/count method at the end would make sense? On Mon, Sep 22, 2014 at 5:22 PM, Steve Lewis <[email protected]> wrote: > The only way I find is to turn it into a list - in effect holding > everything in memory (see code below). Surely Spark has a better way. > > Also what about unterminated iterables like a Fibonacci series - (useful > only if limited in some other way ) > > > /** > * make an RDD from an iterable > * @param inp input iterator > * @param ctx context > * @param <T> type > * @return rdd from inerator as a list > */ > public static @Nonnull <T> JavaRDD<T> fromIterable(@Nonnull final > Iterable<T> inp,@Nonnull final JavaSparkContext ctx) { > List<T> holder = new ArrayList<T>(); > for (T k : inp) { > holder.add(k); > } > return ctx.parallelize(holder); > } > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
