Hi,
I just started to fiddle with Spark a week ago. I am trying to understand take
function more. I have a JavaRDD<T> variable, let’s say dataSet which is
initialized by getSequenceFileAsRDD, when I try dataSet.take(1), it throughs an
error as follows:
java.lang.IllegalStateException: Must not use direct buffers with
InputStream
But when I add add the following lines before the take call:
dataSet.context().setCheckpointDir("/tmp/checkpoint/", true);
dataSet.checkpoint();
dataSet.saveAsTextFile("/tmp/backup.txt”);
dataSet.take(1);
Then take() works. I am confused here by how RDD's take method. Can anybody
help me explain why this happens?
Thanks a lot,
-cj
