reverse an rdd
hello... what is the best way to iterate through an rdd backward (last element first, first element last)? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reverse-an-rdd-tp16602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: reverse an rdd
Since you're concerned with the particular ordering, you will need to sort your RDD to ensure the ordering you have in mind. Simply reverse the Ordering with Ordering.reverse() and sort by that instead, and then use toLocalIterator() I suppose. Depending on what you're really trying to achieve, there may be a better way. On Thu, Oct 16, 2014 at 2:49 PM, ll duy.huynh@gmail.com wrote: hello... what is the best way to iterate through an rdd backward (last element first, first element last)? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reverse-an-rdd-tp16602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: reverse an rdd
Just to have this clear, can you answer with quick yes or no: Does it mean that when I create RDD from a file and I simply iterate through it like this: sc.textFile(some_text_file.txt).foreach(line = println(line)) then the actual lines might come in different order then they are in the file? On Thu, Oct 16, 2014 at 9:13 PM, Sean Owen so...@cloudera.com wrote: Since you're concerned with the particular ordering, you will need to sort your RDD to ensure the ordering you have in mind. Simply reverse the Ordering with Ordering.reverse() and sort by that instead, and then use toLocalIterator() I suppose. Depending on what you're really trying to achieve, there may be a better way. On Thu, Oct 16, 2014 at 2:49 PM, ll duy.huynh@gmail.com wrote: hello... what is the best way to iterate through an rdd backward (last element first, first element last)? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reverse-an-rdd-tp16602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: reverse an rdd
Nevermind, I've just run the code in the REPL. Indeed if we do not sort, then the order is totally random. Which actually makes sens if you think about it On Thu, Oct 16, 2014 at 9:58 PM, Paweł Szulc paul.sz...@gmail.com wrote: Just to have this clear, can you answer with quick yes or no: Does it mean that when I create RDD from a file and I simply iterate through it like this: sc.textFile(some_text_file.txt).foreach(line = println(line)) then the actual lines might come in different order then they are in the file? On Thu, Oct 16, 2014 at 9:13 PM, Sean Owen so...@cloudera.com wrote: Since you're concerned with the particular ordering, you will need to sort your RDD to ensure the ordering you have in mind. Simply reverse the Ordering with Ordering.reverse() and sort by that instead, and then use toLocalIterator() I suppose. Depending on what you're really trying to achieve, there may be a better way. On Thu, Oct 16, 2014 at 2:49 PM, ll duy.huynh@gmail.com wrote: hello... what is the best way to iterate through an rdd backward (last element first, first element last)? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reverse-an-rdd-tp16602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
