[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16276 Thanks @linbojin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user linbojin commented on the issue: https://github.com/apache/spark/pull/16276 @rxin Thx, I just deep dive into spark, I hope i can try to contribute more impactful things later ð . I will close this one. @srowen I tried out your idea that i created `TraversableRDDFunctions` file and `implicit def rddToTraversableRDDFunctions[U](rdd: RDD[TraversableRDDFunctions[U]])` inside RDD object. It's not very complex, but the problem is class RDD is **invariant** so that it's hard to make this method generic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16276 Do we really need to add this? For the time we spent we can work on more impactful things ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user linbojin commented on the issue: https://github.com/apache/spark/pull/16276 @srowen So any problems with my current implementation, i mean use `implicit asTraversable: T => TraversableOnce[U]`. Because I refer to `flatten` implementation in scala source code: https://github.com/scala/scala/blob/05016d9035ab9b1c866bd9f12fdd0491f1ea0cbb/src/library/scala/collection/generic/GenericTraversableTemplate.scala#L169 and if user call flatten on invalid rdd, it will just error out: ``` scala> val rdd = sc.parallelize(Seq(1,2,3)) rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at :24 scala> rdd.flatten :27: error: No implicit view available from Int => scala.collection.TraversableOnce[U]. rdd.flatten ^ ``` same as scala ``` scala> val l = List(1,2,3) l: List[Int] = List(1, 2, 3) scala> l.flatten :13: error: No implicit view available from Int => scala.collection.GenTraversableOnce[B]. l.flatten ^ ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16276 Yes you're hitting a key point, that `flatten` only makes sense on an RDD of `TraversableOnce`. I think we'd implement this in the way that `DoubleRDDFunctions` adds operations for only RDDs of doubles instead of adding this implicit transformation. In that case you can get an `iterator` directly from each element, and then concatenating them. I think it goes something like: ``` iter => iter.foldLeft(Iterator.empty)((a,b) => a ++ b.toIterator) ``` ... but I'm sure that's not 100% correct. Anyway the complexity makes me wonder if it's worth doing unless the lack of a direct method is a bottleneck. (We could also look at whether Spark internally could replace flatMap(x => x) with the new method) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user linbojin commented on the issue: https://github.com/apache/spark/pull/16276 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16276: [SPARK-18855][CORE] Add RDD flatten function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16276 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
