Hi Reynold, Is this already reported and tracked somewhere. I'm quite sure that people will be asking about the reasons Spark does this. Where are such issues reported usually?
Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Jul 5, 2016 at 6:19 PM, Reynold Xin <r...@databricks.com> wrote: > This seems like a Scala compiler bug. > > > On Tuesday, July 5, 2016, Jacek Laskowski <ja...@japila.pl> wrote: >> >> Well, there is foreach for Java and another foreach for Scala. That's >> what I can understand. But while supporting two language-specific APIs >> -- Scala and Java -- Dataset API lost support for such simple calls >> without type annotations so you have to be explicit about the variant >> (since I'm using Scala I want to use Scala API right). It appears that >> any single-argument-function operators in Datasets are affected :( >> >> My question was to know whether there are works to fix it (if possible >> -- I don't know if it is). >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> >> On Tue, Jul 5, 2016 at 4:21 PM, Sean Owen <so...@cloudera.com> wrote: >> > Right, should have noticed that in your second mail. But foreach >> > already does what you want, right? it would be identical here. >> > >> > How these two methods do conceptually different things on different >> > arguments. I don't think I'd expect them to accept the same functions. >> > >> > On Tue, Jul 5, 2016 at 3:18 PM, Jacek Laskowski <ja...@japila.pl> wrote: >> >> ds is Dataset and the problem is that println (or any other >> >> one-element function) would not work here (and perhaps other methods >> >> with two variants - Java's and Scala's). >> >> >> >> Pozdrawiam, >> >> Jacek Laskowski >> >> ---- >> >> https://medium.com/@jaceklaskowski/ >> >> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> >> Follow me at https://twitter.com/jaceklaskowski >> >> >> >> >> >> On Tue, Jul 5, 2016 at 3:53 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> A DStream is a sequence of RDDs, not of elements. I don't think I'd >> >>> expect to express an operation on a DStream as if it were elements. >> >>> >> >>> On Tue, Jul 5, 2016 at 2:47 PM, Jacek Laskowski <ja...@japila.pl> >> >>> wrote: >> >>>> Sort of. Your example works, but could you do a mere >> >>>> ds.foreachPartition(println)? Why not? What should I even see the >> >>>> Java >> >>>> version? >> >>>> >> >>>> scala> val ds = spark.range(10) >> >>>> ds: org.apache.spark.sql.Dataset[Long] = [id: bigint] >> >>>> >> >>>> scala> ds.foreachPartition(println) >> >>>> <console>:26: error: overloaded method value foreachPartition with >> >>>> alternatives: >> >>>> (func: >> >>>> org.apache.spark.api.java.function.ForeachPartitionFunction[Long])Unit >> >>>> <and> >> >>>> (f: Iterator[Long] => Unit)Unit >> >>>> cannot be applied to (Unit) >> >>>> ds.foreachPartition(println) >> >>>> ^ >> >>>> >> >>>> Pozdrawiam, >> >>>> Jacek Laskowski >> >>>> ---- >> >>>> https://medium.com/@jaceklaskowski/ >> >>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> >>>> Follow me at https://twitter.com/jaceklaskowski >> >>>> >> >>>> >> >>>> On Tue, Jul 5, 2016 at 3:32 PM, Sean Owen <so...@cloudera.com> wrote: >> >>>>> Do you not mean ds.foreachPartition(_.foreach(println)) or similar? >> >>>>> >> >>>>> On Tue, Jul 5, 2016 at 2:22 PM, Jacek Laskowski <ja...@japila.pl> >> >>>>> wrote: >> >>>>>> Hi, >> >>>>>> >> >>>>>> It's with the master built today. Why can't I call >> >>>>>> ds.foreachPartition(println)? Is using type annotation the only way >> >>>>>> to >> >>>>>> go forward? I'd be so sad if that's the case. >> >>>>>> >> >>>>>> scala> ds.foreachPartition(println) >> >>>>>> <console>:28: error: overloaded method value foreachPartition with >> >>>>>> alternatives: >> >>>>>> (func: >> >>>>>> org.apache.spark.api.java.function.ForeachPartitionFunction[Record])Unit >> >>>>>> <and> >> >>>>>> (f: Iterator[Record] => Unit)Unit >> >>>>>> cannot be applied to (Unit) >> >>>>>> ds.foreachPartition(println) >> >>>>>> ^ >> >>>>>> >> >>>>>> scala> sc.version >> >>>>>> res9: String = 2.0.0-SNAPSHOT >> >>>>>> >> >>>>>> Pozdrawiam, >> >>>>>> Jacek Laskowski >> >>>>>> ---- >> >>>>>> https://medium.com/@jaceklaskowski/ >> >>>>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> >>>>>> Follow me at https://twitter.com/jaceklaskowski >> >>>>>> >> >>>>>> >> >>>>>> --------------------------------------------------------------------- >> >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>>>>> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
