Thank you for providing another answer, Holden. So I did what Tarun and Michal suggested, and it didn’t work out as I want to have a new transformation method in RDD class, and need to use that RDD’s spark context which is private. So I guess the only thing I can do now is to sbt publishLocal?
> On Dec 5, 2016, at 9:19 AM, Holden Karau <hol...@pigscanfly.ca> wrote: > > Doing that requires publishing a custom version of Spark, you can edit the > version number do do a publishLocal - but maintaining that change is going to > be difficult. The other approaches suggested are probably better, but also > does your method need to be defined on the RDD class? Could you instead make > a helper object or class to expose whatever functionality you need? > > On Mon, Dec 5, 2016 at 6:06 PM long <longteng...@gmail.com > <mailto:longteng...@gmail.com>> wrote: > Thank you very much! But why can’t I just add new methods in to the source > code of RDD? > > >> On Dec 5, 2016, at 3:15 AM, Michal Šenkýř [via Apache Spark Developers List] >> <[hidden email] <http://user/SendEmail.jtp?type=node&node=20107&i=0>> wrote: >> > >> A simple Scala example of implicit classes: >> >> implicit class EnhancedString(str: String) { >> def prefix(prefix: String) = prefix + str >> } >> >> println("World".prefix("Hello ")) >> As Tarun said, you have to import it if it's not in the same class where you >> use it. >> >> Hope this makes it clearer, >> >> Michal Senkyr >> >> >> On 5.12.2016 07:43, Tarun Kumar wrote: > >>> Not sure if that's documented in terms of Spark but this is a fairly common >>> pattern in scala known as "pimp my library" pattern, you can easily find >>> many generic example of using this pattern. >>> >>> If you want I can quickly cook up a short conplete example with >>> rdd(although there is nothing really more to my example in earlier mail) ? >>> >>> Thanks >>> Tarun Kumar >>> > >>> On Mon, 5 Dec 2016 at 7:15 AM, long <<a >>> href="x-msg://22/user/SendEmail.jtp?type=node&node=20106&i=0" >>> target="_top" rel="nofollow" link="external" class="">[hidden email]> wrote: > >>> So is there documentation of this I can refer to? >>> >>>> On Dec 5, 2016, at 1:07 AM, Tarun Kumar [via Apache Spark Developers List] >>>> <[hidden email] <http://user/SendEmail.jtp?type=node&node=20104&i=0>> >>>> wrote: >>>> >>> >>>> Hi Tenglong, >>>> >>>> In addition to trsell's reply, you can add any method to an rdd without >>>> making changes to spark code. >>>> >>>> This can be achieved by using implicit class in your own client code: >>>> >>>> implicit class extendRDD[T](rdd: RDD[T]){ >>>> >>>> def foo() >>>> >>>> } >>>> >>>> Then you basically nees to import this implicit class in scope where you >>>> want to use the new foo method. >>>> >>>> Thanks >>>> Tarun Kumar >>>> > >>>> On Mon, 5 Dec 2016 at 6:59 AM, <<a href="<a >>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=0" >>>> class="">x-msg://19/user/SendEmail.jtp?type=node&node=20102&i=0" >>>> target="_top" rel="nofollow" link="external" class="">[hidden email]> >>>> wrote: >>> >>>> How does your application fetch the spark dependency? Perhaps list your >>>> project dependencies and check it's using your dev build. >>>> >>>> > >>>> On Mon, 5 Dec 2016, 08:47 tenglong, <<a href="<a >>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=1" >>>> class="">x-msg://19/user/SendEmail.jtp?type=node&node=20102&i=1" >>>> target="_top" rel="nofollow" link="external" class="">[hidden email]> >>>> wrote: >>> >>>> Hi, >>>> >>>> Apparently, I've already tried adding a new method to RDD, >>>> >>>> for example, >>>> >>>> class RDD { >>>> def foo() // this is the one I added >>>> >>>> def map() >>>> >>>> def collect() >>>> } >>>> >>>> I can build Spark successfully, but I can't compile my application code >>>> which calls rdd.foo(), and the error message says >>>> >>>> value foo is not a member of org.apache.spark.rdd.RDD[String] >>>> >>>> So I am wondering if there is any mechanism prevents me from doing this or >>>> something I'm doing wrong? >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html >>>> >>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html> >>>> Sent from the Apache Spark Developers List mailing list archive at >>>> Nabble.com <http://nabble.com/>. >>>> >>>> --------------------------------------------------------------------- > >>>> To unsubscribe e-mail: <a href="<a >>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=2" >>>> class="">x-msg://19/user/SendEmail.jtp?type=node&node=20102&i=2" >>>> target="_top" rel="nofollow" link="external" class="">[hidden email] >>>> > >>>> >>>> >>>> If you reply to this email, your message will be added to the discussion >>>> below: >>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html >>>> >>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html> >>>> To unsubscribe from Can I add a new method to RDD class?, click here <>. >>>> NAML >>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >>> View this message in context: Re: Can I add a new method to RDD class? >>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20104.html> >>> Sent from the Apache Spark Developers List mailing list archive >>> <http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com >>> <http://nabble.com/>. > >> If you reply to this email, your message will be added to the discussion >> below: > >> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html >> >> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html> >> To unsubscribe from Can I add a new method to RDD class?, click here <>. >> NAML >> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > View this message in context: Re: Can I add a new method to RDD class? > <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20107.html> > Sent from the Apache Spark Developers List mailing list archive > <http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com.
