Re: Can I add a new method to RDD class?

2016-12-07 Thread Teng Long
ersions:set -DnewVersion=your_new_version > > On Wed, Dec 7, 2016 at 11:31 AM, Teng Long <[hidden email] > > wrote: > Hi Holden, > > Can you please tell me how to edit version numbers efficiently? the correct > way? I'm really struggling with this and don't know where to loo

Re: Can I add a new method to RDD class?

2016-12-06 Thread Teng Long
, I think changing the property (line 29) in spark's root > pom.xml should be sufficient. However, keep in mind that you'll also > need to publish spark locally before you can access it in your test > application. > > On Tue, Dec 6, 2016 at 2:50 AM, Teng Long <[hidden email

Re: Can I add a new method to RDD class?

2016-12-06 Thread Teng Long
Thank you Jokob for clearing things up for me. Before, I thought my application was compiled against my local build since I can get all the logs I just added in spark-core. But it was all along using spark downloaded from remote maven repository, and that’s why I “cannot" add new RDD methods

Re: Can I add a new method to RDD class?

2016-12-05 Thread Teng Long
k.rdd”. It seems like my import statement is wrong, but I don’t know how? Thanks! > On Dec 5, 2016, at 5:14 PM, Teng Long <longteng...@gmail.com> wrote: > > I’m trying to implement a transformation that can merge partitions (to align > with GPU specs) and move them onto GPU memory

Re: Can I add a new method to RDD class?

2016-12-05 Thread Teng Long
rk-cassandra-connector/blob/master/doc/5_saving.md> > > > case class WordCount(word: String, count: Long) > > val collection = sc.parallelize(Seq(WordCount("dog", 50), WordCount("cow", > 60))) > > collection.saveToCassandra("test", &q

Re: Can I add a new method to RDD class?

2016-12-05 Thread Teng Long
he.spark.SparkContext > > <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD@sparkContext:org.apache.spark.SparkContext> > > On Mon, Dec 5, 2016 at 1:04 PM, Teng Long <longteng...@gmail.com > <mailto:longteng...@gmail.com>> w

Re: Can I add a new method to RDD class?

2016-12-05 Thread Teng Long
Thank you for providing another answer, Holden. So I did what Tarun and Michal suggested, and it didn’t work out as I want to have a new transformation method in RDD class, and need to use that RDD’s spark context which is private. So I guess the only thing I can do now is to sbt publishLocal?