Regarding this, there is this ticket still open:
https://spark-project.atlassian.net/browse/SPARK-750
For my tests, I've temporarily published on my local sbt repo the test jar
by adding :
publishArtifact in Test := true,
to the SparkBuild.scala file (ub the Settings seq of course).
Now I'm able to add this to my project:
libraryDependencies += "org.apache.spark" %% "spark-core" %
"0.9.0-incubating" % "test" classifier "test"
HTH
Andy Petrella
Belgium (Liège)
* ******** *
Data Engineer in *NextLab <http://nextlab.be/> sprl* (owner)
Engaged Citizen Coder for *WAJUG <http://wajug.be/>* (co-founder)
Author of *Learning Play! Framework 2
<http://www.packtpub.com/learning-play-framework-2/book>*
Bio: on visify <https://www.vizify.com/es/52c3feec2163aa0010001eaa>
* *********
Mobile: *+32 495 99 11 04*
Mails:
- [email protected]
- [email protected]
* *********
Socials:
- Twitter: https://twitter.com/#!/noootsab
- LinkedIn: http://be.linkedin.com/in/andypetrella
- Blogger: http://ska-la.blogspot.com/
- GitHub: https://github.com/andypetrella
- Masterbranch: https://masterbranch.com/andy.petrella
On Wed, Feb 19, 2014 at 4:57 PM, Ameet Kini <[email protected]> wrote:
> Thanks, that really helps.
>
> So that helps me cache the spark context within a suite but not across
> suites. The closest I could find to caching across suites is extending
> Suites [1] and adding @DoNotDiscover annotations to the nested suites
>
> class SparkSuites extends Suites {
> new SomeSuite1,
> new SomeSuite2,
> }
>
> But that means every time I add a new Suite, I have to go add it to
> SparkSuites.
>
> Ameet
>
> [1] http://www.artima.com/docs-scalatest-1.7.RC1/org/scalatest/Suites.html
>
>
> On Wed, Feb 19, 2014 at 1:53 AM, Heiko Braun <[email protected]>wrote:
>
>> Take a look at the trait the spark tests are using:
>>
>>
>> https://github.com/apache/incubator-spark/blob/master/core/src/test/scala/org/apache/spark/SharedSparkContext.scala?source=cc
>>
>> /Heiko
>>
>> On 18 Feb 2014, at 22:36, Ameet Kini <[email protected]> wrote:
>>
>>
>> I'm writing unit tests with Spark and need some help.
>>
>> I've already read this helpful article:
>> http://blog.quantifind.com/posts/spark-unit-test/
>>
>> There are a couple differences in my testing environment versus the blog.
>> 1. I'm using FunSpec instead of FunSuite. So my tests look like
>>
>> class MyTestSpec {
>> describe("A suite of tests") {
>> it("should do something") {
>> // test code
>> }
>> it("should do something else") {
>> // test code
>> }
>> }
>> describe("Another suite of tests") {
>> it("should do something") {
>> // test code
>> }
>> it("should do something else") {
>> // test code
>> }
>> }
>> }
>> 2. I'd like to ideally reuse the SparkContext as much as possible.
>> Currently I'm using fixture.FunSpec's withFixture and using the loan
>> pattern to loan the SparkContext to the test.
>>
>> So,
>> trait SparkEnvironment extends fixture.FunSpec {
>> def withFixture(test: OneArgTest) {
>> val sc = SparkUtils.createSparkContext("local", "some name")
>>
>> try {
>> test(sc)
>> } finally {
>> sc.stop
>> System.clearProperty("spark.driver.port")
>> }
>> }
>> }
>>
>>
>> While that works, it ends up creating a spark context per test. I'd like
>> to ideally share it across all suites (so, across more than one of my
>> TestSpec classes), and less preferably, across multiple suites within a
>> MyTestSpec class, and even less preferably, across tests within a suite,
>> but don't know how. Right now, each of my "it" tests creates a new spark
>> context, and it's really slowing it down.
>>
>> I tried creating a singleton object and loaning that object to multiple
>> tests, but Spark threw an exception saying it can't find some file. I'm
>> sure its something I'm (not) doing, as I can't think of a reason why
>> SparkContexts can't be shared across tests like that.
>>
>> object SparkEnvironment {
>> var _sc: SparkContext = null
>> def sc = {
>> if(_sc == null) _sc = SparkUtils.createSparkContext(..)
>> _sc
>> }
>> }
>>
>> Thanks,
>> Ameet
>>
>>
>>
>
