Hi, we share the same spark/hive context between tests (executed in parallel), so the main problem is that the temporary tables are overwritten each time they are created, this may create race conditions as these tempTables may be seen as global mutable shared state.
So each time we create a temporary table, we add an unique, incremented, thread safe id (AtomicInteger) to its name so that there are only specific, non-shared temporary tables used for a test. -- Bedrytski Aliaksandr sp...@bedryt.ski > On Sat, Aug 20, 2016, at 01:25, Everett Anderson wrote: > Hi! > > Just following up on this -- > > When people talk about a shared session/context for testing like this, > I assume it's still within one test class. So it's still the case that > if you have a lot of test classes that test Spark-related things, you > must configure your build system to not run in them in parallel. > You'll get the benefit of not creating and tearing down a Spark > session/context between test cases with a test class, though. > > Is that right? > > Or have people figured out a way to have sbt (or Maven/Gradle/etc) > share Spark sessions/contexts across integration tests in a safe way? > > > On Mon, Aug 1, 2016 at 3:23 PM, Holden Karau > <hol...@pigscanfly.ca> wrote: > Thats a good point - there is an open issue for spark-testing-base to > support this shared sparksession approach - but I haven't had the > time ( https://github.com/holdenk/spark-testing-base/issues/123 ). > I'll try and include this in the next release :) > > On Mon, Aug 1, 2016 at 9:22 AM, Koert Kuipers > <ko...@tresata.com> wrote: > we share a single single sparksession across tests, and they can run > in parallel. is pretty fast > > On Mon, Aug 1, 2016 at 12:02 PM, Everett Anderson > <ever...@nuna.com.invalid> wrote: > Hi, > > Right now, if any code uses DataFrame/Dataset, I need a test setup > that brings up a local master as in this article[1]. > > That's a lot of overhead for unit testing and the tests can't run > in parallel, so testing is slow -- this is more like what I'd call > an integration test. > > Do people have any tricks to get around this? Maybe using spy mocks > on fake DataFrame/Datasets? > > Anyone know if there are plans to make more traditional unit > testing possible with Spark SQL, perhaps with a stripped down in- > memory implementation? (I admit this does seem quite hard since > there's so much functionality in these classes!) > > Thanks! > > > - Everett > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau
