Re: Adding Spark Testing functionality
So here is a quick description of the current testing bits (I can expand on it if people are interested) http://bit.ly/pandaPandaPanda . On Tue, Oct 6, 2015 at 3:49 PM, Holden Karauwrote: > I'll put together a google doc and send that out (in the meantime a quick > guide of sort of how the current package can be used is in the blog post I > did at > http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/ > ) If people think its better to keep as a package I am of course happy to > keep doing that. It feels a little strange to have something as core as > being able to test your code live outside. > > On Tue, Oct 6, 2015 at 3:44 PM, Patrick Wendell > wrote: > >> Hey Holden, >> >> It would be helpful if you could outline the set of features you'd >> imagine being part of Spark in a short doc. I didn't see a README on the >> existing repo, so it's hard to know exactly what is being proposed. >> >> As a general point of process, we've typically avoided merging modules >> into Spark that can exist outside of the project. A testing utility package >> that is based on Spark's public API's seems like a really useful thing for >> the community, but it does seem like a good fit for a package library. At >> least, this is my first question after taking a look at the project. >> >> In any case, getting some high level view of the functionality you >> imagine would be helpful to give more detailed feedback. >> >> - Patrick >> >> On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau >> wrote: >> >>> Hi Spark Devs, >>> >>> So this has been brought up a few times before, and generally on the >>> user list people get directed to use spark-testing-base. I'd like to start >>> moving some of spark-testing-base's functionality into Spark so that people >>> don't need a library to do what is (hopefully :p) a very common requirement >>> across all Spark projects. >>> >>> To that end I was wondering what peoples thoughts are on where this >>> should live inside of Spark. I was thinking it could either be a separate >>> testing project (like sql or similar), or just put the bits to enable >>> testing inside of each relevant project. >>> >>> I was also thinking it probably makes sense to only move the unit >>> testing parts at the start and leave things like integration testing in a >>> testing project since that could vary depending on the users environment. >>> >>> What are peoples thoughts? >>> >>> Cheers, >>> >>> Holden :) >>> >> >> > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > Linked In: https://www.linkedin.com/in/holdenkarau > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
Adding Spark Testing functionality
Hi Spark Devs, So this has been brought up a few times before, and generally on the user list people get directed to use spark-testing-base. I'd like to start moving some of spark-testing-base's functionality into Spark so that people don't need a library to do what is (hopefully :p) a very common requirement across all Spark projects. To that end I was wondering what peoples thoughts are on where this should live inside of Spark. I was thinking it could either be a separate testing project (like sql or similar), or just put the bits to enable testing inside of each relevant project. I was also thinking it probably makes sense to only move the unit testing parts at the start and leave things like integration testing in a testing project since that could vary depending on the users environment. What are peoples thoughts? Cheers, Holden :)
Re: Adding Spark Testing functionality
Hey Holden, It would be helpful if you could outline the set of features you'd imagine being part of Spark in a short doc. I didn't see a README on the existing repo, so it's hard to know exactly what is being proposed. As a general point of process, we've typically avoided merging modules into Spark that can exist outside of the project. A testing utility package that is based on Spark's public API's seems like a really useful thing for the community, but it does seem like a good fit for a package library. At least, this is my first question after taking a look at the project. In any case, getting some high level view of the functionality you imagine would be helpful to give more detailed feedback. - Patrick On Tue, Oct 6, 2015 at 3:12 PM, Holden Karauwrote: > Hi Spark Devs, > > So this has been brought up a few times before, and generally on the user > list people get directed to use spark-testing-base. I'd like to start > moving some of spark-testing-base's functionality into Spark so that people > don't need a library to do what is (hopefully :p) a very common requirement > across all Spark projects. > > To that end I was wondering what peoples thoughts are on where this should > live inside of Spark. I was thinking it could either be a separate testing > project (like sql or similar), or just put the bits to enable testing > inside of each relevant project. > > I was also thinking it probably makes sense to only move the unit testing > parts at the start and leave things like integration testing in a testing > project since that could vary depending on the users environment. > > What are peoples thoughts? > > Cheers, > > Holden :) >
Re: Adding Spark Testing functionality
I'll put together a google doc and send that out (in the meantime a quick guide of sort of how the current package can be used is in the blog post I did at http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/ ) If people think its better to keep as a package I am of course happy to keep doing that. It feels a little strange to have something as core as being able to test your code live outside. On Tue, Oct 6, 2015 at 3:44 PM, Patrick Wendellwrote: > Hey Holden, > > It would be helpful if you could outline the set of features you'd imagine > being part of Spark in a short doc. I didn't see a README on the existing > repo, so it's hard to know exactly what is being proposed. > > As a general point of process, we've typically avoided merging modules > into Spark that can exist outside of the project. A testing utility package > that is based on Spark's public API's seems like a really useful thing for > the community, but it does seem like a good fit for a package library. At > least, this is my first question after taking a look at the project. > > In any case, getting some high level view of the functionality you imagine > would be helpful to give more detailed feedback. > > - Patrick > > On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau wrote: > >> Hi Spark Devs, >> >> So this has been brought up a few times before, and generally on the user >> list people get directed to use spark-testing-base. I'd like to start >> moving some of spark-testing-base's functionality into Spark so that people >> don't need a library to do what is (hopefully :p) a very common requirement >> across all Spark projects. >> >> To that end I was wondering what peoples thoughts are on where this >> should live inside of Spark. I was thinking it could either be a separate >> testing project (like sql or similar), or just put the bits to enable >> testing inside of each relevant project. >> >> I was also thinking it probably makes sense to only move the unit testing >> parts at the start and leave things like integration testing in a testing >> project since that could vary depending on the users environment. >> >> What are peoples thoughts? >> >> Cheers, >> >> Holden :) >> > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau