Hi all, We are developing an application which uses Spark & Hive to do static and ad-hoc reporting. For these static reports, they take a number of parameters and then run over a data set. We would like to make it easier to test performance of these reports on a cluster.
If we have a test cluster running with a sufficient sample data set which developers can share. To speed up development time, what is the best way to deploy a Spark application to a Spark cluster (in standalone) via an IDE? I'm thinking we would create an SBT task which would run the spark submit script. Is there a better way? Eventually this will feed into some automated performance testing which we plan to run as a twice daily Jenkins job. If its an SBT deploy task, it makes it easy to call in Jenkins. Is there a better way to do this? Posted on StackOverflow as well; http://stackoverflow.com/questions/25048784/spark-automated-deployment-performance-testing Any advice/experience appreciated! Cheers! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Deployment-Patterns-Automated-Deployment-Performance-Testing-tp11000.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
