Hi, I think we should get SparkServer into our integration tests. That is, instead of using local[4] everywhere, we actually go against SparkServer. I know many situations where local[4] works but SparkServer fails (usually around jar mismatch stuff --- all project should use mvn-enforcer! (though thats another story)). While we are at, why not have Hadoop/HDFS in integration testing as well. ?
Does anyone have a good idea on how to do that? Thanks, Marko. http://markorodriguez.com
