Is anyone else nervous about ignoring this issue or relying on non-build (hand run) test driven transitive dependency checking. I hope someone else will chime in.
As to running unit tests on a TEST_MASTER I’ll look into it. Can we set up the build machine to do this? I’d feel better about eyeballing deps if we could have a TEST_MASTER automatically run during builds at Apache. Maybe the regular unit tests are OK for building locally ourselves. > > On Oct 20, 2014, at 12:23 PM, Dmitriy Lyubimov <[email protected]> wrote: > > On Mon, Oct 20, 2014 at 11:44 AM, Pat Ferrel <[email protected]> wrote: > >> Maybe a more fundamental issue is that we don’t know for sure whether we >> have missing classes or not. The job.jar at least used the pom dependencies >> to guarantee every needed class was present. So the job.jar seems to solve >> the problem but may ship some unnecessary duplicate code, right? >> > > No, as i wrote spark doesn't work with job jar format. Neither as it turns > out more recent hadoop MR btw. Not speaking literally of the format. Spark understands jars and maven can build one from transitive dependencies. > > Yes, this is A LOT of duplicate code (will take normally MINUTES to startup > tasks with all of it just on copy time). This is absolutely not the way to > go with this. > Lack of guarantee to load seems like a bigger problem than startup time. Clearly we can’t just ignore this. > >> There may be any number of bugs waiting for the time we try running on a >> node machine that doesn’t have some class in it’s classpath. > > > No. Assuming any given method is tested on all its execution paths, there > will be no bugs. The bugs of that sort will only appear if the user is > using algebra directly and calls something that is not on the path, from > the closure. In which case our answer to this is the same as for the solver > methodology developers -- use customized SparkConf while creating context > to include stuff you really want. > > Also another right answer to this is that we probably should reasonably > provide the toolset here. For example, all the stats stuff found in R base > and R stat packages so the user is not compelled to go non-native. > > Huh? this is not true. The one I ran into was found by calling something in math from something in math-scala. It led outside and you can encounter such things even in algebra. In fact you have no idea if these problems exists except for the fact you have used it a lot personally. Test catches these only if they are _not_ local unit tests but run on a TEST_MASTER and we would need very high test coverage. Test dependent rather than linker/statically checked—isn’t that why we love type safety? Actually I personally have no problem with non-type safe test-driven guarantees just so we’re all clear about what this implies. Javascript seems to thrive after all... > > >> This is exactly what happened with RandomGenerator when it was dropped >> from Spark. If I hadn’t run a test by hand on the cluster it would never >> have shown up in the unit tests. I suspect that this may have led to other >> odd error reports. >> >> Would a script to run all unit tests on a cluster help find out whether we >> have missing classes or not? As I understand it without a job.jar we can’t >> really be sure. >> > > this is probably a good idea, indeed. in fact, i may have introduced some > of those when i transitioned stochastic stuff to mahout random utils > without retesting it in distributed setting. > > But i would think all one'd need is some little mod to standard spark-based > test trait that creates the context in order to check out for something > like TEST_MASTER in the environment and use the $TEST_MASTER master instead > of local if one is found. once that tweak is done, one can easily rerun > unit tests simply by giving > > TEST_MASTER=spark://localhost:7077 mvn test > > (similarly the way master is overriden for shell -- but of course we don't > want tests to react to global MASTER variable just in case it is defined, > so we need aptly named but different one). > Yeah should be simple. I’ll look into this.
