This problem appears to be a regression on HEAD/master: when running against 2.0.2 the pyspark job completes successfully including running predictions.
2016-11-23 19:36 GMT-08:00 Stephen Boesch <java...@gmail.com>: > > For a pyspark job with 54 executors all of the task outputs have a single > line in both the stderr and stdout similar to: > > Error: invalid log directory > /shared/sparkmaven/work/app-20161119222540-0000/0/ > > > Note: the directory /shared/sparkmaven/work exists and is owned by the > same user running the job. There are plenty of other app-*** subdirectories > that do have contents in the stdout/stderr files. > > > $ls -lrta /shared/sparkmaven/work > total 0 > drwxr-xr-x 59 steve staff 2006 Nov 23 05:01 .. > drwxr-xr-x 41 steve staff 1394 Nov 23 18:22 app-20161123050122-0002 > drwxr-xr-x 6 steve staff 204 Nov 23 18:22 app-20161123182031-0005 > drwxr-xr-x 6 steve staff 204 Nov 23 18:44 app-20161123184349-0006 > drwxr-xr-x 6 steve staff 204 Nov 23 18:46 app-20161123184613-0007 > drwxr-xr-x 3 steve staff 102 Nov 23 19:20 app-20161123192048-0008 > > > > Here is a sample of the contents > > /shared/sparkmaven/work/app-20161123184613-0007/2: > total 16 > -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout > drwxr-xr-x 4 steve staff 136 Nov 23 18:46 . > -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr > drwxr-xr-x 6 steve staff 204 Nov 23 18:46 .. > > /shared/sparkmaven/work/app-20161123184613-0007/3: > total 16 > -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout > drwxr-xr-x 6 steve staff 204 Nov 23 18:46 .. > drwxr-xr-x 4 steve staff 136 Nov 23 18:46 . > -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr > > > Note also: the *SparkPI* program does run succesfully - which validates > the basic spark installation/functionality. > >
