In PySpark 2.0.0, despite adding snappy and lzo to my spark.jars path, I get errors that say these classes can't be found when I save to a parquet file. I tried switching from default snappy to lzo and added that jar and I get the same error.
What am I to do? I can't figure out any other steps to take. Note that this bug appeared when I upgraded from Spark 1.6 to 2.0.0. Other than spark.jars, what steps can I take to give Parquet access to its jars? Why isn't PySpark just handling this, since Parquet is included in Spark? Is this a bug? A gist of my code and the error is here: https://gist.github.com/rjurney/6783d19397cf3b4b88af3603d6e256bd -- Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io