Re: How to use a custom filesystem provider?

2016-09-21 Thread Jean-Philippe Martin
> > There's a bit of confusion setting in here; the FileSystem implementations > spark uses are subclasses of org.apache.hadoop.fs.FileSystem; the nio > class with the same name is different. > grab the google cloud storage connector and put it on your classpath I was using the gs:// filesystem

How to use a custom filesystem provider?

2016-09-21 Thread Jean-Philippe Martin
The full source for my example is available on github <https://github.com/jean-philippe-martin/SparkRepro>. I'm using maven to depend on gcloud-java-nio <https://mvnrepository.com/artifact/com.google.cloud/gcloud-java-nio/0.2.5>, which provides a Java FileSystem for Google Cloud Stor