Thank you for the suggestions, the problem exists in the fact we need to initialize the vfs s3 driver so what you suggested Akhil wouldn’t fix the problem.
Basically a job is submitted to the cluster and it tries to pull down the data from s3, but fails because the s3 uri hasn’t been initilized in the vfs and it doesn’t know how to handle the URI. What I’m asking is, how do we before the job is ran, run some bootstrapping or setup code that will let us do this initialization or configuration step for the vfs so that when it executes the job it has the information it needs to be able to handle the s3 URI. Thanks, Steve On May 13, 2015, at 12:35 PM, jay vyas <jayunit100.apa...@gmail.com<mailto:jayunit100.apa...@gmail.com>> wrote: Might I ask why vfs? I'm new to vfs and not sure wether or not it predates the hadoop file system interfaces (HCFS). After all spark natively supports any HCFS by leveraging the hadoop FileSystem api and class loaders and so on. So simply putting those resources on your classpath should be sufficient to directly connect to s3. By using the sc.hadoopFile (...) commands. On May 13, 2015 12:16 PM, "Akhil Das" <ak...@sigmoidanalytics.com<mailto:ak...@sigmoidanalytics.com>> wrote: Did you happened to have a look at this https://github.com/abashev/vfs-s3 Thanks Best Regards On Tue, May 12, 2015 at 11:33 PM, Stephen Carman <scar...@coldlight.com<mailto:scar...@coldlight.com>> wrote: > We have a small mesos cluster and these slaves need to have a vfs setup on > them so that the slaves can pull down the data they need from S3 when spark > runs. > > There doesn’t seem to be any obvious way online on how to do this or how > easily accomplish this. Does anyone have some best practices or some ideas > about how to accomplish this? > > An example stack trace when a job is ran on the mesos cluster… > > Any idea how to get this going? Like somehow bootstrapping spark on run or > something? > > Thanks, > Steve > > > java.io.IOException: Unsupported scheme s3n for URI s3n://removed > at com.coldlight.ccc.vfs.NeuronPath.toPath(NeuronPath.java:43) > at > com.coldlight.neuron.data.ClquetPartitionedData.makeInputStream(ClquetPartitionedData.java:465) > at > com.coldlight.neuron.data.ClquetPartitionedData.access$200(ClquetPartitionedData.java:42) > at > com.coldlight.neuron.data.ClquetPartitionedData$Iter.<init>(ClquetPartitionedData.java:330) > at > com.coldlight.neuron.data.ClquetPartitionedData.compute(ClquetPartitionedData.java:304) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 15/05/12 13:57:51 ERROR Executor: Exception in task 0.1 in stage 0.0 (TID > 1) > java.lang.RuntimeException: java.io.IOException: Unsupported scheme s3n > for URI s3n://removed > at > com.coldlight.neuron.data.ClquetPartitionedData.compute(ClquetPartitionedData.java:307) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Unsupported scheme s3n for URI > s3n://removed > at com.coldlight.ccc.vfs.NeuronPath.toPath(NeuronPath.java:43) > at > com.coldlight.neuron.data.ClquetPartitionedData.makeInputStream(ClquetPartitionedData.java:465) > at > com.coldlight.neuron.data.ClquetPartitionedData.access$200(ClquetPartitionedData.java:42) > at > com.coldlight.neuron.data.ClquetPartitionedData$Iter.<init>(ClquetPartitionedData.java:330) > at > com.coldlight.neuron.data.ClquetPartitionedData.compute(ClquetPartitionedData.java:304) > ... 8 more > > This e-mail is intended solely for the above-mentioned recipient and it > may contain confidential or privileged information. If you have received it > in error, please notify us immediately and delete the e-mail. You must not > copy, distribute, disclose or take any action in reliance on it. In > addition, the contents of an attachment to this e-mail may contain software > viruses which could damage your own computer system. While ColdLight > Solutions, LLC has taken every reasonable precaution to minimize this risk, > we cannot accept liability for any damage which you sustain as a result of > software viruses. You should perform your own virus checks before opening > the attachment. > This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment.
