Okay my apologies perhaps I’m just not explaining this well. So we have an application, that creates a spark context and adds all the necessary jars, the job is submitted to the cluster and as it runs the code it attempts to pull the data necessary for the job from s3. We use our own version of Presto’s s3 filesystem to be able to read and write the data from s3, but that needs to be initialized with our keys and the s3 bucket before the code can understand how to handle that file. All the Jars and everything get to the slave fine, so it has no problems with ClassNotFound, it just doesn’t seem to be initilizing this s3 file system, so when it tries to download the data from s3 it fails saying it doesn’t know how to handle an s3 URI.
Hopefully that makes sense, any insight? Thanks, Steve On May 13, 2015, at 2:03 PM, Tim Chen <[email protected]<mailto:[email protected]>> wrote: Hi Stephen, I'm not quite sure what you mean by bootstrapping classes, do you have some particular examples? Usually to run any user jar you just need it to be reachable by your slaves so it can be either S3 or any accessible place, then you just provide your jar url when you run spark-submit. Tim On Wed, May 13, 2015 at 8:09 AM, Stephen Carman <[email protected]<mailto:[email protected]>> wrote: Hi, We have a small mesos cluster and we’d like to be able to initialize some of our classes, mostly we have a vfs we setup to be allow our code to access S3, but there doesn’t seem to be any readily obvious way to bootstrap these kind of classes so that they have the properly initialized configuration they need to operate. Is there some accepted way to accomplish this? thanks, Steve This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment. This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment.
