[
https://issues.apache.org/jira/browse/SPARK-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942250#comment-14942250
]
Sean Owen commented on SPARK-10846:
-----------------------------------
I looked a little more and yes this is easy to reproduce. I verified that Java
/ ServiceLoader doesn't by nature examine the working directory. So, something
is causing the search to include the current working directory. But my
driver/executor classpath in a simple local run don't include the cwd on the
classpath.
Of course the workaround is just to not do that, not have META-INF lying
around. Still I do not know how the cwd ends up on the classpath in this
situation.
> Stray META-INF in directory spark-shell is launched from causes problems
> ------------------------------------------------------------------------
>
> Key: SPARK-10846
> URL: https://issues.apache.org/jira/browse/SPARK-10846
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 1.5.0
> Reporter: Ryan Williams
> Priority: Minor
>
> I observed some perplexing errors while running
> {{$SPARK_HOME/bin/spark-shell}} yesterday (with {{$SPARK_HOME}} pointing at a
> clean 1.5.0 install):
> {code}
> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
> Provider org.apache.hadoop.fs.s3.S3FileSystem not found
> {code}
> while initializing {{HiveContext}}; full example output is
> [here|https://gist.github.com/ryan-williams/34210ad640687113e5c3#file-1-5-0-failure].
> The issue was that a stray {{META-INF}} directory from some other project I'd
> built months ago was sitting in the directory that I'd run {{spark-shell}}
> from (*not* in my {{$SPARK_HOME}}, just in the directory I happened to be in
> when I ran {{$SPARK_HOME/bin/spark-shell}}).
> That {{META-INF}} had a {{services/org.apache.hadoop.fs.FileSystem}} file
> specifying some provider classes ({{S3FileSystem}} in the example above) that
> were unsurprisingly not resolvable by Spark.
> I'm not sure if this is purely my fault for attempting to run Spark from a
> directory with another project's config files laying around, but I find it
> somewhat surprising that, given a {{$SPARK_HOME}} pointing to a clean Spark
> install, that {{$SPARK_HOME/bin/spark-shell}} picks up detritus from the
> {{cwd}} it is called from, so I wanted to at least document it here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
