Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9503#issuecomment-155334719
In fact, `httpcore-4.2.4` doesn't even show up in the `-classpath` argument
passed to `ForkMain`.
I used some of the handy classpath-debugging tools in
[JHades](https://github.com/jhades/jhades) to dump more information and it
looks like `httpcore-4.2.4` is included in the Tachyon JARs' `MANIFEST.MF`
files. Here's the manfest from `tachyon-client`:
```
Manifest-Version: 1.0
Built-By: DreamSweet
Build-Jdk: 1.7.0_75
Class-Path: guava-14.0.1.jar commons-lang-2.4.jar commons-io-2.4.jar c
ommons-lang3-3.0.jar curator-framework-2.1.0-incubating.jar zookeeper
-3.4.5.jar slf4j-log4j12-1.7.2.jar log4j-1.2.15.jar mail-1.4.jar acti
vation-1.1.jar jline-0.9.94.jar netty-3.2.2.Final.jar curator-client-
2.1.0-incubating.jar curator-recipes-2.1.0-incubating.jar slf4j-api-1
.7.2.jar hadoop-client-1.0.4.jar hadoop-core-1.0.4.jar xmlenc-0.52.ja
r commons-codec-1.4.jar commons-math-2.1.jar commons-configuration-1.
6.jar commons-collections-3.2.1.jar commons-digester-1.8.jar commons-
beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-net-1.4.
1.jar commons-el-1.0.jar hsqldb-1.8.0.10.jar oro-2.0.8.jar jackson-ma
pper-asl-1.0.1.jar jackson-core-asl-1.0.1.jar tachyon-common-0.8.1.ja
r netty-all-4.0.28.Final.jar libthrift-0.9.2.jar httpclient-4.2.5.jar
httpcore-4.2.4.jar tachyon-client-unshaded-0.8.1.jar tachyon-underfs
-hdfs-0.8.1.jar tachyon-underfs-glusterfs-0.8.1.jar glusterfs-hadoop-
2.3.13.jar hadoop-yarn-server-nodemanager-2.3.0.jar hadoop-yarn-commo
n-2.3.0.jar hadoop-yarn-api-2.3.0.jar jaxb-api-2.2.2.jar stax-api-1.0
-2.jar jettison-1.1.jar servlet-api-2.5.jar jersey-core-1.9.jar jerse
y-client-1.9.jar jetty-util-6.1.26.jar hadoop-annotations-2.3.0.jar g
uice-servlet-3.0.jar protobuf-java-2.5.0.jar guice-3.0.jar javax.inje
ct-1.jar aopalliance-1.0.jar jersey-json-1.9.jar jaxb-impl-2.2.3-1.ja
r jackson-jaxrs-1.8.3.jar jackson-xc-1.8.3.jar jersey-guice-1.9.jar h
adoop-yarn-server-common-2.3.0.jar hadoop-common-2.3.0.jar commons-cl
i-1.2.jar commons-math3-3.1.1.jar jetty-6.1.26.jar jersey-server-1.9.
jar asm-3.1.jar jasper-compiler-5.5.23.jar jasper-runtime-5.5.23.jar
jsp-api-2.1.jar avro-1.7.4.jar paranamer-2.3.jar snappy-java-1.0.4.1.
jar hadoop-auth-2.3.0.jar jsch-0.1.42.jar jsr305-1.3.9.jar commons-co
mpress-1.4.1.jar xz-1.0.jar tachyon-underfs-s3-0.8.1.jar jets3t-0.8.1
.jar commons-logging-1.1.1.jar commons-httpclient-3.1.jar java-xmlbui
lder-0.4.jar tachyon-underfs-local-0.8.1.jar
Created-By: Apache Maven 3.3.3
Archiver-Version: Plexus Archiver
```
According to the [Java 7 JAR
specification](https://docs.oracle.com/javase/7/docs/technotes/guides/jar/jar.html#Manifest_Specification)
(emphasis mine):
> Class-Path: The value of this attribute specifies the **relative** URLs
of the extensions or libraries that this application or extension needs. URLs
are separated by one or more spaces. The application or extension class loader
uses the value of this attribute to construct its internal search path.
Assuming that "relative" means "relative to the JAR containing the
manifest", I think that what's happening is that this is causing the
`httpcore-4.2.4` JAR from `lib_managed` to be added to the classpath, which
explains why testing this patch after rebasing on top of #9575 (which minimizes
our use of `lib_managed/jars` to only include the Datanucleus JARs) seemed to
fix things.
What a giant mess. It looks like the `<addClassPath>` manifest
configuration ended up being added to Tachyon's root POM as part of
https://github.com/amplab/tachyon/pull/939.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
