[
https://issues.apache.org/jira/browse/PHOENIX-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013662#comment-16013662
]
shekhar reddy commented on PHOENIX-3729:
----------------------------------------
Hello Josh Mahonin,
I am integrating Phoenix with spark as blow.
step1. coping below files into /usr/lib/spark/lib
phoenix-core-4.7.0-HBase-1.0.jar
phoenix-server-4.7.0-HBase-1.0.jar
phoenix-spark-4.7.0-HBase-1.0.jar
phoenix-HBase-client.jar
Step2. creating soft links in /usr/lib/spark/lib
phoenix-HBase-client.jar ->
/usr/lib/spark/lib/phoenix-server-client-4.7.0-HBase-1.0.jar
phoenix-spark-HBase.jar -> /usr/lib/spark/lib/phoenix-spark-4.7.0-HBase-1.0.jar
I had integration issue with jackson-core-XXX.jar files. As Spark 2.1 using
2.6.5 version of Jackson apis but Phonenix-Core using 1.9.13. phoneix-core
module pom.xml having <addClasspath>true</addClasspath> so the output of
MANIFEST file has below content and Class-Path configured with older version of
jackson api and start getting exception like some of the methods not
available. I solved the issue by removing <addClasspath>true</addClasspath> and
building Phonenix-Core module then Class-path attribute not generated in
Manifest file so it's started using Spark version of jars in run time.
Manifest file with <addClasspath>true</addClasspath>:
Manifest-Version: 1.0
Implementation-Title: Phoenix Core
Implementation-Version: 4.7.0-HBase-1.0
Archiver-Version: Plexus Archiver
Built-By: mas_user
Specification-Vendor: Apache Software Foundation
Specification-Title: Phoenix Core
Implementation-Vendor-Id: org.apache.phoenix
Class-Path: tephra-api-0.7.0.jar tephra-core-0.7.0.jar gson-2.2.4.jar
guice-3.0.jar javax.inject-1.jar aopalliance-1.0.jar guice-assistedin
ject-3.0.jar libthrift-0.9.0.jar fastutil-6.5.6.jar twill-common-0.6.
0-incubating.jar twill-core-0.6.0-incubating.jar twill-api-0.6.0-incu
bating.jar asm-all-5.0.2.jar twill-discovery-api-0.6.0-incubating.jar
twill-discovery-core-0.6.0-incubating.jar twill-zookeeper-0.6.0-incu
bating.jar metrics-core-3.1.0.jar tephra-hbase-compat-1.0-0.7.0.jar a
ntlr-3.5.jar ST4-4.0.7.jar antlr-runtime-3.5.jar stringtemplate-3.2.1
.jar antlr-2.7.7.jar jline-2.11.jar sqlline-1.1.8.jar guava-13.0.1.ja
r joda-time-1.6.jar annotations-1.3.2.jar jackson-core-asl-1.9.13.jar
jackson-mapper-asl-1.9.13.jar junit-4.12.jar hamcrest-core-1.3.jar p
rotobuf-java-2.5.0.jar httpclient-4.0.1.jar httpcore-4.0.1.jar common
s-logging-1.2.jar log4j-1.2.17.jar slf4j-api-1.6.4.jar snappy-0.3.jar
htrace-core-3.1.0-incubating.jar netty-all-4.0.23.Final.jar commons-
codec-1.7.jar commons-collections-3.2.1.jar commons-csv-1.0.jar jsr30
5-2.0.1.jar slf4j-log4j12-1.7.5.jar hadoop-auth-2.6.0-cdh5.4.8.jar ap
acheds-kerberos-codec-2.0.0-M15.jar apacheds-i18n-2.0.0-M15.jar api-a
sn1-api-1.0.0-M20.jar api-util-1.0.0-M20.jar curator-framework-2.7.1.
jar hadoop-core-2.6.0-mr1-cdh5.4.8.jar hsqldb-1.8.0.10.jar core-3.1.1
.jar hadoop-hdfs-2.6.0-cdh5.4.8.jar commons-daemon-1.0.13.jar leveldb
jni-all-1.8.jar hadoop-hdfs-2.6.0-cdh5.4.8-tests.jar findbugs-annotat
ions-1.3.9-1.jar jersey-client-1.9.jar commons-cli-1.2.jar commons-ma
th-2.1.jar commons-lang-2.6.jar hbase-common-1.0.0-cdh5.4.8.jar hbase
-annotations-1.0.0-cdh5.4.8.jar commons-io-2.4.jar jetty-util-6.1.26.
cloudera.4.jar hbase-protocol-1.0.0-cdh5.4.8.jar hbase-client-1.0.0-c
dh5.4.8.jar netty-3.6.6.Final.jar zookeeper-3.4.5-cdh5.4.8.jar jcodin
gs-1.0.8.jar hbase-server-1.0.0-cdh5.4.8.jar commons-httpclient-3.1.j
ar jersey-core-1.9.jar jersey-server-1.9.jar asm-3.1.jar jetty-6.1.26
.cloudera.4.jar jetty-sslengine-6.1.26.cloudera.4.jar jsp-2.1-6.1.14.
jar jsp-api-2.1-6.1.14.jar servlet-api-2.5-6.1.14.jar jasper-compiler
-5.5.23.jar jasper-runtime-5.5.23.jar jamon-runtime-2.3.1.jar jersey-
json-1.9.jar jettison-1.1.jar jaxb-impl-2.2.3-1.jar jaxb-api-2.1.jar
activation-1.1.jar hbase-server-1.0.0-cdh5.4.8-tests.jar hadoop-commo
n-2.6.0-cdh5.4.8.jar commons-math3-3.1.1.jar xmlenc-0.52.jar commons-
net-3.1.jar servlet-api-2.5.jar jsp-api-2.1.jar commons-el-1.0.jar je
ts3t-0.9.0.jar java-xmlbuilder-0.4.jar commons-configuration-1.6.jar
commons-digester-1.8.jar commons-beanutils-1.7.0.jar commons-beanutil
s-core-1.8.0.jar avro-1.7.6-cdh5.4.8.jar paranamer-2.3.jar jsch-0.1.4
2.jar curator-client-2.7.1.jar curator-recipes-2.7.1.jar htrace-core-
3.0.4.jar commons-compress-1.4.1.jar xz-1.0.jar hadoop-annotations-2.
6.0-cdh5.4.8.jar hadoop-mapreduce-client-core-2.6.0-cdh5.4.8.jar hado
op-yarn-common-2.6.0-cdh5.4.8.jar jersey-guice-1.9.jar guice-servlet-
3.0.jar hadoop-yarn-api-2.6.0-cdh5.4.8.jar joni-2.1.2.jar
Implementation-Vendor: Apache Software Foundation
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_65
Specification-Version: 4.7.0-HBase-1.0
Manifest file without <addClasspath>true</addClasspath>:
Manifest-Version: 1.0
Implementation-Title: Phoenix Core
Implementation-Version: 4.7.0-HBase-1.0
Archiver-Version: Plexus Archiver
Built-By: mas_user
Specification-Vendor: Apache Software Foundation
Specification-Title: Phoenix Core
Implementation-Vendor-Id: org.apache.phoenix
Implementation-Vendor: Apache Software Foundation
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_65
Specification-Version: 4.7.0-HBase-1.0
thanks,
Shekhar Reddy
> Phoenix 4.7 integration issue with Spark 2.1.0
> ----------------------------------------------
>
> Key: PHOENIX-3729
> URL: https://issues.apache.org/jira/browse/PHOENIX-3729
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.7.0, 4.9.0, 4.8.1, 4.10.0, 5.0
> Reporter: shekhar reddy
> Priority: Blocker
>
> I was using phoenix 4.7 with spark 1.6 after I upgraded spark 1.6 to 2.1.0 my
> spark stopped working. I build Phoenix 4.7.0 with spark version 2.1.0 and
> copied latest jars in /usr/lib/spark/jars then started getting Jackson jars
> conflicts as spark 2.1.0 also having different versions of Jackson jars in
> /usr/lb/spark/jars folder.
> I fixed this issue by updating phoenix-core/pom.xml by commenting below line
> and Spark started working!!!!!!!!
> <addClasspath>true</addClasspath>
> Please let me how to push this changes in Phoenix code if you guys feel this
> approach OK.
> I am happy to contribute!!!!!!!!
> Thanks,
> Shekhar Reddy
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
