[
https://issues.apache.org/jira/browse/SPARK-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984114#comment-13984114
]
Guoqiang Li commented on SPARK-1644:
------------------------------------
{code}
# When Hive support is needed, Datanucleus jars must be included on the
classpath.
# Datanucleus jars do not work if only included in the uber jar as plugin.xml
metadata is lost.
# Both sbt and maven will populate "lib_managed/jars/" with the datanucleus
jars when Spark is
# built with Hive, so first check if the datanucleus jars exist, and then
ensure the current Spark
# assembly is built for Hive, before actually populating the CLASSPATH with the
jars.
# Note that this check order is faster (by up to half a second) in the case
where Hive is not used.
num_datanucleus_jars=$(ls "$FWDIR"/lib_managed/jars/ 2>/dev/null | grep
"datanucleus-.*\\.jar" | wc -l)
if [ $num_datanucleus_jars -gt 0 ]; then
AN_ASSEMBLY_JAR=${ASSEMBLY_JAR:-$DEPS_ASSEMBLY_JAR}
num_hive_files=$(jar tvf "$AN_ASSEMBLY_JAR" org/apache/hadoop/hive/ql/exec
2>/dev/null | wc -l)
if [ $num_hive_files -gt 0 ]; then
echo "Spark assembly has been built with Hive, including Datanucleus jars
on classpath" 1>&2
DATANUCLEUSJARS=$(echo "$FWDIR/lib_managed/jars"/datanucleus-*.jar | tr " "
:)
CLASSPATH=$CLASSPATH:$DATANUCLEUSJARS
fi
fi
{code}
only add /lib_managed/jars of files to the CLASSPATH. In the current directory
is dist is unable to work
> hql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") throw an
> exception
> ---------------------------------------------------------------------------------
>
> Key: SPARK-1644
> URL: https://issues.apache.org/jira/browse/SPARK-1644
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Guoqiang Li
> Assignee: Guoqiang Li
> Attachments: spark.log
>
>
> cat conf/hive-site.xml
> {code:xml}
> <configuration>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:postgresql://bj-java-hugedata1:7432/hive</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>org.postgresql.Driver</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>passwd</value>
> </property>
> <property>
> <name>hive.metastore.local</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>hdfs://host:8020/user/hive/warehouse</value>
> </property>
> </configuration>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
