Thanks Donald and Mars,
I created a new distribution (
<https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
with the added CLASSPATH code and pointed to the distribution with
the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
echo "$CLASSPATH"
It didn't seem to force the aws-java-sdk to load first as I reviewed the
release logs. Should the aws-java-sdk.jar show up as the first file within
the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
lib/spark/aws-java-sdk.jar:$CLASSPATH".
I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have other
suggestions to try? I was also testing locally to change the order of the
--jars but changes to the compute-classpath.sh didn't seem to change the
order of the jars in the logs.
Running train on release…
Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
[INFO] [Runner$] Submission command:
/app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit
--driver-memory 13g --class
org.apache.predictionio.workflow.CreateWorkflow --jars
file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar*
--files
file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml
--driver-class-path
/app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf
--driver-java-options -Dpio.log.dir=/app
file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar
--engine-id org.template.liftscoring.LiftScoringEngine
--engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20
--engine-variant file:/app/engine.json --verbosity 0 --json-extractor
Both --env
Error:
Exception in thread "main" java.lang.NoSuchMethodError:
com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
*Shane Johnson | LIFT IQ*
*Founder | CEO*
*www.liftiq.com <http://www.liftiq.com/>* or *[email protected]
<[email protected]>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter
<https://twitter.com/SWaldenJ> | Facebook
<https://www.facebook.com/shane.johnson.71653>
On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <[email protected]> wrote:
> Shane,
>
> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <[email protected]> wrote:
>
>>
>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>> referring to...(line at the bottom in red)?
>>
>
>
> I believe the code would need to look like this to effect the output
> classpath as intended:
>
>
>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>> echo "$CLASSPATH"
>>
>
>
> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
> need to be skip or remove it first.
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Figure out where PredictionIO is installed
FWDIR="$(cd `dirname $0`/..; pwd)"
. ${FWDIR}/bin/load-pio-env.sh
if [ -n "$JAVA_HOME" ]; then
JAR_CMD="$JAVA_HOME/bin/jar"
else
JAR_CMD="jar"
fi
# Use pio-assembly JAR from either RELEASE or assembly directory
if [ -f "${FWDIR}/RELEASE" ]; then
assembly_folder="${FWDIR}"/lib
else
assembly_folder="${FWDIR}"/assembly/src/universal/lib
fi
MAIN_JAR=$(ls "${assembly_folder}"/pio-assembly*.jar 2>/dev/null)
DATA_JARS=$(ls "${assembly_folder}"/spark/pio-data-*assembly*.jar 2>/dev/null)
# Comma-separated list of assembly jars for submitting to spark-shell
ASSEMBLY_JARS=$(printf "${MAIN_JAR}\n${DATA_JARS}" | paste -sd "," -)
# Build up classpath
CLASSPATH="${PIO_CONF_DIR}"
# stable classpath for plugin JARs
if [ -d "${FWDIR}/plugins" ]; then
lib_plugin_jars=`ls "${FWDIR}"/plugins/*`
lib_plugin_classpath=''
for J in $lib_plugin_jars; do
lib_plugin_classpath="${lib_plugin_classpath}:${J}"
done
CLASSPATH="$CLASSPATH${lib_plugin_classpath}"
fi
# stable classpath for Spark JARs
lib_spark_jars=`ls "${assembly_folder}"/spark/*.jar`
lib_spark_classpath=''
for J in $lib_spark_jars; do
lib_spark_classpath="${lib_spark_classpath}:${J}"
done
CLASSPATH="$CLASSPATH${lib_spark_classpath}"
CLASSPATH="$CLASSPATH:${MAIN_JAR}"
# Add hadoop conf dir if given -- otherwise FileSystem.*, etc fail ! Note, this
# assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
# the configurtion files.
if [ -n "$HADOOP_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$HADOOP_CONF_DIR"
fi
if [ -n "$YARN_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$YARN_CONF_DIR"
fi
if [ -n "$HBASE_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$HBASE_CONF_DIR"
fi
if [ -n "$ES_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$ES_CONF_DIR"
fi
if [ -n "$POSTGRES_JDBC_DRIVER" ]; then
CLASSPATH="$CLASSPATH:$POSTGRES_JDBC_DRIVER"
ASSEMBLY_JARS="$ASSEMBLY_JARS,$POSTGRES_JDBC_DRIVER"
fi
if [ -n "$MYSQL_JDBC_DRIVER" ]; then
CLASSPATH="$CLASSPATH:$MYSQL_JDBC_DRIVER"
ASSEMBLY_JARS="$ASSEMBLY_JARS,$MYSQL_JDBC_DRIVER"
fi
CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
echo "$CLASSPATH"
