sagarlakshmipathy commented on issue #489: URL: https://github.com/apache/incubator-gluten/issues/489#issuecomment-1987470236
@CodingCat Did you end up writing that doc you mentioned above? I'm trying to run this on AL2, and I'm facing the below error. `java.lang.ClassNotFoundException: org.apache.spark.shuffle.sort.ColumnarShuffleManager` ``` #!/bin/bash # Install git-core sudo yum install -y git-core # Clone the gluten repo git clone https://github.com/oap-project/gluten.git # Install Maven (assuming you have Maven installation steps) # Please follow the link you provided: https://devopscube.com/install-maven-guide/ # Install Maven wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz sudo tar xvf apache-maven-3.9.6-bin.tar.gz -C /opt sudo ln -s /opt/apache-maven-3.9.6 /opt/maven # Update Maven environment variables echo "export M2_HOME=/opt/maven" | sudo tee /etc/profile.d/maven.sh echo "export PATH=\${M2_HOME}/bin:\${PATH}" | sudo tee -a /etc/profile.d/maven.sh sudo chmod +x /etc/profile.d/maven.sh source /etc/profile.d/maven.sh # Copy hudi-benchmarks jar from S3 aws s3 cp s3://s3-calls-log-bucket/hudi-benchmarks/hudi-benchmarks-0.1-SNAPSHOT.jar . # Copy and extract Spark from S3 aws s3 cp s3://s3-calls-log-bucket/hudi-benchmarks/spark-3.3.1-bin-hadoop3.tgz . tar -xvzf spark-3.3.1-bin-hadoop3.tgz # Build gluten with Velox cd gluten mvn package -Pbackends-velox -Pspark-3.3 -Pfull-scala-compiler -DskipTests -Dcheckstyle.skip -Dbuild_cpp=ON -Dbuild_velox=ON -Dbuild_velox_from_source=ON -Dbuild_arrow=ON #export gluten_jar export gluten_jar=/home/hadoop/gluten/backends-velox/target/backends-velox-1.2.0-SNAPSHOT-3.3.jar #export HADOOP_CONF export HADOOP_CONF_DIR=/etc/hadoop/conf #export SPARK_HOME export SPARK_HOME=/home/hadoop/spark-3.3.1-bin-hadoop3 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
