Tyler-Rendina commented on issue #10590:
URL: https://github.com/apache/hudi/issues/10590#issuecomment-1986499887
I have the properties set and it works with out of the box hudi versions. I
think it's got something to do with the custom hudi build I created.
Part of my bootstrap script
```
sudo mkdir -p /mnt1/hudi-jars
sudo chown -R $USER /mnt1/hudi-jars
sudo chmod -R ugo+rw /mnt1/hudi-jars
aws s3 cp s3://bucket/jars/hudi-aws-bundle-0.14.1.jar /mnt1/hudi-jars
aws s3 cp s3://bucket/jars/hudi-spark3.3-bundle_2.12-0.14.1.jar
/mnt1/hudi-jars
aws s3 cp s3://bucket/jars/hudi-hive-sync-bundle-0.14.1.jar /mnt1/hudi-jars
sudo ln -sf /mnt1/hudi-jars/hudi-aws-bundle-0.14.1.jar hudi-aws-bundle.jar
sudo ln -sf /mnt1/hudi-jars/hudi-spark3.3-bundle_2.12-0.14.1.jar
hudi-spark3.3-bundle.jar
sudo ln -sf /mnt1/hudi-jars/hudi-hive-sync-bundle-0.14.1.jar
hudi-hive-sync-bundle.jar
PATH=$PATH:/mnt1/hudi-jars/
```
Part of my spark-submit
```
"--jars",
"/mnt1/hudi-jars/hudi-spark3.3-bundle_2.12-0.14.1.jar",
"/mnt1/hudi-jars/hudi-aws-bundle-0.14.1.jar,/mnt1/hudi-jars/hudi-spark3.3-bundle_2.12-0.14.1.jar,/mnt1/hudi-jars/hudi-hive-sync-bundle-0.14.1.jar",
"--conf",
"spark.driver.extraClassPath=/mnt1/hudi-jars/*",
"--conf",
"spark.executor.extraClassPath=/mnt1/hudi-jars/*",
"--packages",
"org.apache.hadoop:hadoop-aws:3.3.3,com.johnsnowlabs.nlp:spark-nlp_2.12:5.1.4",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
