rchamarthi commented on issue #1977:
URL: https://github.com/apache/hudi/issues/1977#issuecomment-706445482
@kimberlyamandalu I have a version of this working, but with
GlueDevEndpoints, which has pretty similar behavior. Here are the settings I
have.
* Extra jars should have the *full path* of the jar file
```
--extra-jars
"s3://mybucket/jars//hudi-spark-bundle_2.11-0.5.2-incubating.jar"
```
Job configuration should include the following config paramters
```
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf
"spark.sql.hive.convertMetastoreParquet=false"
```
Here's the sample pyspark command that I ran and the format I used is just
"hudi"
```
(inputDF1.write
.format("hudi")
.options(**hudi_options)
.mode("overwrite")
.save(args['target_s3_path']))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
