alberttwong commented on issue #10697: URL: https://github.com/apache/hudi/issues/10697#issuecomment-1957155089
I would also add that since the parquet was compressed with zippy, this is why spark had a tough time with processing the file. You can either 1. increase memory or 2. break up the parquet file into smaller files. Also had to run driver memory with a lot of memory. executor memory was not needed since running spark locally. ``` spark-shell --packages org.apache.hudi:hudi-spark$SPARK_VERSION-bundle_2.12:0.14.1 --driver-memory 24G ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
