LiLiu88 commented on issue #1823: URL: https://github.com/apache/hudi/issues/1823#issuecomment-874558679
> @RajasekarSribalan For your first question, unfortunately currently in Spark Streaming writes only support inline compaction is supported. So you have to enable that config. Good news is, this [PR](https://github.com/apache/hudi/pull/1752) is working on enabling the async compaction for Spark Streaming and is in priority. > > For second question, couple clarifications. > > * Hudi moved to Spark 2.4. I see that you are using spark 2.2.0. Could you try on spark 2.4* ? > * Also, in your spark submit command are you passing in these jars and cones - https://hudi.apache.org/docs/quick-start-guide.html#setup > > 1. The conf `'spark.serializer=org.apache.spark.serializer.KryoSerializer'` > 2. in addition to hudi-spark-bundle, you need to pass `,org.apache.spark:spark-avro_2.11:2.4.4` Note the spark-avro must match your spark version which is 2.4.4. This applies if you are using spark-shell as it does not ship with spark-avro explicitly. can we disable compaction in sink ,and run compaction using hudi-cli manully? both inline or async all disable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
