RajasekarSribalan commented on issue #1823: URL: https://github.com/apache/hudi/issues/1823#issuecomment-657337822
Thank you for your response Bhavani. 1.May I know the purpose of compaction schedule and compaction run command from Hudi CLI? 2. If inline compaction is only possible from spark streaming for MOR tables then it is similar to CopyOnWrite table? There is no difference in using them? Pls correct me if I am wrong. Kind regards, Rajasekar On Mon, 13 Jul 2020, 5:43 am Bhavani Sudha Saktheeswaran, < [email protected]> wrote: > @RajasekarSribalan <https://github.com/RajasekarSribalan> For your first > question, unfortunately currently in Spark Streaming writes only support > inline compaction is supported. So you have to enable that config. Good > news is, this PR <https://github.com/apache/hudi/pull/1752> is working on > enabling the async compaction for Spark Streaming and is in priority. > > For second question, couple clarifications. > > - Hudi moved to Spark 2.4. I see that you are using spark 2.2.0. Could > you try on spark 2.4* ? > - Also, in your spark submit command are you passing in these jars and > cones - https://hudi.apache.org/docs/quick-start-guide.html#setup > > > 1. The conf > 'spark.serializer=org.apache.spark.serializer.KryoSerializer' > 2. in addition to hudi-spark-bundle, you need to pass > ,org.apache.spark:spark-avro_2.11:2.4.4 Note the spark-avro must match > your spark version which is 2.4.4. This applies if you are using > spark-shell as it does not ship with spark-avro explicitly. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/hudi/issues/1823#issuecomment-657297717>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFMO6I26RADG2PR3ZGDG7GDR3JGRVANCNFSM4OXMTUNQ> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
