RajasekarSribalan opened a new issue #1814: URL: https://github.com/apache/hudi/issues/1814
Hi, We are trying to run compaction job for MOR tables via hudi cli. Unfortunately , as we use cloudera cluster we have both Spark 1.6 and Spark 2.2.0 in our cluster. So, spark-submit - 1.6 spark2-submit - 2.2.0 Even though we provide the SPARK_HOME(2.2.0), the compaction submits the job in Spark 1.6 but we need the spark compaction to be submitted Spark2-submit i.e., 2.2.0. I understand ,we use Apache's SparkLauncher internally so it submits the jobs with spark-submit but not with spark2-submit. Could you please share the manual spark submit command to run compaction job for MOR tables so that we can schedule it manually? with 1.6 version, compaction job is failing with below error. ERROR utilities.HoodieCompactor: java.lang.NoSuchMethodError: org.eclipse.jetty.util.thread.QueuedThreadPool.<init>(III)V Please provide your support. and do please let us know how we can do compaction for MOR tables. We get 100million records every day for single table with 50:50 insert to upsert ratio. Pls help @vinothchandar @bhasudha ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
