codejoyan commented on issue #2852: URL: https://github.com/apache/hudi/issues/2852#issuecomment-824269118
Can we pass the jar when we are creating the Hudi table using Spark datasourcewriter and add the jars while reading using Hive from beeline/hive cli? In that case the inputformatclasses will be visible to the HMS while creating of the table as well as reading? spark-shell \ --packages org.apache.hudi:hudi-spark-bundle_2.11:0.7.0,org.apache.spark:spark-avro_2.11:2.4.4,org.apache.avro:avro:1.8.2 \ --jars /u/users/j0s0j7j/.ivy2/jars/org.apache.avro_avro-1.8.2.jar,**hudi-hadoop-mr-bundle-0.7.0.jar,hudi-common-0.7.0.jar** \ --conf spark.driver.extraClassPath=/u/users/j0s0j7j/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \ --conf spark.executor.extraClassPath=/u/users/j0s0j7j/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \ --conf "spark.sql.hive.convertMetastoreParquet=false" \ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' And while reading from beeline add the jars using the add jars command. add JAR hdfs:///tmp/hudi-hadoop-mr-bundle-0.7.0.jar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
