zuyanton commented on issue #1790: URL: https://github.com/apache/hudi/issues/1790#issuecomment-656228790
I was able to confirm that master branch fixed the issue however not without difficulties. More specifically: When I deploy master branch to EMR and run select query in spark-shell, I get ```NoSuchMethodException HiveDecimalWritable.enforcePrecisionScale``` at [this line](https://github.com/apache/hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L213). I tried to lookup where spark shell grabs that class definition via ``` scala> classOf[org.apache.hadoop.hive.serde2.io.HiveDecimalWritable].getProtectionDomain res3: java.security.ProtectionDomain = ProtectionDomain (file:/usr/lib/spark/jars/hive-exec-1.2.1-spark2-amzn-1.jar <no signer certificates>) sun.misc.Launcher$AppClassLoader@36f6e879 <no principals> java.security.Permissions@49e335a5 ( ("java.io.FilePermission" "/usr/lib/spark/jars/hive-exec-1.2.1-spark2-amzn-1.jar" "read") ("java.lang.RuntimePermission" "exitVM") ) ``` and it looks like it comes from outdated hive-exec-1.2.1-spark2-amzn-1.jar which indeed does not have "enforcePrecisionScale" method. I tried uploading hive-storage-api (dependency of hive-exec and actual package containing HiveDecimalWritable definition) to the cluster and referencing it to spark-shell via ```--jar /some_temp_location/hive-storage-api-2.3.1.jar``` - didn't help. The only thing that worked for me is adding hive-storage-api-2.3.1.jar directly to ```/usr/lib/spark/jars``` folder, which does not seems right to me. @garyli1019 have you seen similar issues with [your change](https://github.com/apache/hudi/commit/37838cea6094ddc66191df42e8b2c84f132d1623#diff-68b6e6f1a2c961fea254a2fc3b93ac23R213) ? is it specific to AWS EMR Spark ? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
