neerajpadarthi opened a new issue, #5989: URL: https://github.com/apache/hudi/issues/5989
When using Spark-SQL, it is not showing the new fields that are added at the end of the dataset but show up when using spark.read.format(‘org.apache.hudi‘).load(‘<s3-path>‘). I have linked the Glue catalog with the Hive metastore to access data in Athena & Spectrum and it worked as expected. Spark-SQL - New columns are not present. Spark.read - New columns are present. Athena/Spectrum - New columns are present. Q. Do I need to add any configurations? or is this an issue with this V0.9? Thanks in Advance. Environment Description EMR: emr-6.5.0 Hudi version : 0.9 Spark version : Spark 3.1.2 Hive version : Hive 3.1.2 Hadoop version : Storage (HDFS/S3/GCS..) : S3 Running on Docker? (yes/no) : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
