ranjanankur opened a new issue, #9355: URL: https://github.com/apache/hudi/issues/9355
**Problem while reading from BQ tables which are synced on Hudi table** We have one streaming application where we are getting data from Kafka. We are applying transformation using Spark Structure Streaming and then loading the data to **Hudi tables**(GCS bucket). After creating hudi tables we are syncing the same Hudi table with **GCP BigQuery** using BigQuerySync tool. Now we are facing we are very weird problem and if our understanding of these bug is correct then we might have to remove hudi from every usecase from our team at Walmart. Let me explain the problem. BigQuery sync tools internally is creating two BQ tables and one View. 1. Manifest table 2. Version table 3. BQ view - This contain latest UPSERTED data. For maintaining latest UPSERT. Hudi is creating one _**latest-snapshot.csv.**_ in the **gs://<GCP bucket>/<folder>/.hoodie/manifest/latest-snapshot.csv** After every microbatch Hudi is actually replacing this file. It is deleting and recreating this file. Our batch duration of streaming pipeline is 5 min. So after 5 min, new file will be created with the same name. Now the problem is let suppose, I am calling **` BQ view`** and apply some join or some operation with other tables which takes more than 5 min to run then our BQ job throw error because latest-snapshot.csv is changed during this. Error - Not found .hoodie/manifest/latest-snapshot.csv (Version 1690978310289362) **To Reproduce** Steps to reproduce the behavior: 1. Create streaming pipeline 3. Load into Hudi table 4. Sync Hudi table with BigQuery using BigQuerySync tool 5. Fire a query which uses BQ view created due to BigQuerySync and query should take more than the batch duration of streaming pipeline **Expected behavior** When I run some BiqQuery Statement then it should run without error. **Environment Description** * Hudi version : 0.13.1 * Spark version : 3.1.3 * Hive version : 3.1.3 * Storage (HDFS/S3/GCS..) : GCP * Running on Docker? (yes/no) : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
