prashantwason commented on issue #1457: [HUDI-741] Added checks to validate Hoodie's schema evolution. URL: https://github.com/apache/incubator-hudi/pull/1457#issuecomment-612262645 > What will happen if there is incompatible message in Kafka? Will pipeline stall? What will be the way to fix it without purging whole kafka topic? The current state is that: 1. COW tables: - Update to existing parquet file: Will raise as exception during commit as conversion of record to the writerSchema will fail. - Insert to new parquet file: Will be ok. 2. MOR Table: - Update and insert both will be successful. But will raise exception during compaction. I am not very sure on the reader side. Either an exception or the record may be missing the fields. So even today, the pipeline may stall (due to exception). I dont think HUDI has a way out of it yet. You may drop the offending record (before calling HoodieWriteClient::insert()). This change only checks the schema. So if the writerSchema is same, then this code has no extra effect.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
