nsivabalan commented on issue #6308: URL: https://github.com/apache/hudi/issues/6308#issuecomment-1233264138
yes. we have a config which you can flip to ensure spark streaming fails if hudi write fails. [hoodie.datasource.write.streaming.ignore.failed.batch](https://hudi.apache.org/docs/configurations/#hoodiedatasourcewritestreamingignorefailedbatch) default value for this config is set to true until 2 weeks back. we have flipped the default to false. if you can set this config value to false, atleast you won' see any data loss. if there are any valid errors, your pipeline should fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
