nsivabalan commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834436516
@sathyaprakashg and others: trying to understand the use-case here. I understand its related to deltastreamer receiving events in old schema after Hudi's dataset schema got evolved. what's the schema from schema provider when source is producing events in old schema (after schema got evolved w/ hudi dataset)? if the schema provider's schema is updated, I guess there is no need to store the writer schema w/ payload. AvroConversionUtils.createDataFrame() will ensure to convert the JavaRDD<GenRec> w/ old schema to Dataset<Row> w/ new schema if schemaProvider.SourceSchema() has the evolved schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
