dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-463650960 @zsxwing @HeartSaVioR Thank you for the comments. Here are some explanations on compatibility issues: ### 1. About the old brokers (e.g., Kafka 0.10.x broker) which doesn't support `headers.` Kafka's consumer already handles this case: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecord.java#L86 If the given record was forwarded by an old broker without `headers` functionality. As @HeartSaVioR pointed out, we are already using 2.1.0 client, so we don't have to worry about this. ### 2. About `KafkaOffsetReader#kafkaSchema` modification As all of you know, this PR adds the 8th field, `headers,` to `KafkaOffsetReader#kafkaSchema.` For continuous reading, it won't be problematic - this field will be ignored with the old SQL queries. In my opinion, the same holds for continuous writing. Older writing queries without the `headers` column will not conflict with the update, and the queries with `headers` can't be valid. +1. > I wonder if there has been some cases on datasource option to be allowed to affect schema. (by @HeartSaVioR) As far as I know, there is no datasource option which affects the schema.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
