dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-514697187 Here is the fix. Yes, you are right - [The official documentation states that 'Note that multiple headers may have the same key.'](https://kafka.apache.org/23/javadoc/org/apache/kafka/connect/header/Headers.html) `ArrayType` is more reasonable than `MapType`. So I updated the schema and related routines. Another significant modification in this update is removing `KafkaRecordToUnsafeRowConverter`. After reviewing the code, I found that using `UnsafeProjection#create` instead of `UnsafeRowWriter` is the general way - Besides, there is no way to instantiate `UnsafaArrayData` from an array of structs. I also updated the documentation accordingly. Please have a look when you are free. Here are some additional questions I hope to ask: 1. As of present, object `KafkaOffsetReader` has no relationship with class `KafkaOffsetReader`; should we change its name? 2. Do we need some Kafka-specific sql function like `Headers#retainLatest` or `Headers#lastWithName`? cc/ @srowen @HeartSaVioR
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
