srowen commented on a change in pull request #22282: [SPARK-23539][SS] Add
support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#discussion_r294976671
##########
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousStream.scala
##########
@@ -38,19 +38,21 @@ import org.apache.spark.sql.sources.v2.reader.streaming._
* read by per-task consumers generated later.
* @param kafkaParams String params for per-task Kafka consumers.
* @param sourceOptions Params which are not Kafka consumer params.
- * @param metadataPath Path to a directory this reader can use for writing
metadata.
+ * @param metadataPath Path to a directory this reader can use for writing
metadata.
* @param initialOffsets The Kafka offsets to start reading data at.
* @param failOnDataLoss Flag indicating whether reading should fail in data
loss
* scenarios, where some offsets after the specified
initial ones can't be
* properly read.
+ * @param includeHeaders Flag indicating whether to include Kafka records'
headers.
*/
class KafkaContinuousStream(
offsetReader: KafkaOffsetReader,
kafkaParams: ju.Map[String, Object],
sourceOptions: Map[String, String],
metadataPath: String,
initialOffsets: KafkaOffsetRangeLimit,
- failOnDataLoss: Boolean)
+ failOnDataLoss: Boolean,
+ includeHeaders: Boolean)
Review comment:
I think we might need to retain the previous constructor's signature for
compatibility?
But could this just be an option in sourceOptions? that sort of seems to be
what it's for. I haven't checked, but, it might avoid plumbing through this
boolean all over the place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
