GitHub user dongjinleekr opened a pull request:
https://github.com/apache/spark/pull/22282
[SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
## What changes were proposed in this pull request?
This update adds support for Kafka Headers functionality in Structured
Streaming.
## How was this patch tested?
With following unit tests:
- KafkaRelationSuite: "default starting and ending offsets with headers"
(new)
- KafkaSinkSuite: "batch - write to kafka" (updated)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjinleekr/spark feature/SPARK-23539
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22282.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22282
----
commit ddd08612ef8bdb173f974059d2dc6311e1c7d9a3
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-26T13:19:52Z
Remove unused methods from UnsafeArrayData
commit 2af13899ab052cc7b52c25b57f154b78a2c45b2a
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-26T13:25:25Z
Implement UnsafeArrayData#fromBinaryArray
commit a8e5c5c0f478a795af1236771236da2074093f3e
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-27T12:22:18Z
Implement UnsafeArrayData#fromStringArray
commit 2ca181046cf1102aed14f4957e11e4dd901ba3c7
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-27T13:28:58Z
Implement UnsafeMapData#of
commit d0d746d99d0a19ecbb2dc098589adbfd1ef0b5ae
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-29T13:25:57Z
Allow empty UnsafeArrayData: does not throw IllegalArgumentException on
empty or null array argument anymore.
commit b459cf3f391d6e4ee9cb77a7b5ed510d027d9ddd
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-28T07:50:59Z
Fix invalid formatting: UnsafeArraySuite
commit f077c5d75a83df3541a95a628726e1d74af8c153
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-28T07:51:30Z
Implemenet kafka headers functionality
commit 6b4d7754d01e4211d05c83746c848fdfd873f229
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-29T09:39:55Z
Add KafkaTestUtils#{sendMessage, sendMessages(String, Array[(String,
String, Array[(String, String)])], Option[Int])}
commit c7fb9819989056da4910e2cfc81af332cd603d41
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-29T13:24:09Z
Extend KafkaRelationSuite, KafkaSinkSuite to test headers functionality
commit dd2d9390478e4c69b01a7c699e28bfe923ef0db1
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-30T10:50:38Z
Minor refinements
commit 229aac85442b03736fc850cae2c3b26becaedade
Author: Lee Dongjin <dongjin@...>
Date: 2018-08-30T12:21:30Z
Specify #selectExpr on KafkaSourceSuiteBase's 'Kafka column types' test
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
