Jackey Lee created SPARK-25937:
----------------------------------
Summary: Support user-defined schema in Kafka Source & Sink
Key: SPARK-25937
URL: https://issues.apache.org/jira/browse/SPARK-25937
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Jackey Lee
Kafka Source & Sink is widely used in Spark and has the highest frequency
in streaming production environment. But at present, both Kafka Source and Link
use the fixed schema, which force user to do data conversion when reading and
writing Kafka. So why not we use fileformat to do this just like hive?
Flink has implemented Kafka's Json/Csv/Avro extended Source & Sink, we can
also support it in Spark.
*Main Goals:*
1. Provide a Source and Sink that support user defined Schema. Users can read
and write Kafka directly in the program without additional data conversion.
2. Provides read-write mechanism based on FileFormat. User's data conversion is
similar to FileFormat's read and write process, we can provide a mechanism
similar to FileFormat, which provide common read-write format conversion. It
also allow users to customize format conversion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
