amanchaudhary-95 commented on issue #9992:
URL: https://github.com/apache/seatunnel/issues/9992#issuecomment-3535977159
@chl-wxp @davidzollo
Yes, I would like Kafka to auto-generate field structures when used as a
source, especially for sensor/IoT data coming in raw JSON format.
In many streaming setups, the incoming JSON payload contains dynamic fields
or evolving schemas, and having SeaTunnel automatically infer the structure
(similar to schema-on-read) would significantly simplify the pipeline
configuration. Currently, the JSON parser does not auto-detect the schema from
Kafka messages, which means the entire schema must be manually defined in the
`schema = {}` block for every field. This becomes challenging when dealing with
large and frequently changing sensor payloads.
It would be very helpful if SeaTunnel could:
* Automatically parse JSON messages
* Infer fields and their data types
* Create the internal schema at runtime
* Optionally allow overriding or refining inferred fields
This feature would be especially valuable for IoT use cases where thousands
of sensors publish diverse JSON messages through Kafka, where some fields may
be **Float**, some **Int**, some **Boolean**, and others **String**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
