B Anil Kumar created DRILL-5976:
-----------------------------------
Summary: Kafka MessageReader config optimization
Key: DRILL-5976
URL: https://issues.apache.org/jira/browse/DRILL-5976
Project: Apache Drill
Issue Type: Improvement
Reporter: B Anil Kumar
Assignee: B Anil Kumar
Kafka storage plugin currently supports JSON message format, but going forward
it will support Avro etc.
Also there might be multiple scenarios where user have to implement their own
MessageReader (Corresponding their custom Deserializer/Decoder).
So, this JIRA is to brainstorm on whether to go with MessageReader as a *Plugin
config* OR system/session option.
Paul's suggestion as part review comment is as below.
{noformat}
Suppose this is two or three releases from now and we support other forms of
Kafka messages. Different topics use different formats.
If the message format is a system/session option, then I need to switch the
option before each query. Very cumbersome and error prone.
Instead, perhaps this information should be part of the storage plugin config.
Then, I can define different plugins: one for each message format.
Further, can I have multiple Kafka servers? If so, would I need different
plugin configs for each?
So, should we be thinking about encoding most properties as plugin config
properties?
Now, the plugin might have a format property, one of which is json. The JSON
config properties would be defined in the json format within the overall
storage plugin config.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
