Nan Zhu created SPARK-19788:
-------------------------------
Summary: DataStreamReader.option shall accept user-defined type
Key: SPARK-19788
URL: https://issues.apache.org/jira/browse/SPARK-19788
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Nan Zhu
There are many other data sources which has very different configuration ways
than Kafka, FileSystem, etc.
The expected type of the configuration entry passed to them might be a nested
collection type, e.g. Map[String, Map[String, String]], or even a user-defined
type....(for example, the one I am working on)
Right now, option can only accept String -> String/Boolean/Long/Double OR a
complete Map[String, String]...my suggestion is that we can accept Map[String,
Any], and the type of 'parameters' in SourceProvider.createSource can also be
Map[String, Any], this will create much more flexibility to the user....
The drawback is that, it is a breaking change ( we can mitigate this by
deprecate the current one, and progressively evolve to the new one if the
proposal is accepted)
[~zsxwing] what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
