[
https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust resolved SPARK-19788.
--------------------------------------
Resolution: Won't Fix
Thanks for the suggestion. However, as [~zsxwing] said, the goal here is a
small, cross-language compatible API that is the same as the batch version. I
think it totally reasonable for specific source to produce typesafe bindings on
top of this API. (look at spark-avro for an example)
> DataStreamReader/DataStreamWriter.option shall accept user-defined type
> -----------------------------------------------------------------------
>
> Key: SPARK-19788
> URL: https://issues.apache.org/jira/browse/SPARK-19788
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Nan Zhu
>
> There are many other data sources/sinks which has very different
> configuration ways than Kafka, FileSystem, etc.
> The expected type of the configuration entry passed to them might be a nested
> collection type, e.g. Map[String, Map[String, String]], or even a
> user-defined type....(for example, the one I am working on)
> Right now, option can only accept String -> String/Boolean/Long/Double OR a
> complete Map[String, String]...my suggestion is that we can accept
> Map[String, Any], and the type of 'parameters' in SourceProvider.createSource
> can also be Map[String, Any], this will create much more flexibility to the
> user....
> The drawback is that, it is a breaking change ( we can mitigate this by
> deprecating the current one, and progressively evolve to the new one if the
> proposal is accepted)
> [~zsxwing] what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
