[
https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847565#comment-15847565
]
sam elamin commented on SPARK-19414:
------------------------------------
Thanks [~zsxwing] but I dont have access to the dataframe at the
StreamSourceProvider, so how can I set it at that level?
Let me explain, I am calling the source first then setting the schema from the
returned dataframe
At the source level I have the dataframe so can override the schema, but at the
StreamSourceProvider I dont know where to get it from?
> Inferring schema in a structured streaming source
> -------------------------------------------------
>
> Key: SPARK-19414
> URL: https://issues.apache.org/jira/browse/SPARK-19414
> Project: Spark
> Issue Type: Question
> Components: Structured Streaming
> Reporter: sam elamin
>
> Hi All
> I am writing a connector to BigQuery that uses structured streaming, my
> question is about schemas
> I would like to be able to infer the schema from BQ rather than pass it in,
> is there any way to overwrite the source schema in anything that extends
> org.apache.spark.sql.execution.streaming.Source
> Overriding the schema method doesnt seem to work
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
