[
https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847581#comment-15847581
]
sam elamin commented on SPARK-19414:
------------------------------------
Ah thanks for clarifying!
Thats a bit of a shame, ideally id like to read a stream and infer the schema
from it, but I guess spark.read.json isnt quite the same as
spark.readStream.json since one is under the sql namespace!
my solution so far is to read one record from BigQuery, infer the schema there
then pass it down to the readStream method
Its a bit convoluted to be honest so if there is a cleaner or nicer way other
than hardcoding the schema then happy to hear it!
> Inferring schema in a structured streaming source
> -------------------------------------------------
>
> Key: SPARK-19414
> URL: https://issues.apache.org/jira/browse/SPARK-19414
> Project: Spark
> Issue Type: Question
> Components: Structured Streaming
> Reporter: sam elamin
>
> Hi All
> I am writing a connector to BigQuery that uses structured streaming, my
> question is about schemas
> I would like to be able to infer the schema from BQ rather than pass it in,
> is there any way to overwrite the source schema in anything that extends
> org.apache.spark.sql.execution.streaming.Source
> Overriding the schema method doesnt seem to work
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
