[
https://issues.apache.org/jira/browse/SPARK-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980377#comment-15980377
]
Apache Spark commented on SPARK-20432:
--------------------------------------
User 'lw-lin' has created a pull request for this issue:
https://github.com/apache/spark/pull/17735
> Unioning two identical Streaming DataFrames fails during attribute resolution
> -----------------------------------------------------------------------------
>
> Key: SPARK-20432
> URL: https://issues.apache.org/jira/browse/SPARK-20432
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Burak Yavuz
>
> To reproduce, try unioning two identical Kafka Streams:
> {code}
> df = spark.readStream.format("kafka")... \
> .select(from_json(col("value").cast("string"),
> simpleSchema).alias("parsed_value"))
> df.union(df).writeStream...
> {code}
> Exception is confusing:
> {code}
> org.apache.spark.sql.AnalysisException: resolved attribute(s) value#526
> missing from
> value#511,topic#512,partition#513,offset#514L,timestampType#516,key#510,timestamp#515
> in operator !Project [jsontostructs(...) AS parsed_value#357];
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
