Burak Yavuz created SPARK-20432:
-----------------------------------
Summary: Unioning two identical Streaming DataFrames fails during
attribute resolution
Key: SPARK-20432
URL: https://issues.apache.org/jira/browse/SPARK-20432
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Burak Yavuz
To reproduce, try unioning two identical Kafka Streams:
{code}
df = spark.readStream.format("kafka")... \
.select(from_json(col("value").cast("string"),
simpleSchema).alias("parsed_value"))
df.union(df).writeStream...
{code}
Exception is confusing:
{code}
org.apache.spark.sql.AnalysisException: resolved attribute(s) value#526 missing
from
value#511,topic#512,partition#513,offset#514L,timestampType#516,key#510,timestamp#515
in operator !Project [jsontostructs(...) AS parsed_value#357];
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
