[
https://issues.apache.org/jira/browse/SPARK-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-19926:
---------------------------------
Labels: (was: bulk-closed)
> Make pyspark exception more readable
> ------------------------------------
>
> Key: SPARK-19926
> URL: https://issues.apache.org/jira/browse/SPARK-19926
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.0.2, 2.1.0
> Reporter: Genmao Yu
> Assignee: Xianjin YE
> Priority: Minor
>
> Exception in pyspark is a little difficult to read.
> like:
> {code}
> Traceback (most recent call last):
> File "<stdin>", line 5, in <module>
> File "/root/dev/spark/dist/python/pyspark/sql/streaming.py", line 853, in
> start
> return self._sq(self._jwrite.start())
> File
> "/root/dev/spark/dist/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py",
> line 1133, in __call__
> File "/root/dev/spark/dist/python/pyspark/sql/utils.py", line 69, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.AnalysisException: u'Append output mode not supported when
> there are streaming aggregations on streaming DataFrames/DataSets without
> watermark;;\nAggregate [window#17, word#5], [window#17 AS window#11, word#5,
> count(1) AS count#16L]\n+- Filter ((t#6 >= window#17.start) && (t#6 <
> window#17.end))\n +- Expand [ArrayBuffer(named_struct(start,
> ((((CEIL((cast((precisetimestamp(t#6) - 0) as double) / cast(30000000 as
> double))) + cast(0 as bigint)) - cast(1 as bigint)) * 30000000) + 0), end,
> (((((CEIL((cast((precisetimestamp(t#6) - 0) as double) / cast(30000000 as
> double))) + cast(0 as bigint)) - cast(1 as bigint)) * 30000000) + 0) +
> 30000000)), word#5, t#6-T30000ms), ArrayBuffer(named_struct(start,
> ((((CEIL((cast((precisetimestamp(t#6) - 0) as double) / cast(30000000 as
> double))) + cast(1 as bigint)) - cast(1 as bigint)) * 30000000) + 0), end,
> (((((CEIL((cast((precisetimestamp(t#6) - 0) as double) / cast(30000000 as
> double))) + cast(1 as bigint)) - cast(1 as bigint)) * 30000000) + 0) +
> 30000000)), word#5, t#6-T30000ms)], [window#17, word#5, t#6-T30000ms]\n
> +- EventTimeWatermark t#6: timestamp, interval 30 seconds\n +-
> Project [cast(word#0 as string) AS word#5, cast(t#1 as timestamp) AS t#6]\n
> +- StreamingRelation
> DataSource(org.apache.spark.sql.SparkSession@c4079ca,csv,List(),Some(StructType(StructField(word,StringType,true),
> StructField(t,IntegerType,true))),List(),None,Map(sep -> ;, path ->
> /tmp/data),None), FileSource[/tmp/data], [word#0, t#1]\n'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
