Dongjoon Hyun created SPARK-27624:
-------------------------------------
Summary: Fix CalenderInterval to show an empty interval correctly
Key: SPARK-27624
URL: https://issues.apache.org/jira/browse/SPARK-27624
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.2, 2.3.3, 3.0.0
Reporter: Dongjoon Hyun
*BEFORE*
{code}
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
== Physical Plan ==
EventTimeWatermark ts#0: timestamp, interval 1 microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#0]
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
== Physical Plan ==
EventTimeWatermark ts#3: timestamp, interval
+- StreamingRelation FileSource[/tmp/t], [ts#3]
{code}
*AFTER*
{code}
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
== Physical Plan ==
EventTimeWatermark ts#0: timestamp, interval 1 microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#0]
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
== Physical Plan ==
EventTimeWatermark ts#3: timestamp, interval 0 microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#3]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
