gengliangwang commented on code in PR #40561:
URL: https://github.com/apache/spark/pull/40561#discussion_r1153965249
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -980,3 +1022,65 @@ object StreamingDeduplicateExec {
private val EMPTY_ROW =
UnsafeProjection.create(Array[DataType](NullType)).apply(InternalRow.apply(null))
}
+
+case class StreamingDeduplicateWithinWatermarkExec(
+ keyExpressions: Seq[Attribute],
+ child: SparkPlan,
+ stateInfo: Option[StatefulOperatorStateInfo] = None,
+ eventTimeWatermarkForLateEvents: Option[Long] = None,
+ eventTimeWatermarkForEviction: Option[Long] = None)
+ extends BaseStreamingDeduplicateExec {
+
+ protected val schemaForValueRow: StructType = StructType(
+ Array(StructField("expiresAt", LongType, nullable = false)))
Review Comment:
@HeartSaVioR both TimestampType and TimestampNTZType are based on epoch.
However, for TimestampType, the result will be adjusted based on the SQL conf
spark.sql.session.timeZone.
For example, we stored a timestamp of 2023-03-30 20:00:00 as TimestampType
in Los Angeles time. If users set the spark.sql.session.timeZone as Beijing
time(+08:00), the result will be 2023-03-31 11:00:00.
If we store it as TimestampNTZ type, the read result will always be
2023-03-30 20:00:00
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
