singhpk234 commented on code in PR #37025:
URL: https://github.com/apache/spark/pull/37025#discussion_r910645370
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/SupportsCatalogOptionsSuite.scala:
##########
@@ -322,6 +323,12 @@ class SupportsCatalogOptionsSuite extends QueryTest with
SharedSparkSession with
timestamp = Some("2019-01-29 00:37:58")), df3.toDF())
checkAnswer(load("t", Some(catalogName), version = None,
timestamp = Some("2021-01-29 00:37:58")), df4.toDF())
+
+ // load with timestamp in number format
+ checkAnswer(load("t", Some(catalogName), version = None,
+ timestamp = Some(MICROSECONDS.toSeconds(ts1).toString)), df3.toDF())
Review Comment:
> I think this over: do we need to support the microsecond? Seems to me it
doesn't make sense for user to specify a long value in the query
+1 on this there is also one risk in supporting this as well, we need to
specify the time always in **seconds** precision only, other wise it can return
timestamp in wrong precision, to the catalog and hence making the query fail .
[CodePointer](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L743)
> For timestamps, only date or timestamp strings should be accepted. I
don't think we should accept a long value.
Are you suggesting we make changes in the grammar for not supporting it in
the SQL ?
I raised this change considering :
1. SQL was supporting this and df options were not (apologies, i thought it
was kind of parity gap)
2. Iceberg supports time travel via custom options in data frame, such as
[`as-of-timestamp`](https://iceberg.apache.org/docs/latest/spark-queries/#time-travel)
which expects the value of option as milliseconds value, since now spark
supports this via SupportCatalogOptions, i thought we can use this.
Happy to make the changes accordingly, or even close it if we think it's not
an issue from df reader perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
