[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-566656502 > date_add/date_sub functions only accept int/tinynit/smallint as the second arg Do you mean time_add/time_sub? BTW we should have a migration guide for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-562142064 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-562024330 looks pretty good, let's see how tests go this time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-561978229 It seems like `UnresolvedBinaryExpression` brings some troubles and may add maintenance overhead. How about this: 1. We still create `Add` in the parser 2. type coercion rules only deal with the normal Add operation, e.g. int + int, interval + interval. 3. the new rule `ResolveBinaryArithmetic` finds the unresolved `Add`, and turn them into `DateAdd`, etc. depending on the data types. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-561703286 The result is unexpected. In `ResolveAlias`, we only generate the alias if expression is resolved. How does pyspark generate alias for its Row object? cc @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type
+/- null should be null as Postgres
URL: https://github.com/apache/spark/pull/26412#issuecomment-561702201
```
File
"/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/dataframe.py",
line 1334, in pyspark.sql.dataframe.DataFrame.selectExpr
Failed example:
df.selectExpr("age * 2", "abs(age)").collect()
Differences (ndiff with -expected +actual):
- [Row((age * 2)=4, abs(age)=2), Row((age * 2)=10, abs(age)=5)]
+ [Row(unresolvedmultiply(age, 2)=4, abs(age)=2),
Row(unresolvedmultiply(age, 2)=10, abs(age)=5)]
**
1 of 1 in pyspark.sql.dataframe.DataFrame.selectExpr
***Test Failed*** 1 failures.
```
There is an actual error
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-561525573 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560222550 I think it's all because we hack the `Add` operator to do date add. Now we need to add more hacks in the type coercion rules. How about we create `UnresolvedAdd` in the parser, and convert it to either `Add` or `DateAdd` in the analyzer? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
