GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/17901
[SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with
documentation improvement
## What changes were proposed in this pull request?
This PR proposes two things as below:
- Support single argument for `to_timestamp` similarly with APIs in other
languages.
For example, the one below works
```
import org.apache.spark.sql.functions._
Seq("2016-12-31
00:12:00.00").toDF("a").select(to_timestamp(col("a"))).show()
```
prints
```
+----------------------------------------+
|to_timestamp(`a`, 'yyyy-MM-dd HH:mm:ss')|
+----------------------------------------+
| 2016-12-31 00:12:00|
+----------------------------------------+
```
whereas this does not work in SQL.
**Before**
```
spark-sql> SELECT to_timestamp('2016-12-31 00:12:00.00');
Error in query: Invalid number of arguments for function to_timestamp;
line 1 pos 7
```
**After**
```
spark-sql> SELECT to_timestamp('2016-12-31 00:12:00.00');
2016-12-31 00:12:00
```
- Related document improvement for SQL function descriptions and other API
descriptions accordingly.
**Before**
```
spark-sql> DESCRIBE FUNCTION extended to_date;
...
Usage: to_date(date_str, fmt) - Parses the `left` expression with the
`fmt` expression. Returns null with invalid input.
Extended Usage:
Examples:
> SELECT to_date('2016-12-31', 'yyyy-MM-dd');
2016-12-31
```
```
spark-sql> DESCRIBE FUNCTION extended to_timestamp;
...
Usage: to_timestamp(timestamp, fmt) - Parses the `left` expression with
the `format` expression to a timestamp. Returns null with invalid input.
Extended Usage:
Examples:
> SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
2016-12-31 00:00:00.0
```
**After**
```
spark-sql> DESCRIBE FUNCTION extended to_date;
...
Usage:
to_date(date_str[, fmt]) - Parses the `date_str` expression with the
`fmt` expression.
Casts the `date_str` into a date if `fmt` is not given. Returns
null with invalid input.
Extended Usage:
Examples:
> SELECT to_date('2009-07-30 04:17:52');
2009-07-30
> SELECT to_date('2016-12-31', 'yyyy-MM-dd');
2016-12-31
```
```
spark-sql> DESCRIBE FUNCTION extended to_timestamp;
...
Usage:
to_timestamp(timestamp[, fmt]) - Parses the `timestamp` expression
with the `format` expression to
a timestamp. Returns null with invalid input. Default `fmt` is
'yyyy-MM-dd HH:mm:ss'.
Extended Usage:
Examples:
> SELECT to_timestamp('2016-12-31 00:12:00.00');
2016-12-31 00:12:00.0
> SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
2016-12-31 00:00:00.0
```
## How was this patch tested?
Added tests in `datetime.sql`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark to_timestamp_arg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17901.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17901
----
commit 9fcf1071cfe0693b67fb6e6cbec5a1bf808dd873
Author: hyukjinkwon <[email protected]>
Date: 2017-05-08T16:53:53Z
Add single argument support for to_timestamp in SQL with documentation
improvement
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
