[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535681#comment-16535681
]
Takeshi Yamamuro commented on SPARK-24673:
--
memo: not sure it is worth fixing though, other timestamp/date utility
functions (unit_timestamp, date_format, ) have the same issue;
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Assignee: Antonio Murgia
>Priority: Minor
> Fix For: 2.4.0
>
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534465#comment-16534465
]
Apache Spark commented on SPARK-24673:
--
User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/21723
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Assignee: Antonio Murgia
>Priority: Minor
> Fix For: 2.4.0
>
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530317#comment-16530317
]
Apache Spark commented on SPARK-24673:
--
User 'tmnd1991' has created a pull request for this issue:
https://github.com/apache/spark/pull/21693
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Priority: Minor
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529479#comment-16529479
]
Antonio Murgia commented on SPARK-24673:
I have created a PR, I have added the overload to both functions. Can you have
a look at it? Especially [Xiao
Li|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=smilegator] and
[Takuya
Ueshin|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ueshin] for
the user-facing api matter.
[https://github.com/apache/spark/pull/21693]
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Priority: Minor
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527020#comment-16527020
]
Takeshi Yamamuro commented on SPARK-24673:
--
I'm not 100% sure though, probably I think we cannot touch the existing
signature. So, we need to add an new entry `from_utc_timestamp(ts: Column, tz:
Column)` there.
But, the user-facing api issues are more sensitive, so you need to ask
qualified committers first before making a pr: [~smilegator] [~ueshin]
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Priority: Minor
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526307#comment-16526307
]
Antonio Murgia commented on SPARK-24673:
Looks doable. Should I go with a method overload, resulting in:
{code:java}
functions.from_utc_timestamp(ts: Column, tz: String)
functions.from_utc_timestamp(ts: Column, tz: Column)
{code}
Or is there some limitation I am not aware of?
Also do you think
{code:java}
to_utc_timestamp{code}
should receive the same treatment?
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Priority: Minor
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String
[
https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526118#comment-16526118
]
Takeshi Yamamuro commented on SPARK-24673:
--
It makes sense. Can you make a pr?
> scala sql function from_utc_timestamp second argument could be Column instead
> of String
> ---
>
> Key: SPARK-24673
> URL: https://issues.apache.org/jira/browse/SPARK-24673
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.1
>Reporter: Antonio Murgia
>Priority: Minor
>
> As of 2.3.1 the scala API for the built-in function from_utc_timestamp
> (org.apache.spark.sql.functions#from_utc_timestamp) is less powerful than its
> SQL counter part. In particular, given a dataset/dataframe with the following
> schema:
> {code:java}
> CREATE TABLE MY_TABLE (
> ts TIMESTAMP,
> tz STRING
> ){code}
> from the SQL api I can do something like:
> {code:java}
> SELECT FROM_UTC_TIMESTAMP(TS, TZ){code}
> while from the programmatic api I simply cannot because
> {code:java}
> functions.from_utc_timestamp(ts: Column, tz: String){code}
> second argument is a String.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
