[jira] [Updated] (SPARK-34357) Map JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone
[ https://issues.apache.org/jira/browse/SPARK-34357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duc Hoa Nguyen updated SPARK-34357: --- Summary: Map JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone (was: Revert JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone) > Map JDBC SQL TIME type to TimestampType with time portion fixed regardless of > timezone > -- > > Key: SPARK-34357 > URL: https://issues.apache.org/jira/browse/SPARK-34357 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Duc Hoa Nguyen >Priority: Minor > > Due to user-experience (confusing to Spark users - java.sql.Time using > milliseconds vs Spark using microseconds; and user losing useful functions > like hour(), minute(), etc on the column), we have decided to revert back to > use TimestampType but this time we will enforce the hour to be consistently > across system timezone (via offset manipulation) > Full Discussion with Wenchen Fan [~cloud_fan] regarding this ticket is here > https://github.com/apache/spark/pull/30902#discussion_r569186823 > Related issues: > [SPARK-33888|https://issues.apache.org/jira/browse/SPARK-33888] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34357) Revert JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone
Duc Hoa Nguyen created SPARK-34357: -- Summary: Revert JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone Key: SPARK-34357 URL: https://issues.apache.org/jira/browse/SPARK-34357 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.2.0 Reporter: Duc Hoa Nguyen Due to user-experience (confusing to Spark users - java.sql.Time using milliseconds vs Spark using microseconds; and user losing useful functions like hour(), minute(), etc on the column), we have decided to revert back to use TimestampType but this time we will enforce the hour to be consistently across system timezone (via offset manipulation) Full Discussion with Wenchen Fan [~cloud_fan] regarding this ticket is here https://github.com/apache/spark/pull/30902#discussion_r569186823 Related issues: [SPARK-33888|https://issues.apache.org/jira/browse/SPARK-33888] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33888) JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257370#comment-17257370
]
Duc Hoa Nguyen commented on SPARK-33888:
Seems like the PR is accepted. Anything else that is needed before this can be
merged?
> JDBC SQL TIME type represents incorrectly as TimestampType, it should be
> physical Int in millis
> ---
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> Currently, for JDBC, SQL TIME type represents incorrectly as Spark
> TimestampType. This should be represent as physical int in millis Represents
> a time of day, with no reference to a particular calendar, time zone or date,
> with a precision of one millisecond. It stores the number of milliseconds
> after midnight, 00:00:00.000.
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Summary: JDBC SQL TIME type represents incorrectly as TimestampType, it
should be physical Int in millis (was: AVRO SchemaConverts - logicalType
TimeMillis not being converted to Timestamp type)
> JDBC SQL TIME type represents incorrectly as TimestampType, it should be
> physical Int in millis
> ---
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> Currently, for JDBC, SQL TIME type represents incorrectly as Spark
> TimestampType. This should be represent as physical int in millis Represents
> a time of day, with no reference to a particular calendar, time zone or date,
> with a precision of one millisecond. It stores the number of milliseconds
> after midnight, 00:00:00.000.
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Description:
Currently, for JDBC, SQL TIME type represents incorrectly as Spark
TimestampType. This should be represent as physical int in millis Represents a
time of day, with no reference to a particular calendar, time zone or date,
with a precision of one millisecond. It stores the number of milliseconds after
midnight, 00:00:00.000.
We encountered the issue of Avro logical type of `TimeMillis` not being
converted correctly to Spark `Timestamp` struct type using the
`SchemaConverters`, but it converts to regular `int` instead. Reproducible by
ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
will get the correct type (Timestamp), but enforcing our avro schema (`{"type":
"int"," logicalType": "time-millis"}`) externally will fail to apply with the
following exception:
{{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
for schema of int}}
was:
We encountered the issue of Avro logical type of `TimeMillis` not being
converted correctly to Spark `Timestamp` struct type using the
`SchemaConverters`, but it converts to regular `int` instead. Reproducible by
ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
will get the correct type (Timestamp), but enforcing our avro schema (`{"type":
"int"," logicalType": "time-millis"}`) externally will fail to apply with the
following exception:
{{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
for schema of int}}
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> Currently, for JDBC, SQL TIME type represents incorrectly as Spark
> TimestampType. This should be represent as physical int in millis Represents
> a time of day, with no reference to a particular calendar, time zone or date,
> with a precision of one millisecond. It stores the number of milliseconds
> after midnight, 00:00:00.000.
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Issue Type: Bug (was: New Feature)
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Issue Type: New Feature (was: Bug)
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253936#comment-17253936
]
Duc Hoa Nguyen commented on SPARK-33888:
I'm making a PR for this at https://github.com/apache/spark/pull/30902 . Let me
figure out the adding test part then I can submit for review.
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Assignee: Apache Spark
>Priority: Minor
>
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Description:
We encountered the issue of Avro logical type of `TimeMillis` not being
converted correctly to Spark `Timestamp` struct type using the
`SchemaConverters`, but it converts to regular `int` instead. Reproducible by
ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
will get the correct type (Timestamp), but enforcing our avro schema (`{"type":
"int"," logicalType": "time-millis"}`) externally will fail to apply with the
following exception:
{{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
for schema of int}}
was:
We encountered the issue of Avro logical type of `TimeMillis` not being
converted correctly to Spark `Timestamp` struct type using the
`SchemaConverters`, but it converts to regular `int` instead. Reproducible by
ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
will get the correct type (Timestamp), but enforcing our avro schema (`{"type":
"int","logicalType": "time-millis"}`) externally will fail to apply with the
following exception:
{{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
for schema of int}}
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Priority: Minor
>
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int"," logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33888) AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp type
[
https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duc Hoa Nguyen updated SPARK-33888:
---
Summary: AVRO SchemaConverts - logicalType TimeMillis not being converted
to Timestamp type (was: java.lang.RuntimeException: java.sql.Timestamp is not
a valid external type for schema of int)
> AVRO SchemaConverts - logicalType TimeMillis not being converted to Timestamp
> type
> --
>
> Key: SPARK-33888
> URL: https://issues.apache.org/jira/browse/SPARK-33888
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3, 3.0.0, 3.0.1
>Reporter: Duc Hoa Nguyen
>Priority: Minor
>
> We encountered the issue of Avro logical type of `TimeMillis` not being
> converted correctly to Spark `Timestamp` struct type using the
> `SchemaConverters`, but it converts to regular `int` instead. Reproducible by
> ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
> will get the correct type (Timestamp), but enforcing our avro schema
> (`{"type": "int","logicalType": "time-millis"}`) externally will fail to
> apply with the following exception:
> {{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
> for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33888) java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for schema of int
Duc Hoa Nguyen created SPARK-33888:
--
Summary: java.lang.RuntimeException: java.sql.Timestamp is not a
valid external type for schema of int
Key: SPARK-33888
URL: https://issues.apache.org/jira/browse/SPARK-33888
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.1, 3.0.0, 2.4.3
Reporter: Duc Hoa Nguyen
We encountered the issue of Avro logical type of `TimeMillis` not being
converted correctly to Spark `Timestamp` struct type using the
`SchemaConverters`, but it converts to regular `int` instead. Reproducible by
ingest data from MySQL table with a column of TIME type: Spark JDBC dataframe
will get the correct type (Timestamp), but enforcing our avro schema (`{"type":
"int","logicalType": "time-millis"}`) externally will fail to apply with the
following exception:
{{java.lang.RuntimeException: java.sql.Timestamp is not a valid external type
for schema of int}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
