[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. In real test： We can not insert '-01-01 00:00:01' from spark into TIMESTAMP column of mysql currently because of out of range. but we can insert the former '-01-01 00:00:01'into DATETIME column of mysql for `datetime` contains the range of `timestamp` in mysql. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657153038 Hi @srowen , The zone info is according to local zone of mysql, it seems we should not consider it as timezone loss because we have considered it when write. The different between DATETIME and TIMESTAMP is zone index rather than loss it and it belongs to mysql design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657153038 @srowen The zone info is according to local zone of mysql, it seems we should not consider it as timezone loss because we have consider it when writting. The different between DATETIME and TIMESTAMP is zone index rather than loss it and it belongs to mysql design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657153038 @srowen The zone info is according to local zone of mysql, it seems we should not consider it as timezone loss because we have considered it when write. The different between DATETIME and TIMESTAMP is zone index rather than loss it and it belongs to mysql design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657026639 > Yeah DATETIME is definitely not the same thing as TIMESTAMP. You need to start with a Spark date-time type, logically. Hi, @srowen Thank you for your response. The range of date-time type in spark is [0001-01-01, -12-31], which ignores hour, min, sec...，is it what you mean ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. after real test： we can not insert '-01-01 00:00:01' into `timestamp` column of mysql currently because of out of range. but we can insert '-01-01 00:00:01' into `datetime` column of mysql for `datetime` contains the range of `timestamp` in mysql. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. In real test： We can not insert '-01-01 00:00:01' into `timestamp` column of mysql currently because of out of range. but we can insert '-01-01 00:00:01' into `datetime` column of mysql for `datetime` contains the range of `timestamp` in mysql. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. After real test： we can not insert '-01-01 00:00:01' into `timestamp` column of mysql currently because of out of range. but we can insert '-01-01 00:00:01' into `datetime` column of mysql for `datetime` contains the range of `timestamp` in mysql. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. So, we can not insert '-01-01 00:00:01' into `timestamp` column of mysql currently because of out of range. But we can insert '-01-01 00:00:01' into `datetime` column of mysql for `datetime` contains the range of `timestamp` in mysql. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. Finally, we can not insert '-01-01 00:00:01' into timestamp column of mysql currently because of out of range. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. Finally, we can not insert '-01-01 00:00:01' into timestamp column of mysql currently because of out or range. See: https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. Finally, we could not insert '-01-01 00:00:01' into timestamp column of mysql currently because of out or range. See https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657026639 > Yeah DATETIME is definitely not the same thing as TIMESTAMP. You need to start with a Spark date-time type, logically. Hi, @srowen Thank you for your response. The range of date-time type in spark is [0001-01-01, -12-31], which ignore hour, min, sec...，is it what you mean ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range is '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a smaller range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range is '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative date type seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a small range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range is '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative `Datetype` seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a small range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range is '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative `Datetype` seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a small range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. https://dev.mysql.com/doc/refman/5.7/en/datetime.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-657024230 @huaxingao Thank you for your detail response. Actually, `Timestamp` in spark has the range is '0001-01-01T00:00:00.00Z, -12-31T23:59:59.99Z', see `org.apache.spark.sql.types.TimestampType`(8 bytes), and the relative `Datetype` seems should be `datetime` (8 bytes) in mysql, not the timestamp, only 4 bytes and have a small range '1970-01-01 00:00:01.00' to '2038-01-19 03:14:07.99'. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-656959177 @srowen @huaxingao So could we use `datetime` in mysql dialect of spark? The zoneinfo index seems should be considered in mysql side because when we write timestamp to mysql, the zone is considered. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-656959177 @srowen So could we use `datetime` in mysql dialect of spark? The zoneinfo index seems should be considered in mysql side because when we write timestamp to mysql, the zone is considered. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-656959177 @srowen So could we use `datetime` in mysql dialect of spark, the zoneinfo index seems should be considered in mysql side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 edited a comment on pull request #29043: [SPARK-32205][SQL] Writing timestamp to mysql should be datetime type
TJX2014 edited a comment on pull request #29043: URL: https://github.com/apache/spark/pull/29043#issuecomment-656959177 @srowen So could we use `datetime` in mysql dialect, the zoneinfo index seems should be considered in mysql side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
