cloud-fan commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-533090426 Seems java `Period` (defines years, months and days) doesn't match the SQL year-month interval either. The parquet one defines months, days and millisecond, which is also different from SQL standard. The following are my personal thoughts, which is debatable. I don't quite understand why the SQL standard defines 2 interval types. It complicates the type system and seems to me the only benefit is to make interval comparable. I do agree that interval should hold several fields, like months and seconds, as the number of days in a month can vary. I read the comments of java `Period`. It convinces me that 1 day doesn't always mean 24 hours, when day-light saving happens. But I don't quite understand why java `Period` defines both years and months. AFAIK 1 year is always 12 months. The parquet interval looks most reasonable to me. This is also what PostgreSQL does: https://www.postgresql.org/docs/11/datatype-datetime.html > Internally interval values are stored as months, days, and seconds. This is done because the number of days in a month varies, and a day can have 23 or 25 hours if a daylight savings time adjustment is involved. The months and days fields are integers while the seconds field can store fractions. The proposal from me is: 1. keep interval type as it is 2. update `CalendarInterval` to follow parquet/pgsql, and expose it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
