Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@ckadner Had updated the PR to fallback to java.sql.Timestamp to do the
reverse lookup, that should have better resolution (especially when converting
a timestamp to another timezone).
The test you posted still failed on 52 cases, because the equality of
java.sql.Date is based on unix timestamp (in UTC), but it could be multiple
timestamp mapping to same human time, so the equality of java.sql.Date could
have false negative.
```
52 incorrect conversions 1900..2020
1939-11-19 in Africa/Algiers
1900-01-01 in Africa/Ceuta
1922-01-01 in America/Cancun
1965-10-31 in America/Dawson
1965-10-31 in America/Inuvik
1999-10-31 in America/Iqaluit
1922-01-01 in America/Mexico_City
1999-10-31 in America/Pangnirtung
1950-04-16 in America/Santarem
1965-10-31 in America/Whitehorse
1900-01-01 in Antarctica/Casey
1900-01-01 in Antarctica/Davis
2009-10-18 in Antarctica/Davis
2011-10-28 in Antarctica/Davis
1900-01-01 in Antarctica/DumontDUrville
1900-01-01 in Antarctica/Mawson
1900-01-01 in Antarctica/Syowa
1900-01-01 in Antarctica/Vostok
1916-11-01 in Atlantic/Azores
1916-11-01 in Atlantic/Madeira
1917-10-21 in Atlantic/Reykjavik
1918-11-16 in Atlantic/Reykjavik
1919-11-16 in Atlantic/Reykjavik
1921-06-23 in Atlantic/Reykjavik
1965-10-31 in Canada/Yukon
1900-01-01 in Europe/Brussels
1900-01-01 in Europe/Gibraltar
1916-10-01 in Europe/Kaliningrad
1916-10-01 in Europe/Kaliningrad
1922-10-08 in Europe/Luxembourg
1924-10-05 in Europe/Luxembourg
1925-10-04 in Europe/Luxembourg
1926-10-03 in Europe/Luxembourg
1927-10-02 in Europe/Luxembourg
1928-10-07 in Europe/Luxembourg
1900-01-01 in Europe/Madrid
1992-09-27 in Europe/Minsk
1911-03-11 in Europe/Paris
1990-07-01 in Europe/Uzhgorod
1990-07-01 in Europe/Uzhgorod
1917-10-21 in Iceland
1918-11-16 in Iceland
1919-11-16 in Iceland
1921-06-23 in Iceland
1900-01-01 in Indian/Kerguelen
1922-01-01 in Mexico/General
1900-01-01 in Pacific/Apia
1900-01-01 in Pacific/Enderbury
1900-01-01 in Pacific/Fakaofo
1900-01-01 in Pacific/Kiritimati
1911-03-11 in ECT
1900-01-01 in MIT
```
For 52 cases here, I checked that the Date objects are actually point to
the start of the day (create a Timestamp objects using the unix timestamp of
the two Date objects).
```
(1939-11-19 00:00:00.0,1939-11-19 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1922-01-01 00:00:00.0,1922-01-01 00:00:00.0)
(1965-10-31 00:00:00.0,1965-10-31 00:00:00.0)
(1965-10-31 00:00:00.0,1965-10-31 00:00:00.0)
(1999-10-31 00:00:00.0,1999-10-31 00:00:00.0)
(1922-01-01 00:00:00.0,1922-01-01 00:00:00.0)
(1999-10-31 00:00:00.0,1999-10-31 00:00:00.0)
(1950-04-16 00:00:00.0,1950-04-16 00:00:00.0)
(1965-10-31 00:00:00.0,1965-10-31 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(2009-10-18 00:00:00.0,2009-10-18 00:00:00.0)
(2011-10-28 00:00:00.0,2011-10-28 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1916-11-01 00:00:00.0,1916-11-01 00:00:00.0)
(1916-11-01 00:00:00.0,1916-11-01 00:00:00.0)
(1917-10-21 00:00:00.0,1917-10-21 00:00:00.0)
(1918-11-16 00:00:00.0,1918-11-16 00:00:00.0)
(1919-11-16 00:00:00.0,1919-11-16 00:00:00.0)
(1921-06-23 00:00:00.0,1921-06-23 00:00:00.0)
(1965-10-31 00:00:00.0,1965-10-31 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1916-10-01 00:00:00.0,1916-10-01 00:00:00.0)
(1916-10-01 00:00:00.0,1916-10-01 00:00:00.0)
(1922-10-08 00:00:00.0,1922-10-08 00:00:00.0)
(1924-10-05 00:00:00.0,1924-10-05 00:00:00.0)
(1925-10-04 00:00:00.0,1925-10-04 00:00:00.0)
(1926-10-03 00:00:00.0,1926-10-03 00:00:00.0)
(1927-10-02 00:00:00.0,1927-10-02 00:00:00.0)
(1928-10-07 00:00:00.0,1928-10-07 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1992-09-27 00:00:00.0,1992-09-27 00:00:00.0)
(1911-03-11 00:00:00.0,1911-03-11 00:00:00.0)
(1990-07-01 00:00:00.0,1990-07-01 00:00:00.0)
(1990-07-01 00:00:00.0,1990-07-01 00:00:00.0)
(1917-10-21 00:00:00.0,1917-10-21 00:00:00.0)
(1918-11-16 00:00:00.0,1918-11-16 00:00:00.0)
(1919-11-16 00:00:00.0,1919-11-16 00:00:00.0)
(1921-06-23 00:00:00.0,1921-06-23 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1922-01-01 00:00:00.0,1922-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
(1911-03-11 00:00:00.0,1911-03-11 00:00:00.0)
(1900-01-01 00:00:00.0,1900-01-01 00:00:00.0)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
