ZhongxuanWang commented on code in PR #12345:
URL: https://github.com/apache/arrow-rs/pull/12345#discussion_r4636232491
##########
parquet/src/record/api.rs:
##########
@@ -922,32 +921,60 @@ impl fmt::Display for Field {
/// Helper method to convert Parquet date into a string.
/// Input `value` is a number of days since the epoch in UTC.
-/// Date is displayed in local timezone.
+/// Date is displayed in UTC timezone.
#[inline]
fn convert_date_to_string(value: i32) -> String {
- static NUM_SECONDS_IN_DAY: i64 = 60 * 60 * 24;
- let dt = Utc
- .timestamp_opt(value as i64 * NUM_SECONDS_IN_DAY, 0)
- .unwrap();
- format!("{}", dt.format("%Y-%m-%d"))
+ format_date(value as i64)
}
/// Helper method to convert Parquet timestamp into a string.
/// Input `value` is a number of milliseconds since the epoch in UTC.
/// Datetime is displayed in UTC timezone.
#[inline]
fn convert_timestamp_millis_to_string(value: i64) -> String {
- let dt = Utc.timestamp_millis_opt(value).unwrap();
- format!("{}", dt.format("%Y-%m-%d %H:%M:%S%.3f %:z"))
+ let (secs, millis) = (value.div_euclid(1_000), value.rem_euclid(1_000));
+ format!("{}.{millis:03} +00:00", format_datetime(secs))
}
/// Helper method to convert Parquet timestamp into a string.
/// Input `value` is a number of microseconds since the epoch in UTC.
/// Datetime is displayed in UTC timezone.
#[inline]
fn convert_timestamp_micros_to_string(value: i64) -> String {
- let dt = Utc.timestamp_micros(value).unwrap();
- format!("{}", dt.format("%Y-%m-%d %H:%M:%S%.6f %:z"))
+ let (secs, micros) = (value.div_euclid(1_000_000),
value.rem_euclid(1_000_000));
+ format!("{}.{micros:06} +00:00", format_datetime(secs))
+}
+
+/// Formats seconds since the epoch as `YYYY-MM-DD hh:mm:ss` in UTC.
+fn format_datetime(secs: i64) -> String {
+ let time = secs.rem_euclid(86_400);
+ format!(
+ "{} {:02}:{:02}:{:02}",
+ format_date(secs.div_euclid(86_400)),
+ time / 3_600,
+ time / 60 % 60,
+ time % 60
+ )
+}
+
+/// Formats days since the epoch as a proleptic Gregorian `YYYY-MM-DD` date.
Years
+/// outside 0..=9999 are signed, as in ISO 8601 and chrono's `%Y`.
+fn format_date(days: i64) -> String {
Review Comment:
Thanks for taking a look!
**Precedent.** The same file already formats `TIME_MILLIS` / `TIME_MICROS`
by hand (`convert_time_millis_to_string` and `convert_time_micros_to_string`,
just below these helpers). For calendar arithmetic, `arrow-cast` folds dates
into the 400-year Gregorian cycle (`era * 146_097`) in `parse_date_to_days` in
`arrow-cast/src/parse.rs`, though it still uses chrono for the date inside the
cycle. I'm not aware of a full days-to-civil conversion elsewhere in the repo.
**Equivalence with chrono 0.4.45** (the version in the lock file):
- Day count to year/month/day: chrono does this in
[`NaiveDate::from_num_days_from_ce_opt`](https://github.com/chronotope/chrono/blob/c4f09c0cf2ae49bdcce11440c197363e73ced067/src/naive/date/mod.rs#L378-L386),
a 400-year cycle plus a lookup table. This PR uses Howard Hinnant's
`civil_from_days`, a different algorithm for the same proleptic Gregorian
calendar, so I checked the two against each other directly rather than by
reading: every day chrono accepts (191,491,529 days), 400 million random
millisecond and microsecond timestamps, and about 40,000 timestamps around the
ends of chrono's range and the epoch all format byte-for-byte the same.
- `%Y`: chrono's
[`write_year`](https://github.com/chronotope/chrono/blob/c4f09c0cf2ae49bdcce11440c197363e73ced067/src/format/formatting.rs#L150-L158)
zero-pads to four digits and writes a sign outside `0..10000`. The new code
does the same, and `test_convert_date_and_timestamp_to_string_extreme_years`
pins those cases (year 0, negative years, year 10000, `NaiveDate::MIN`/`MAX`)
against chrono itself.
- `%.3f` / `%.6f` and `%:z`: fixed-width fractions, and always `+00:00` for
`Utc`.
If you'd rather not carry date code in `parquet`, an alternative is to keep
chrono for these three helpers but behind an optional feature that is on by
default, so `default-features = false` builds still drop it. Happy to switch
the PR to that if you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]