Thanks for the excellent summary everyone. I agree with these
summaries that have been pointed out. It seems like things are moving
towards consensus.
> I think Instant is what is represented as Arrow's Timestamp with Timezone.
> I don't think Arrow has a type for DateTime because we don't have any type
> that allows for different time zones per slot in an Array (which I think
> would be the expectation for DateTime).
I'm not sure it's fair to call Arrow's Timestamp with Timezone an
Instant. An Instant is limited, it cannot have fields extracted from
it, etc.. Arrow's "Timestamp with Timezone" can have fields extracted
from it. Technically the types Instant / DateTime (ZonedDateTime) /
LocalDateTime are all scalar monikers so it's unclear that requiring
the timestamp be consistent across a column would disqualify you from
calling it DateTime / ZonedDateTime. Maybe instead something like
FixedZonedDateTime and VariableZonedDateTime in the same way "list"
becomes FixedList or VariableList when you store a column of them.
As it seems we are approaching consensus I think we should clarify the
docs. First, the schema.fbs should probably be changed is to change
"This data can be though of as UTC" to something like "This data will
be treated as UTC" and that libraries will need to convert a datetime
to UTC before storing it in this field. For example, if I read a list
of integers from some JSON file in python and I knew they were Unix
timestamps in seconds my naive assumption would be to do
pa.array(timestamp, type=pa.timestamp('s')) and that would be a bad
idea.
However, it seems like there is a lot of information in these two
email chains that is not going to easily fit into schema.fbs. I think
it'd be worthwhile to create some kind of standalone FAQ page for
working with temporal data. I'll take a stab at this today before I
forget everything.
On Tue, Jun 15, 2021 at 6:38 AM Micah Kornfield <emkornfi...@gmail.com> wrote:
>
> Actually after some thought
>
> I think Instant is what is represented as Arrow's Timestamp with Timezone.
> I don't think Arrow has a type for DateTime because we don't have any type
> that allows for different time zones per slot in an Array (which I think
> would be the expectation for DateTime).
>
> On Tue, Jun 15, 2021 at 9:23 AM Micah Kornfield <emkornfi...@gmail.com>
> wrote:
>
> > My interpretation has always been:
> >
> > > * Instant - an instantaneous point on the time-line
> > > * DateTime - full date and time with time-zone
> >
> > These two do not have distinct types and are both handled via timestamp
> > with a timezone.
> >
> > > * LocalDateTime - date-time without a time-zone
> >
> > Arrow Timestamp without timezone (although as noted above the
> > representation we've used makes this the cause for debate).
> >
> > I guess the alternative would be to have a first class LocalDateTime type.
> >
> >
> >
> > On Tue, Jun 15, 2021 at 9:14 AM Antoine Pitrou <anto...@python.org> wrote:
> >
> >>
> >> Le 15/06/2021 à 16:53, Adam Hooper a écrit :
> >> > - *"Datetime"* lets you extract fields, parse strings, format to
> >> string.
> >> > You can't sort (because clocks sometimes go backwards). You can't
> >> convert
> >> > between timestamps and future datetimes (because timezones change).
> >>
> >> Not true if the timezone is UTC, though.
> >> (which is a strong argument, IMHO, for representing all values in the
> >> UTC reference, regardless of any optional "timezone" information)
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >
