omus commented on issue #367:
URL: https://github.com/apache/arrow-julia/issues/367#issuecomment-1371389919
> - In general, I wonder if it'd be better for a Timestamp's time zone to be
a value rather than "lifted" into the type domain. idk what restrictions are
being imposed on us by the Arrow format itself, though.
Our timestamp implementation of `Timestamp{U, TZ}` where `U` is the unit
precision and `TZ` is the name of the time zone is very closely aligned to the
pyarrow implementation:
https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp.
Looking at the [Arrow specification for a
`Timestamp`](https://github.com/apache/arrow/blob/c6eb4aad30280e1df269f526de057a479c5bc68a/format/Schema.fbs#L233-L352)
also confirms this is the underlying representation. One repercussion of this
to note is you cannot convert a vector of `ZonedDateTime`s with mixed time
zones into the Arrow Timestamp representation.
Due to `ZonedDateTime` contains the time zone name within instances of the
type we cannot define our `ArrowType(::Type{ZonedDateTime})` to return a
concrete type as long as we keep `TZ` as a type parameter in `Timestamp`. We
could remove `TZ` as a type parameter to resolve this but we'd end up running
into issues with our `Timestamp` Julia type appearing that multiple time zones
could be supported in single column.
> One thing we could do to improve the handrolled inference might be to
always generate an Arrow.default for every concrete Julia type present in the
input eltype, and always include those values' types in our starting T before
the promoteunion loop?
Generating the `ToArrow` eltype with `Arrow.default` as the starting `T`
problematic as we cannot support multiple time zones within the same vector.
Always using a default would result in `"UTC"` being the only allowed time
zone. What we can do however is fall back on using `default` when the inferred
eltype is `Missing`. The result of is that `default` is only called when all
elements of the vector are `missing` and the union type contains a non-concrete
type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
