Hi,

I recently came across some limitations in expressing timestamp type with
Substrait in the Acero substrait consumer and am curious to hear what
people's thoughts are.

The particular issue that I have is when specifying timestamp type in
substrait, the unit is "microseconds" and there is no way to change that.
When integrating with Arrow, often we have timestamps in an internal system
that is of another unit, e.g., a flight service that returns a timestamp in
nanos. Also, interop with pandas, because pandas internally use
nanoseconds, that is another gap.

Currently as a result, we often need to convert from nanos <-> micro when a
substrait plan is involved to specify timestamps. It feels to me as
something missing in substrait but I wonder what other people think.

(Sending this to Arrow mailing list because I know some people here are
pretty involved with substrait and I am more familiar with the folks in the
Arrow community. Therefore wanted to get some thoughts from the people
here).

Li

Reply via email to