We need to figure out a way to make sure we can read the data without losing precision. It will likely be case by case.
-Rui On Wed, Aug 28, 2019 at 2:27 AM Alex Van Boxel <a...@vanboxel.be> wrote: > Thanks, how will ZetaSQL support higher precision as the input in general > will be Instant anyway. Will it rely on the "pending" standardized logical > types? > > _/ > _/ Alex Van Boxel > > > On Mon, Aug 19, 2019 at 7:02 AM Rui Wang <ruw...@google.com> wrote: > >> However, more challengings come from: >> >> 1. How to read data without losing precision. Beam Java SDK uses Joda >> already so it's very likely that you will need update IO somehow to support >> higher precision. >> 2. How to process higher precision in BeamSQL. It means SQL functions >> should support higher precision. If you use Beam Calcite, unfortunately it >> will only support up to millis. If you use Beam ZetaSQL (under review), >> there are opportunities to support higher precision for SQL functions. >> >> >> -Rui >> >> On Sun, Aug 18, 2019 at 9:52 PM Rui Wang <ruw...@google.com> wrote: >> >>> We have been discussing it for a long time. I think if you only want to >>> support more precision (e.g. up to nanosecond) for BeamSQL, it's actually >>> relatively straightforward to support it by using a logical type for >>> BeamSQL. >>> >>> >>> -Rui >>> >>> On Sat, Aug 17, 2019 at 7:21 AM Alex Van Boxel <a...@vanboxel.be> wrote: >>> >>>> I know it's probably futile, but the more I'm working on features that >>>> are related to schema awareness I'm getting a bit frustrated about the lack >>>> of precision of the joda instance. >>>> >>>> As soon as we have a conversion to the DateTime I need to drop >>>> precession, this happens with the Protobuf timestamp (nanoseconds), but I >>>> also notice it with BigQuery (milliseconds). >>>> >>>> Suggestions? >>>> >>>> _/ >>>> _/ Alex Van Boxel >>>> >>>
