Hi,
It now appears that if we want a clean solution then we have to add a fixed
size primitive type to Beam atomic types. Or, we then have a millis_instant
logical type that does not have to_language_type and to_representation_type
implementations. Any suggestions are welcome!
Best,
Yi
On Thu,
On Wed, Aug 17, 2022 at 5:14 PM Chamikara Jayalath
wrote:
>
> I think this is fine (even though it would add a small perf hit to
> JdbcIO.Read). We also probably should make this conversion a utility method
> that can be used elsewhere when we need to encode datetime fields.
> We should also
On Wed, Aug 17, 2022 at 12:59 PM Yi Hu via dev wrote:
> Hi Brian,
>
> Currently Java's JDBCIO does not do special cross-language things. A
> DATETIME field type appears because the Row contains an Instant object. And
> the sdk will always encode the Instant object using InstantCoder. This is
>
Hi Brian,
Currently Java's JDBCIO does not do special cross-language things. A
DATETIME field type appears because the Row contains an Instant object. And
the sdk will always encode the Instant object using InstantCoder. This is
done for both Java pipelines and cross-language pipelines. To use
Ah sorry, I forgot that INT64 is encoded with VarIntCoder, so we can't
simulate TimestampCoder with a logical type.
I think the ideal end state would be to have a well-defined
beam:logical_type:millis_instant that we use for cross-language (when
appropriate), and never use DATETIME at
Hi Cham,
Thanks for the comments.
>
>>
>> ii. "beam:logical_type:instant:v1" is still backed by INT64, but in
>> implementation it will use BigEndianLongCoder to encode/decode the stream.
>>
>>
> Is this to be compatible with the current Java implementation ? And we
> have to update other SDKs
big endian. Note that this is different from
> (2) because INT64 will be decoded using a VarInt coder which causes data
> corruption when other sdk (e.g. python) pulls rows from java ptransform.
>
> To resolve this I try to implement the "actual" representation of logical
> ty
pulls rows from java ptransform.
To resolve this I try to implement the "actual" representation of logical
type "beam:logical_type:datetime:v1" (
https://github.com/apache/beam/pull/22561 for contexts). It then becomes
necessary to resolve this inconsistency because we have reache
