ahmedabu98 commented on code in PR #32688:
URL: https://github.com/apache/beam/pull/32688#discussion_r1792675191
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:
##########
@@ -282,12 +306,24 @@ private static void copyFieldIntoRecord(Record rec,
Types.NestedField field, Row
Optional.ofNullable(value.getDouble(name)).ifPresent(v ->
rec.setField(name, v));
break;
case DATE:
- throw new UnsupportedOperationException("Date fields not yet
supported");
+ Optional.ofNullable(value.getLogicalTypeValue(name, LocalDate.class))
+ .ifPresent(v -> rec.setField(name, v));
+ break;
case TIME:
- throw new UnsupportedOperationException("Time fields not yet
supported");
+ Optional.ofNullable(value.getLogicalTypeValue(name, LocalTime.class))
+ .ifPresent(v -> rec.setField(name, v));
+ break;
case TIMESTAMP:
- Optional.ofNullable(value.getDateTime(name))
- .ifPresent(v -> rec.setField(name, v.getMillis()));
+ Object val = value.getValue(name);
+ if (val instanceof Instant) { // case Schema.FieldType.DATETIME
+ rec.setField(name, DateTimeUtil.timestampFromMicros(((Instant)
val).getMillis()));
Review Comment:
Great catch! and thanks for all the info and references.
I've been digging for some time and found that Beam has two field types that
represent timestamps:
`SqlTypes.DATETIME` -- uses java.time.LocalDateTime
`FieldType.DATETIME` -- uses org.joda.time.DateTime
DateTime supports timezone information, but the native Beam Schema doesn't.
When this object gets processed in a Beam Row, the timezone information is
dropped. Fetching the object will always return a timestamp from UTC.
Because of this, I don't think the connect can support timestamp with TZ yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
