nsivabalan commented on a change in pull request #2923:
URL: https://github.com/apache/hudi/pull/2923#discussion_r637669259
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -485,6 +488,10 @@ private static Object
convertValueForAvroLogicalTypes(Schema fieldSchema, Object
return decimalConversion.fromBytes((ByteBuffer) fieldValue,
fieldSchema,
LogicalTypes.decimal(dc.getPrecision(), dc.getScale()));
}
+ } else if (fieldSchema.getLogicalType() == LogicalTypes.timestampMicros())
{
+ return Instant.EPOCH.plus(Long.parseLong(fieldValue.toString()),
ChronoUnit.MICROS).atZone(ZoneId.systemDefault()).toLocalDateTime();
+ } else if (fieldSchema.getLogicalType() == LogicalTypes.timestampMillis())
{
Review comment:
thanks. looks like we don't have coverage for
LogicalTypes.timestampMicros(). have left a comment below.
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestTimestampBasedKeyGenerator.java
##########
@@ -378,4 +382,46 @@ public void
test_ExpectsMatch_MultipleInputFormats_ShortDate_OutputCustomDate()
baseRow = genericRecordToRow(baseRecord);
assertEquals("04/01/2020", keyGen.getPartitionPath(baseRow));
}
+
+ @Test
+ public void test_ExpectsMatch_LogicalType_Date() throws IOException {
+ LocalDate today = LocalDate.now();
+ baseRecord.put("dob", (int) today.toEpochDay());
+ properties = this.getBaseKeyConfig(
+ "DATE",
+ "yyyy-MM-dd'T'HH:mm:ssZ,yyyy-MM-dd'T'HH:mm:ss.SSSZ,yyyyMMdd",
+ "",
+ null,
+ "yyyy-MM-dd",
+ DateTimeZone.getDefault().getID());
+
+ properties.setProperty(KeyGeneratorOptions.PARTITIONPATH_FIELD_OPT_KEY,
"dob");
+ BuiltinKeyGenerator keyGen = new TimestampBasedKeyGenerator(properties);
+ HoodieKey hk1 = keyGen.getKey(baseRecord);
+ Assertions.assertEquals(today.toString(), hk1.getPartitionPath());
+
+ baseRow = genericRecordToRow(baseRecord);
+ assertEquals(today.toString(), keyGen.getPartitionPath(baseRow));
+ }
+
+ @Test
+ public void test_ExpectsMatch_LogicalType_Timestamp() throws IOException {
Review comment:
thanks for clarifying
##########
File path:
hudi-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroUtils.java
##########
@@ -79,6 +82,18 @@
+ "{\"name\": \"nullable_field\",\"type\": [\"null\"
,\"string\"],\"default\": null},"
+ "{\"name\": \"non_nullable_field_with_default\",\"type\": \"string\",
\"default\": \"dummy\"}]}";
+ private static String SCHEMA_WITH_LOGICAL_TYPES = "{\n"
+ + " \"namespace\": \"example.avro\",\n"
+ + " \"type\": \"record\",\n"
+ + " \"name\": \"User\",\n"
+ + " \"fields\": [\n"
+ + " {\"name\": \"field1\", \"type\": [\"null\", \"string\"],
\"default\": null},\n"
+ + " {\"name\": \"createTime\", \"type\": [\"null\", \"long\"],
\"default\": null},\n"
+ + " {\"name\": \"dob\", \"type\": {\"type\": \"int\"
,\"logicalType\": \"date\"}},\n"
+ + " {\"name\": \"updatedAt\", \"type\": {\"type\": \"long\",
\"logicalType\": \"timestamp-millis\"}}\n"
Review comment:
Can you add a field for timestamp-micros as well and enhance
testGetNestedFieldValForLogicalTypes for the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
