yihua commented on code in PR #13711:
URL: https://github.com/apache/hudi/pull/13711#discussion_r2352833905
##########
hudi-common/src/main/java/org/apache/hudi/avro/processors/Parser.java:
##########
@@ -46,21 +48,19 @@ public Pair<Boolean, Object> handleStringValue(String
value) {
public static class DateParser extends Parser {
- private static final long MILLI_SECONDS_PER_DAY = 86400000;
-
@Override
public Pair<Boolean, Object> handleNumberValue(Number value) {
- return Pair.of(true, new java.sql.Date(value.intValue() *
MILLI_SECONDS_PER_DAY));
+ return Pair.of(true, LocalDate.ofEpochDay(value.intValue()));
Review Comment:
What's the motivation behind this change? I see this is used in Json to Row
converter and I'm wondering if this can change the data to be written to a Hudi
table.
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1150,12 +1151,33 @@ public static Object rewritePrimaryType(Object
oldValue, Schema oldSchema, Schem
case NULL:
case BOOLEAN:
case INT:
- case LONG:
case FLOAT:
case DOUBLE:
case BYTES:
case STRING:
return oldValue;
+ case LONG:
+ if (oldSchema.getLogicalType() != newSchema.getLogicalType()) {
+ if (oldSchema.getLogicalType() instanceof
LogicalTypes.TimestampMillis) {
+ if (newSchema.getLogicalType() instanceof
LogicalTypes.TimestampMicros) {
+ return DateTimeUtils.millisToMicros((Long) oldValue);
+ }
+ } else if (oldSchema.getLogicalType() instanceof
LogicalTypes.TimestampMicros) {
+ if (newSchema.getLogicalType() instanceof
LogicalTypes.TimestampMillis) {
+ return DateTimeUtils.microsToMillis((Long) oldValue);
+ }
+ } else if (oldSchema.getLogicalType() instanceof
LogicalTypes.LocalTimestampMillis) {
+ if (newSchema.getLogicalType() instanceof
LogicalTypes.LocalTimestampMicros) {
+ return DateTimeUtils.millisToMicros((Long) oldValue);
+ }
+ } else if (oldSchema.getLogicalType() instanceof
LogicalTypes.LocalTimestampMicros) {
+ if (newSchema.getLogicalType() instanceof
LogicalTypes.LocalTimestampMillis) {
+ return DateTimeUtils.microsToMillis((Long) oldValue);
+ }
+ }
Review Comment:
Do we have tests around schema evolution for these cases (not a blocking
issue to resolve for this PR)?
##########
hudi-common/src/main/java/org/apache/hudi/avro/ConvertingGenericData.java:
##########
@@ -128,7 +125,8 @@ public boolean validate(Schema schema, Object datum) {
case LONG:
return isLong(datum)
|| TIME_MICROS_CONVERSION.getConvertedType().isInstance(datum)
- ||
TIMESTAMP_MICROS_CONVERSION.getConvertedType().isInstance(datum);
+ || TIMESTAMP_MICROS_CONVERSION.getConvertedType().isInstance(datum)
+ ||
LOCAL_TIMESTAMP_MICROS_CONVERSION.getConvertedType().isInstance(datum);
Review Comment:
Should `TIME_MILLIS_CONVERSION`, `LOCAL_TIMESTAMP_MILLIS_CONVERSION`, and
`TIMESTAMP_MILLIS_CONVERSION` also be handled or they are handled somewhere
else?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedTableMetadata.java:
##########
@@ -350,7 +375,7 @@ public void testMetadataRecordKeyExcludeFromPayload(final
HoodieTableType tableT
* plan has not been successfully executed before the new one is scheduled.
*/
@ParameterizedTest
- @CsvSource({"COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6",
"MERGE_ON_READ,8"})
+ @MethodSource("testMetadataTableKeyGeneratorArgs")
Review Comment:
How much is the increased time for running all tests compared to before?
Would like to make sure the time to finish these tests is still reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
