KnightChess commented on code in PR #11637:
URL: https://github.com/apache/hudi/pull/11637#discussion_r1682093503
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor
generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema
schema, boolean shouldSanitize, String invalidCharMask) {
+ // the value can be Number
+ if (value instanceof Number) {
+ LogicalTypes.Decimal decimalType = (LogicalTypes.Decimal)
schema.getLogicalType();
+ BigDecimal bigDecimal = new
java.math.BigDecimal(value.toString()).setScale(decimalType.getScale(),
RoundingMode.HALF_UP);
+ GenericFixed genericFixed = DECIMAL_CONVERSION.toFixed(bigDecimal,
schema, schema.getLogicalType());
+ return Pair.of(true, new GenericData.Fixed(schema,
genericFixed.bytes()));
+ }
+
// The ObjectMapper use List to represent FixedType
// eg: "decimal_val": [0, 0, 14, -63, -52] will convert to
ArrayList<Integer>
List<Integer> converval = (List<Integer>) value;
Review Comment:
thanks, can you provide a JSON example? and add it to ut for make ut more
complete
##########
hudi-common/src/test/java/org/apache/hudi/avro/TestMercifulJsonConverter.java:
##########
@@ -99,4 +106,44 @@ public void conversionWithFieldNameAliases() throws
IOException {
Assertions.assertEquals(rec, CONVERTER.convert(json, sanitizedSchema));
}
+
+ @Test
+ public void testConvertNumberToFixed() throws IOException {
+ String testSchemaStr = "{\"type\": \"record\",\"name\":
\"test_record\",\"namespace\": \"test_namespace\",\"fields\": "
+ + "[{\"name\": \"decimal_field\",\"type\": [\"null\",{\"type\":
\"fixed\",\"name\": \"fixed\",\"namespace\":
\"test_namespace.decimal_field\",\"size\": 9,"
+ + " \"logicalType\": \"decimal\",\"precision\": 20,\"scale\":
0}],\"default\": null},"
+ + "{\"name\": \"decimal2_field\",\"type\": [\"null\",{\"type\":
\"fixed\",\"name\": \"fixed\",\"namespace\":
\"test_namespace.decimal2_field\",\"size\": 9,"
+ + " \"logicalType\": \"decimal\",\"precision\": 20,\"scale\":
2}],\"default\": null},"
+ + "{\"name\": \"decimal3_field\",\"type\": [\"null\",{\"type\":
\"fixed\",\"name\": \"fixed\",\"namespace\":
\"test_namespace.decimal3_field\",\"size\": 9,"
+ + " \"logicalType\": \"decimal\",\"precision\": 20,\"scale\":
2}],\"default\": null},"
+ + "{\"name\": \"int_field\",\"type\": [\"null\",\"int\"],\"default\":
null},"
+ + "{\"name\": \"long_field\",\"type\":
[\"null\",\"long\"],\"default\": null},"
+ + "{\"name\": \"string_field\",\"type\":
[\"null\",\"string\"],\"default\": null}]}";
+ Schema schema = Schema.parse(testSchemaStr);
+
+ String testValueStr = "{\n"
+ + " \"decimal_field\": 1720623716,\n"
Review Comment:
the decimal type is `logicalType`, data type is Fixed, which is bytes, the
logical type `decimal` I think in fixed is used to conver `bytes` to `decimal`,
not conver `number` to `decimal`.
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor
generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema
schema, boolean shouldSanitize, String invalidCharMask) {
+ // the value can be Number
+ if (value instanceof Number) {
Review Comment:
yes, so I think you avro schema `decimal_field` should not be `fixed`,
should be float or double
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
