waitingF commented on code in PR #11637:
URL: https://github.com/apache/hudi/pull/11637#discussion_r1682828247
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor
generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema
schema, boolean shouldSanitize, String invalidCharMask) {
+ // the value can be Number
+ if (value instanceof Number) {
Review Comment:
Yeah, in my case, the decimal(20, 0) is for bigint and decimal(20, 2) for
double, So the debezium decode as I expected.
And the hudi JdbcSchemaProvider convert the mysql table schema as expected
to Decimal(20, 0) and Decimal(20, 2).
But when hudi try decoding the bigint and double field from kafka, it throw
exception which is unexpected.
the source table schema like
```
CREATE TABLE `xxxx` (
`push_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '',
`plan_id` bigint(20) unsigned NOT NULL COMMENT '',
`to_station_id` bigint(20) unsigned NOT NULL,
`plan_status` tinyint(3) NOT NULL,
`api_type` tinyint(3) NOT NULL COMMENT ''
PRIMARY KEY (`push_id`),
)
```
tranlated avro schema like
```
{
"type": "record",
"name": "sample_table",
"namespace": "sample_namespace",
"fields": [
{
"name": "push_id",
"type": [
"null",
{
"type": "fixed",
"name": "fixed",
"namespace": "sample_namespace.sample_table.push_id",
"size": 9,
"logicalType": "decimal",
"precision": 20,
"scale": 0
}
],
"default": null
},
{
"name": "plan_id",
"type": [
"null",
{
"type": "fixed",
"name": "fixed",
"namespace": "sample_namespace.sample_table.plan_id",
"size": 9,
"logicalType": "decimal",
"precision": 20,
"scale": 0
}
],
"default": null
},
{
"name": "to_station_id",
"type": [
"null",
{
"type": "fixed",
"name": "fixed",
"namespace":
"sample_namespace.sample_table.to_station_id",
"size": 9,
"logicalType": "decimal",
"precision": 20,
"scale": 0
}
],
"default": null
},
{
"name": "plan_status",
"type": [
"null",
"int"
],
"default": null
},
{
"name": "api_type",
"type": [
"null",
"int"
],
"default": null
}
]
}
```
The bigint(20) is tranlated to Decimal(20, 0), but the data in kafka is just
long, not the list of Integer, so the json convert throw below exception.
```
Caused by: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to
convert schema from json to avro:
{"_event":{"old":null,"type":"insert","ts_ms":1720447722000,"version":"1.9.5.Final","connector":"mysql","name":"debezium.xx","ts":1720447722,"snapshot":"false","database":"xxx","sequence":null,"table":"xxx","server_id":241149,"gtid":"xxx","file":"mysql-bin.000416","position":161676701,"row":0,"thread":1459349,"query":null},"push_id":11193065,"plan_id":8928834333,"to_station_id":2028,"plan_status":5,"api_type":4}
at
org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:121)
at
org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at
org.apache.hudi.utilities.schema.LazyCastingIterator.computeNext(LazyCastingIterator.java:40)
at
org.apache.hudi.utilities.schema.LazyCastingIterator.computeNext(LazyCastingIterator.java:30)
at
org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 18 more
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.util.List
at
org.apache.hudi.avro.MercifulJsonConverter$9.convert(MercifulJsonConverter.java:306)
at
org.apache.hudi.avro.MercifulJsonConverter$JsonToAvroFieldProcessor.convertToAvro(MercifulJsonConverter.java:203)
at
org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvroField(MercifulJsonConverter.java:192)
at
org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvro(MercifulJsonConverter.java:134)
at
org.apache.hudi.avro.MercifulJsonConverter.convert(MercifulJsonConverter.java:115)
at
org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:118)
... 24 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
