TengHuo commented on issue #7284:
URL: https://github.com/apache/hudi/issues/7284#issuecomment-1327268261
The exception mentioned above can be triggered with this test case
```scala
test("Test Insert Into MOR table 2") {
withTempDir { tmp =>
val tableName = generateTableName
// Create a partitioned table
spark.sql(
s"""
|create table $tableName (
| id int,
| dt string,
| name string,
| price double,
| ts long,
| test_decimal_col decimal(25, 4)
|) using hudi
|options
|(
| type = 'mor'
| ,primaryKey = 'id'
| ,hoodie.index.type = 'INMEMORY'
|)
| tblproperties (primaryKey = 'id')
| partitioned by (dt)
| location '${tmp.getCanonicalPath}'
| """.stripMargin)
// Note: Do not write the field alias, the partition field must be
placed last.
spark.sql(
s"""
| insert into $tableName values
| (1, 'a1', 10, 1000, 1.0, "2021-01-05"),
| (2, 'a2', 20, 2000, 2.0, "2021-01-06"),
| (3, 'a3', 30, 3000, 3.0, "2021-01-07")
""".stripMargin)
spark.sql(
s"""
| insert into $tableName values
| (1, 'a1', 10, null, 1.0, "2021-01-05"),
| (4, 'a2', 20, 2000, null, "2021-01-06"),
| (5, 'a3', 30, 3000, 3.0, "2021-01-07")
""".stripMargin)
spark.sql(s"select id, name, price, ts, cast(test_decimal_col AS
string), dt from $tableName").show(false)
}
}
```
The reason of this exception is `null` value in insert data. When there is a
null data in data, `HoodieSparkSqlWriter` will do a schema evolution here:
https://github.com/apache/hudi/blob/b6124ff85a107ab170430947a24bc71df8612f1c/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L209
Then, the schema will be converted here:
https://github.com/apache/hudi/blob/b6124ff85a107ab170430947a24bc71df8612f1c/hudi-common/src/main/java/org/apache/hudi/internal/schema/convert/AvroInternalSchemaConverter.java#L440
PS: seems @alexeykudinkin just fixed this issue yesterday in
https://github.com/apache/hudi/pull/6358. I'm checking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
