codope commented on issue #3735: URL: https://github.com/apache/hudi/issues/3735#issuecomment-931133022
I think this is a bug. What's happening here is that the [overwrite check](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java#L102) in OverwriteWithLatestAvroPayload is simply checking whether the two objects are equal or not. Since the nullable column gets converted to avro schema, the [defaultVal() method](https://avro.apache.org/docs/1.8.2/api/java/org/apache/avro/Schema.Field.html#defaultVal()) returns the corresponding `JsonProperties.Null` but the other object in the check is not the same type. So, the check returns false and hence that field gets overwritten. Instead we should modify that method to something like: ``` public Boolean overwriteField(Object value, Object defaultValue) { if (value == null) { return defaultValue instanceof JsonProperties.Null; } return Objects.equals(value, defaultValue); } ``` @peanut-chenzhong I have filed [a bug](https://issues.apache.org/jira/browse/HUDI-2509). Please raise a PR if you have the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
