yihua opened a new issue, #19870:
URL: https://github.com/apache/hudi/issues/19870
### Problem
`AvroSchemaEvolutionUtils.reconcileSchema` starts with
/* If incoming schema is null, we fall back on table schema. */
if (incomingSchema.isSchemaNull()) {
which handles an Avro NULL-typed schema but dereferences a Java `null`. A
null incoming schema only reaches it because `reconcileSchemaRequirements`
returns the *source* schema untouched whenever the *target* (table) schema is
NULL-typed or has no fields, so a null source leaks through instead of being
replaced by the table schema. The schema-on-read branch of
`HoodieSchemaUtils.deduceWriterSchema` has the same gap
(`sourceSchema.getFields` on the raw incoming schema).
### How it surfaces
A HoodieStreamer row source whose `SchemaProvider` returns `null` from
`getTargetSchema()` (schema is derived from each batch), writing to a table
whose latest commit carries a zero-column record schema (what a sync over
column-less input leaves behind), fails every empty batch with
java.lang.NullPointerException
at
org.apache.hudi.internal.schema.utils.AvroSchemaEvolutionUtils.reconcileSchema
at
org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchemaWithoutReconcile
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchema
at
org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider
at
org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource
Because the failed sync never advances the checkpoint, the stream cannot
recover on its own; a restart resumes at the same checkpoint and throws again.
With a table schema that has columns the bug is masked, which is why it only
shows on brand-new tables whose first ingested files carried no records.
### Expected
An absent incoming schema falls back to the table schema, as the comment
promises, and an empty batch lands an empty commit that advances the checkpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
