yihua opened a new issue, #20249: URL: https://github.com/apache/hudi/issues/20249
On the Hudi Streamer path without a transformer, `LazyCastingIterator` calls `HoodieAvroUtils.rewriteRecordDeep(record, targetSchema)` for every source record. The target schema is parsed per partition, so it is never the same instance as the record schema, and every record pays a deep Avro `Schema.equals` before being returned unchanged. On a 30-column schema that is about 0.8 us and 4.9 KB of allocation per record. Proposal: compare each distinct source schema instance with the target once, and return records of an equal schema as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
