muyihao commented on issue #11723: URL: https://github.com/apache/hudi/issues/11723#issuecomment-2264995595
> The metaClient only list metadata files with call of `#getActiveTimeline`, which is triggered by `#getLatestCommitMetadataWithValidSchema`, so I think your analysis is reasonable. Maybe we can just pass around the `Option<InternalSchema>` to the constructor of `SchemaEvolutionContext`, currently one reader handles one `HoodieRealtimeFileSplit`, so maybe we can change it to share a singleton `Option<InternalSchema>` for each reader. Thank you for your reply. You are right; metadata files are only listed when calling `#getActiveTimeline`, and constructing the metaClient only reads the hoodie.properties file. Using a singleton `Option<InternalSchema> `is a great idea. Specifically, maybe we can fetch the InternalSchema during the split computation and place it into the `JobConf`, so that each reader can directly obtain the schema from the `JobConf`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
