voonhous commented on issue #19249: URL: https://github.com/apache/hudi/issues/19249#issuecomment-4937143911
Scope update from further debugging on the migration branch: the gap is wider than meta columns. A CUSTOM merger that does not override isProjectionCompatible() (default false) makes the file-group reader demand the FULL table schema for both base and log reads (FileGroupReaderSchemaHandler.generateRequiredSchema short-circuits before consulting getMandatoryFieldsForMerging), so requiredSchema can contain arbitrary unprojected data columns that the reader context cannot type. The connector now fails such reads with a clear NOT_SUPPORTED error and requires custom mergers to be projection compatible (override isProjectionCompatible() and declare getMandatoryFieldsForMerging()), which matches the base-read path that already relies on HudiUtil.getMergeRequiredColumnHandles to prepend merger-mandatory columns. The refactor proposed in this issue (resolve handles from the full table schema) would also lift that restriction; note the base-file read would need to honor requiredSchema as w ell. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
