voonhous commented on issue #17744: URL: https://github.com/apache/hudi/issues/17744#issuecomment-5826048476
Open gap: Spark 4.1 `PushVariantIntoScan` on MOR tables with a payload-based merger and log files. Base-file rows are read in the projection-struct shape while log rows keep full variants (`shouldProjectVariants` excludes payload-based merges), so the payload merge and the final struct-typed projection both see mixed shapes. A fix needs the base read left unprojected on that arm plus `buildVariantProjector` over the merged output. Surfaced on #20041; custom payloads are listed as an untested variant branch in the `TestVariantShreddingMixedLayouts` header. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
