voonhous opened a new issue, #18037: URL: https://github.com/apache/hudi/issues/18037
### Task Description **What needs to be done:** Since we already have parquet configurations for compression and sizing, variant shredding control configs should also be added here instead of using the ones that Spark offers, which will be ignored by Hudi. Spark configs that need to be carried over to Hudi: ``` spark.sql.variant.allowReadingShredded: boolean spark.sql.variant.writeShredding.enabled: boolean spark.sql.variant.forceShreddingSchemaForTest: string // e.g. "a int, b string, c decimal(15, 1) ``` **Why this task is needed:** Allows shredding control within hudi ### Task Type Code improvement/refactoring ### Related Issues **Parent feature issue:** (if applicable ) **Related issues:** NOTE: Use `Relationships` button to add parent/blocking issues after issue is created. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
