hudi-agent commented on code in PR #19808:
URL: https://github.com/apache/hudi/pull/19808#discussion_r3903907221
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -308,22 +310,26 @@ public class HoodieStorageConfig extends HoodieConfig {
public static final ConfigProperty<Boolean>
PARQUET_VARIANT_SHREDDING_SCHEMA_INFERENCE_ENABLED = ConfigProperty
.key("hoodie.parquet.variant.shredding.schema.inference.enabled")
- .defaultValue(false)
+ .defaultValue(true)
Review Comment:
🤖 Flipping this to default-on while it stays a write config (not persisted
per table) inverts the failure mode of the "writer forgot the config" case:
before, a writer missing this wrote unshredded and stayed cross-engine
readable; now any Spark 4.1+ writer that misses the opt-out (a path-based
write, the DataSource writer, a streamer) silently shreds and breaks
Flink/Hive/Spark-4.0 reads of that same table. Have you considered persisting
the shredding decision as a table property so the opt-out is durable across
every writer, rather than relying on each write path to pass it? @yihua since
this is effectively a storage-format-default change, worth a committer/PMC look
at the default-on choice and confirming the non-4.1 reader fail-fast is
complete (no silent payload drop) across all read paths.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
