voonhous commented on code in PR #19808:
URL: https://github.com/apache/hudi/pull/19808#discussion_r3911180013
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -308,22 +310,26 @@ public class HoodieStorageConfig extends HoodieConfig {
public static final ConfigProperty<Boolean>
PARQUET_VARIANT_SHREDDING_SCHEMA_INFERENCE_ENABLED = ConfigProperty
.key("hoodie.parquet.variant.shredding.schema.inference.enabled")
- .defaultValue(false)
+ .defaultValue(true)
.sinceVersion("1.3.0")
- .withDocumentation("When enabled, the shredding schema for variant
columns without an explicit "
- + "typed_value in the write schema is inferred automatically per
parquet file from a sample of "
- + "the records written to that file, mirroring Spark 4.1's "
- + "spark.sql.variant.inferShreddingSchema. Requires Spark 4.1+ on
the writer classpath; "
- + "writes stay unshredded otherwise (Spark 4.0, Flink, Java
engines). Applies to every "
- + "parquet file the writer produces: base files and, on table
version 10+, the native "
+ .withDocumentation("Infers the shredding schema of variant columns that
have no explicit "
+ + "typed_value in the write schema, per parquet file, from a sample
of the records written "
+ + "to that file, mirroring Spark 4.1's
spark.sql.variant.inferShreddingSchema (also on by "
+ + "default there). Takes effect only when a Spark 4.1+ writer is on
the classpath; other "
+ + "writers (Spark 3.x, Spark 4.0, Flink, Java) ignore it and write
unshredded. Applies to "
+ + "every parquet file the writer produces: base files and, on table
version 10+, the native "
+ "parquet log files of MOR tables (each infers its own schema).
Data blocks inside "
+ "Avro-format log files, whether Avro or parquet
(hoodie.logfile.data.block.format), stay "
+ "unshredded and shred at compaction. Applies to top-level variant
columns only; a variant "
- + "nested inside a struct, array or map stays unshredded. This is a
write config rather than "
- + "a table config: SQL DML and procedures called by table name pick
it up from the table's "
- + "catalog properties, while path-based procedures, the DataSource
writer and the streamer "
- + "must be handed it explicitly. Up to 4096 records or 64MB are
buffered per "
- + "open file writer before the writer is created, on top of
parquet's own row-group "
+ + "nested inside a struct, array or map stays unshredded. Shredded
files can only be read "
+ + "back by Spark 4.1+: Spark 4.0, Hive and Flink readers fail fast
on them, so disable this "
+ + "option (or hoodie.parquet.variant.write.shredding.enabled) on
tables those engines read, "
+ + "and rewrite already shredded files through clustering with it
disabled to return to the "
Review Comment:
Right, `generateEffectiveSchema` keys on write shredding. The recipe now
names `hoodie.parquet.variant.write.shredding.enabled=false` and says
explicitly that it, not the inference key, is what strips typed_value from a
schema read back off shredded files.
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/io/storage/hadoop/TestHoodieAvroFileWriterFactoryVariantInference.java:
##########
@@ -54,25 +54,28 @@
/**
* Pins the no-inferrer degradation of shredding-schema inference in
- * {@link HoodieAvroFileWriterFactory}: this module's classpath carries no
Spark version module,
- * so {@link VariantShreddingRuntime#lookupInferrer()} is empty here, which is
what engines without
- * Spark 4.1+ (Flink, Java, Spark 4.0) see in production.
+ * {@link HoodieAvroFileWriterFactory}. Inference is on by default, so this
gate is reached by every
Review Comment:
Reworded. The doc now leads with the provider gate: Spark 4.0 is the
classpath that reaches the inferrer gate with a stock config, since the
provider ships in hudi-spark4-common, while Flink, Java and Spark 3.x stop a
gate earlier. That is also why the test names a provider explicitly - it
isolates the inferrer gate instead of passing through whichever gate happens to
fire first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
