voonhous commented on issue #19035: URL: https://github.com/apache/hudi/issues/19035#issuecomment-4768966605
Reframed the scope to match the benchmark we're actually building: re-centered on (1) Hudi shredded vs vanilla Spark 4.1 native parquet shredding (implementation parity) and (2) shredded-vs-unshredded within Hudi. Demoted the JSON-string / nested-struct baselines to optional, and moved MOR, upsert/MERGE, and bytes-scanned/data-skipping to deferred follow-ups (data skipping depends on colstats for shredded variants, #17988). Also noted the harness is a standalone spark-submit app (`VariantShreddingBenchmark` in hudi-examples-spark) rather than the test-scoped HoodieBenchmark tooling, since it needs Spark 4.1 native-shredding confs + parquet-footer inspection against the spark4.1 bundle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
