vinishjail97 opened a new pull request, #10900: URL: https://github.com/apache/hudi/pull/10900
### Change Logs Add a new config HOODIE_SPARK_DATASOURCE_OPTIONS which is used by the spark dataframe reader for HoodieIncrSource, options like using metadataTable, dataSkipping present inDataSourceOptions.scala can be passed for efficient pruning of files. https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala ### Impact The files will be pruned using colstats and other mechanisms available making `HoodieIncrSource` more efficient. ### Risk level (write none, low medium or high below) Low ### Documentation Update HOODIE_SPARK_DATASOURCE_OPTIONS is the new config being added. A comma separate list of options that can be passed to the spark dataframe reader of a hudi table, eg: hoodie.metadata.enable=true,hoodie.enable.data.skipping=true. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
