alexeykudinkin opened a new pull request, #6352: URL: https://github.com/apache/hudi/pull/6352
### Change Logs `SQLConf` by default isn't propagated to Executors from the Driver. This is the reason why configuration specified w/ `--conf` is not being respected by some components being invoked on the executor (see https://github.com/apache/hudi/issues/6278) In Spark this has been remediated by the introduction of `SQLExecutionRDD` that does snapshot `SQLConf` subsequently overriding it on the Executor side. However this has following limitations: 1. `SQLExecutionRDD` does propagate `SQLConf` only for RDDs that are **preceding** it (ie its dependencies), and doesn't for the ones that are chained after it. 2. `SQLExecutionRDD` is only applicable to `RDD[InternalRow]` To work around #2 we're introduced `SQLConfInjectingRDD` (which is just a generalization of `SQLExecutionRDD`) and make sure that it's properly injected (to address #1) inside, for ex, `HoodieSparkUtils.createRdd` invocation (that is being carried out on the Executor side) ### Impact None ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
