cloud-fan opened a new pull request, #53854:
URL: https://github.com/apache/spark/pull/53854
### What changes were proposed in this pull request?
This PR fixes the stack overflow issue that occurs when `SQLConf.get` is
called during `sessionState` initialization.
The root cause is a circular dependency: `SQLConf.get` calls
`session.sessionState.conf`, but if this happens during `sessionState`
initialization itself, it causes infinite recursion.
The fix moves `sqlConf` to be a direct lazy val on `SparkSession`,
initialized before `sessionState`. This eliminates the recursion because the
confGetter now accesses `session.sqlConf` directly instead of
`session.sessionState.conf`.
**Architecture before (problematic):**
```
SparkSession
└── lazy val sessionState: SessionState
└── conf: SQLConf ← created inside SessionStateBuilder
SQLConf.get → confGetter → session.sessionState.conf → RECURSION during init
```
**Architecture after (fixed):**
```
SparkSession
├── lazy val sqlConf: SQLConf ← Initialized FIRST (no dependencies
on sessionState)
└── lazy val sessionState: SessionState
└── conf → session.sqlConf ← Same instance, no recursion
SQLConf.get → confGetter → session.sqlConf → NO RECURSION
```
**Changes:**
- Add `sqlConf` lazy val to `SparkSession` before `sessionState`
- Update confGetter to use `session.sqlConf` instead of
`session.sessionState.conf`
- Update `RuntimeConfig` to use `sqlConf` directly
- Simplify `BaseSessionStateBuilder.conf` to delegate to `session.sqlConf`
- Remove the old `WithTestConf` trait from builders
- Add new `WithTestConf` trait for test SparkSession subclasses with
`newTestConf()` helper
- Update `TestSparkSession` and `TestHiveSparkSession` to use the new pattern
### Why are the changes needed?
To fix the stack overflow when `SQLConf.get` is called during `sessionState`
initialization.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests. Ran `ListTablesSuite` to verify basic session functionality
works.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
