milenkovicm opened a new issue, #1092:
URL: https://github.com/apache/datafusion-ballista/issues/1092
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
`SessionContextExt` at the moment does not expose methods which could be
used to set up SessionState and/or
configuration.
```rust
let ctx = SessionContext::remote(url &str)
let ctx = SessionContext::standalone()
```
**Describe the solution you'd like**
Extend `SessionStateExt` implementation to support configuring
`SessionContext` like DataFusion does. Provide a methods like
```rust
let state : SessionState = SessionStateBuilder::build()
let ctx = SessionContext::remote_with_state(url &str)
let ctx = SessionContext::standalone_with_state()
```
to enable custom session state setup, as well as providing custom ballista
related configuration as part of the SessionState.
To provide ballista specific configuration we could provide a
`SessionConfigExt` which would.
expose `with_ballista_config(&str, &str)`. Hopefully this way we would not
need to expose `BallistaConfig` to
end user (making it private and simplifying it).
An alternative approach would be to provide
`SessionConfigExt::new_with_ballista()`, this
way we could set ballista config with helper method which would:
```rust
let state = ctx.state_ref();
let mut state = state.write();
let config = state.config_mut().options_mut();
config.set(&key.key, key.value.as_deref().unwrap_or(""));
```
later approach could be better with python support. We would need to try
them out
**Describe alternatives you've considered**
An alternative approach would be keeping `BallistaConfiguration` which would
slightly increase code base,
and have a bit different approach than vanilla datafusion context set up.
This would imply slidhtly different
user facing `SessionContextExt` API where additional configuration property
`config: BallistaConfig` should be exposed.
**Additional context**
If effort is successful we would need to expose similar `SessionState`
configuration API (session state factory), after which we could deprecate
`ballista_core::object_store::with_object_store_registry(config:
RuntimeConfig)` giving ability to users to set up their own context. Something
very similar to `ballista-core::utils::default_session_builder(config:
SessionConfig)`.
If we go extra mile, maybe for datafusion api we would be able to provide a
`cloudpickle` session state factory where users would be able to configure
their onw SessionsState across cluster
task relates to #1088 and #1091 (kind off)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
