goingforstudying-ctrl opened a new pull request, #2562: URL: https://github.com/apache/datafusion-ballista/pull/2562
Spotted while reading #2533 — an executor that already ran a task for a session never picks up S3 settings the client changes later with `SET s3.*`. The scheduler plans with the new values (it builds a fresh context per submission), but the executor's session runtime cache (#1995) is keyed by session id alone, so every later task in that session gets the base RuntimeEnv built from the first task's config. The registry inside that env holds a clone of the original S3Options, so the new access key / session token / region / endpoint never gets read. Refreshing temporary credentials mid-session is the case that actually bites: tasks keep running with the expired token. Fix is Andy's first suggestion from the issue: the cache key is now (session id, fingerprint of the session config's extension entries). Extension iteration is prefix-ordered and each extension's entries get sorted by key before hashing, so two configs with equal extensions produce the same fingerprint and keep sharing a base — the whole point of the cache. A SET that changes any extension entry misses and builds a fresh base; reverting the SET hits the original entry again. Left out on purpose: - Built-in datafusion.* options aren't in the fingerprint. The base env's shared state (registry, disk manager, footer cache) isn't derived from them, and I didn't want to hash a couple hundred options on every task. If a custom RuntimeProducer ever reads built-in options this would need a rethink — matches the shipped producers today. - Didn't touch CustomObjectStoreRegistry (the issue's option 2, resolving options per get_store). That pushes config lookups into the read hot path; rekeying the cache is much smaller. Tests: turned the issue's reproducer into unit tests in runtime_cache.rs — a key-id change mid-session now yields a store built with the new key, equal options still share one base env (ptr_eq), reverting hits the original base, and different sessions with identical options get separate bases. cargo test -p ballista-executor --lib passes locally (81 tests, including the pre-existing session-cache suite). Fixes #2533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
