ericm-db commented on PR #56907: URL: https://github.com/apache/spark/pull/56907#issuecomment-4857998142
Thanks @dbtsai, both are real — fixed in df7b09a. **1. Concurrent first-time startup race.** You're right, and it was masked in tests because `SPARK_TESTING` forces port 0. Fixed with three changes: - A discovery-file lock (`fcntl.flock` on `<discovery>.lock`) serializes start-up across processes, with a double-checked reuse inside the lock so only one process starts a server and the rest reconnect. - If our own spawned daemon exits, we now do a final `_reuse_from_discovery()` and reconnect to the winner before raising. - For the version-upgrade collision you described (stale server still holding the fixed port), if the configured port isn't bindable we fall back to an ephemeral port instead of failing. Verified with two processes launched simultaneously against the fixed port: both return the same endpoint, one starts and one reconnects, neither raises. Also verified the upgrade case falls back to an ephemeral port. Windows note: there's no `fcntl` there, so the lock is skipped and we rely on the reconnect-the-winner fallback alone. **2. Builder options dropped on first startup.** Also fixed. The starting client's confs (minus the keys the daemon controls itself — master/port/token/plugins and the reuse opt-in keys) are now merged the same way `_start_connect_server` merges `PYSPARK_REMOTE_INIT_CONF_*` + `opts`, and passed to the daemon via a `0600` JSON conf file that seeds its `SparkConf`. New end-to-end test asserts a static conf (`spark.sql.warehouse.dir`) takes effect on the server — which the per-session `_apply_options` path can't do, so it proves the seed was forwarded. As you noted, this is inherently first-client-only: a later run reconnecting to an already-warm JVM can't change static confs. I've documented that limitation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
