Firas Bouzazi created SPARK-59985:
-------------------------------------
Summary: TorchDistributor logs a TypeError instead of a warning
when the log streaming server fails to start
Key: SPARK-59985
URL: https://issues.apache.org/jira/browse/SPARK-59985
Project: Spark
Issue Type: Bug
Components: ML, PySpark
Affects Versions: 4.2.0
Reporter: Firas Bouzazi
In \{{TorchDistributor._run_distributed_training}}
(\{{python/pyspark/ml/torch/distributor.py}}), the warning emitted when the log
streaming server fails to start has a stray comma, so the error message is
passed to \{{logger.warning}} as a separate positional argument:
{code:python}
self.logger.warning(
"Start torch distributor log streaming server failed, "
"You cannot receive logs sent from distributor workers, ", # <- stray
comma
f"error: \{repr(e)}.",
)
{code}
Since the message has no %-placeholder, \{{logging}} fails with "TypeError: not
all arguments converted during string formatting" when the record is formatted.
Training continues (the fallback added in SPARK-44909 still sets the port to
-1), but instead of a one-line warning:
* with default logging settings the user sees a "--- Logging error ---"
traceback that looks like a crash;
* with \{{logging.raiseExceptions = False}} the warning is dropped entirely, so
the user never learns why worker logs are missing;
* \{{record.getMessage()}} raises, so the warning cannot be asserted in tests.
h3. Reproduction
Call \{{_run_distributed_training}} with \{{LogStreamingServer.start}} raising,
e.g. \{{OSError("Address already in use")}}:
{noformat}
--- Logging error ---
Traceback (most recent call last):
File ".../logging/__init__.py", line 377, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File ".../pyspark/ml/torch/distributor.py", line 815, in
_run_distributed_training
self.logger.warning(
Message: 'Start torch distributor log streaming server failed, You cannot
receive logs sent from distributor workers, '
Arguments: ("error: OSError('Address already in use').",)
{noformat}
h3. Proposed fix
Make it a single format string with the exception passed as an argument:
{code:python}
self.logger.warning(
"Start torch distributor log streaming server failed. "
"You cannot receive logs sent from distributor workers, error: %r.",
e,
)
{code}
plus a unit test that makes \{{LogStreamingServer.start}} fail and checks the
warning message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
