[
https://issues.apache.org/jira/browse/SPARK-59985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Firas Bouzazi updated SPARK-59985:
----------------------------------
Remaining Estimate: (was: 2h)
Original Estimate: (was: 2h)
> TorchDistributor logs a TypeError instead of a warning when the log streaming
> server fails to start
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-59985
> URL: https://issues.apache.org/jira/browse/SPARK-59985
> Project: Spark
> Issue Type: Bug
> Components: ML, PySpark
> Affects Versions: 4.2.0
> Reporter: Firas Bouzazi
> Assignee: Firas Bouzazi
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> In \{{TorchDistributor._run_distributed_training}}
> (\{{python/pyspark/ml/torch/distributor.py}}), the warning emitted when the
> log streaming server fails to start has a stray comma, so the error message
> is passed to \{{logger.warning}} as a separate positional argument:
> {code:python}
> self.logger.warning(
> "Start torch distributor log streaming server failed, "
> "You cannot receive logs sent from distributor workers, ", # <- stray
> comma
> f"error: \{repr(e)}.",
> )
> {code}
> Since the message has no %-placeholder, \{{logging}} fails with "TypeError:
> not all arguments converted during string formatting" when the record is
> formatted. Training continues (the fallback added in SPARK-44909 still sets
> the port to -1), but instead of a one-line warning:
> * with default logging settings the user sees a "--- Logging error ---"
> traceback that looks like a crash;
> * with \{{logging.raiseExceptions = False}} the warning is dropped entirely,
> so the user never learns why worker logs are missing;
> * \{{record.getMessage()}} raises, so the warning cannot be asserted in tests.
> h3. Reproduction
> Call \{{_run_distributed_training}} with \{{LogStreamingServer.start}}
> raising, e.g. \{{OSError("Address already in use")}}:
> {noformat}
> --- Logging error ---
> Traceback (most recent call last):
> File ".../logging/__init__.py", line 377, in getMessage
> msg = msg % self.args
> TypeError: not all arguments converted during string formatting
> Call stack:
> File ".../pyspark/ml/torch/distributor.py", line 815, in
> _run_distributed_training
> self.logger.warning(
> Message: 'Start torch distributor log streaming server failed, You cannot
> receive logs sent from distributor workers, '
> Arguments: ("error: OSError('Address already in use').",)
> {noformat}
> h3. Proposed fix
> Make it a single format string with the exception passed as an argument:
> {code:python}
> self.logger.warning(
> "Start torch distributor log streaming server failed. "
> "You cannot receive logs sent from distributor workers, error: %r.",
> e,
> )
> {code}
> plus a unit test that makes \{{LogStreamingServer.start}} fail and checks the
> warning message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
