holdenk commented on code in PR #59580:
URL: https://github.com/apache/spark/pull/59580#discussion_r4342391979
##########
python/pyspark/sql/transpile.py:
##########
@@ -928,6 +928,13 @@ def _get_transpilers(session: "SparkSession") ->
List[AbstractTranspiler]:
if not configured_transpilers:
return []
transpiler_names = configured_transpilers.split(",")
+ unknown_names = [name for name in transpiler_names if name not in
AbstractTranspiler.varieties]
+ if unknown_names:
+ warnings.warn(
+ f"Ignoring unknown transpilers {unknown_names} in "
+ "spark.sql.experimental.optimizer.pyTranspilers; registered
transpilers are "
+ f"{sorted(AbstractTranspiler.varieties)}"
+ )
return [
AbstractTranspiler.varieties[name]()
for name in transpiler_names
Review Comment:
We should only construct this list on the known names otherwise we're going
to warn and then throw a hard error, but we just want to warn and keep going.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
