Sudhanva Huruli created SPARK-62027:
---------------------------------------

             Summary: spark-submit in Spark Connect mode drops settings beyond 
the first group of ten, such as spark.jars
                 Key: SPARK-62027
                 URL: https://issues.apache.org/jira/browse/SPARK-62027
             Project: Spark
          Issue Type: Bug
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


When spark-submit runs a Python application in Spark Connect mode, the launcher 
passes its settings to Python through environment variables, split into groups 
of ten. The Python side reads every group but keeps only the last one, so 
settings in the other groups, such as spark.jars or --conf values, are missing 
from the session and the Spark Connect server. Reproduction: run a Python 
script with spark-submit in Spark Connect mode, with --jars pointing at a local 
jar file and about a dozen --conf flags, and have the script print spark.jars. 
Expected: every setting passed on the command line is visible to the session.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to