Shooter23 opened a new pull request #35043:
URL: https://github.com/apache/spark/pull/35043
When running `PYSPARK_DRIVER_PYTHON=ipython pyspark` on xterm the
find-spark-home script calls `ipython /path/to/find_spark_home.py` and the
string printed by that script gets assigned to SPARK_HOME. When run with
IPython that string will start with a sequence bounded by control characters
before the path determined by find_spark_home.py. While this part of the string
does not appear on echo it will cause pyspark to compose paths improperly when
using SPARK_HOME.
To see the sequence run:
>>> import os
>>> p = os.popen('ipython somescript.py')
>>> p.read()
`'\x1b[22;0t\x1b]0;IPython: {current directory}\x07the expected output\n'`
The cut command removes the sequence before "the expected output". Lines
without a bell character (\x07), such as you get when running `python3
find_spark_home.py`, remain unchanged.
### What changes were proposed in this pull request?
Fixing the assignment to SPARK_HOME in find-spark-home to remove the control
characters added when using ipython.
### Why are the changes needed?
On xterm running `PYSPARK_DRIVER_PYTHON=ipython pyspark` causes pyspark to
compose paths improperly, prepending the current working directory to
SPARK_HOME as determined by find_spark_home.py, making it unable to find the
files it seeks.
### Does this PR introduce _any_ user-facing change?
Yes. Before the change I would get "No such file or directory" errors as the
current working directory would get prepended to SPARK_HOME. After the change
the pyspark interactive prompt starts as expected with an ipython prompt.
### How was this patch tested?
I ran pyspark with PYSPARK_DRIVER_PYTHON set to "python", "python3" and
"ipython". All three variations gave the appropriate prompt with the expected
session and context variables set. I also tested the pipe to the cut command
with lines with and without bell characters to ensure that the addition had no
effect on the latter. I didn't modify the current testing scheme because I
couldn't find an extant test for any of the relevant bash scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
