HyukjinKwon commented on code in PR #37288:
URL: https://github.com/apache/spark/pull/37288#discussion_r929788535
##########
python/run-tests.py:
##########
@@ -107,20 +122,26 @@ def run_individual_python_test(target_dir, test_name,
pyspark_python):
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
output_prefix = get_valid_filename(pyspark_python + "__" + test_name +
"__").lstrip("_")
- per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix,
suffix=".log")
+
+ if keep_test_output:
+ # The location is unique because the test is already in a unique
directory.
+ per_test_output = open(os.path.join(tmp_dir, output_prefix + ".log"),
"wb+")
+ else:
+ per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix,
suffix=".log")
Review Comment:
> As for this, my proposal here would be to reuse the existing target dir
option and simply move the log output into this path.
If we want to keep the option as a flag, I believe we can pass
`delete=False`.
##########
python/run-tests.py:
##########
@@ -107,20 +118,26 @@ def run_individual_python_test(target_dir, test_name,
pyspark_python):
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
output_prefix = get_valid_filename(pyspark_python + "__" + test_name +
"__").lstrip("_")
- per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix,
suffix=".log")
+
+ if keep_test_output:
+ # The location is unique because the test is already in a unique
directory.
Review Comment:
I believe we use `target/` directory so `mvn clean` or `sbt clean` remove
them out. If we specify a directory (for the purpose of keeping), I think it
makes sense to allow other directories. Maybe we could just document that it
will not be automatically removed if we you specify the path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
