zhengruifeng commented on code in PR #53528:
URL: https://github.com/apache/spark/pull/53528#discussion_r2652024148
##########
python/run-tests.py:
##########
@@ -234,27 +234,43 @@ def run_individual_python_test(target_dir, test_name,
pyspark_python, keep_test_
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
+ timeout = os.environ.get("PYSPARK_TEST_TIMEOUT")
+ if timeout is not None:
+ env["PYSPARK_TEST_TIMEOUT"] = timeout
+ timeout = int(timeout)
+
output_prefix = get_valid_filename(pyspark_python + "__" + test_name +
"__").lstrip("_")
# Delete is always set to False since the cleanup will be either done by
removing the
# whole test dir, or the test output is retained.
per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix,
dir=tmp_dir,
suffix=".log", delete=False)
LOGGER.info(
"Starting test(%s): %s (temp output: %s)", pyspark_python, test_name,
per_test_output.name)
+ cmd = [os.path.join(SPARK_HOME, "bin/pyspark")] + test_name.split()
start_time = time.time()
+
+ retcode = None
+ proc = None
try:
- retcode = TestRunner(
- [os.path.join(SPARK_HOME, "bin/pyspark")] + test_name.split(),
- env,
- per_test_output
- ).run()
+ if timeout:
Review Comment:
I cannot find a good way to support the `checkpoint()` feature and this PR
together, do you have any ideas? @gaogaotiantian
If it is doable, we can merge the two code path in follow up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
