jscheffl commented on code in PR #65943:
URL: https://github.com/apache/airflow/pull/65943#discussion_r3210893202
##########
providers/edge3/src/airflow/providers/edge3/cli/worker.py:
##########
@@ -447,17 +451,77 @@ def _run_job_via_supervisor(self, workload: ExecuteTask,
results_queue: Queue) -
results_queue.put(e)
return 1
- def _launch_job(self, workload: ExecuteTask) -> tuple[Process,
Queue[Exception]]:
+ def _launch_job_subprocess(self, workload: ExecuteTask) ->
subprocess.Popen:
+ """Launch workload via a fresh Python interpreter
(subprocess.Popen)."""
+ env = os.environ.copy()
+ if self._execution_api_server_url:
+ env["AIRFLOW__CORE__EXECUTION_API_SERVER_URL"] =
self._execution_api_server_url
+
+ # Keep stderr off a PIPE: the worker only inspects stderr after the
task finishes,
+ # so a verbose child could otherwise fill the pipe buffer and block
forever.
+ with tempfile.NamedTemporaryFile(
+ prefix="airflow-edge-task-stderr-", suffix=".log", delete=False
+ ) as stderr_file:
+ stderr_file_path = Path(stderr_file.name)
+ try:
+ process = subprocess.Popen(
+ [
+ sys.executable,
+ "-m",
+ "airflow.sdk.execution_time.execute_workload",
+ "--json-string",
+ workload.model_dump_json(),
+ ],
+ env=env,
+ start_new_session=True,
+ stderr=stderr_file,
Review Comment:
Having a sleep over and seeing the code... I do not actually want to insist
on the queue :-D I just mainly want to pass error details back from supervisor
if somethings failed into task logs. So the "text" content should be
passed-over.
For me it would also be okay to step away from the Queue in general and
transport the error details via a text file in both branches. Then we have one
technical backend for both execution options. Main part I want to achieve is to
have "text" transferred to instead of passing the exception to queue the test
can also be written to file and picked-up. That would make it leaner?
(Including if all is OK we do not need to pass "OK" text, we just use the
file for passing any error text?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
