Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

HyukjinKwon commented on PR #46125: URL: https://github.com/apache/spark/pull/46125#issuecomment-2073665401 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

HyukjinKwon closed pull request #46125: [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark URL: https://github.com/apache/spark/pull/46125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

ericm-db commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576656803 ## core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala: ## @@ -91,11 +91,27 @@ private[spark] class StreamingPythonRunner( new

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

rangadi commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576629583 ## core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala: ## @@ -91,11 +91,27 @@ private[spark] class StreamingPythonRunner( new

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

ericm-db commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576514370 ## core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala: ## @@ -91,6 +91,12 @@ private[spark] class StreamingPythonRunner( new

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

grundprinzip commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576480636 ## python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py: ## @@ -76,16 +73,21 @@ def process(df_id, batch_id): # type: ignore[no-untyped-def]

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

grundprinzip commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576472509 ## python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py: ## @@ -62,10 +62,7 @@ def main(infile: IO, outfile: IO) -> None: assert

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

grundprinzip commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1576471728 ## core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala: ## @@ -91,6 +91,12 @@ private[spark] class StreamingPythonRunner( new

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

rangadi commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1575192686 ## core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala: ## @@ -91,6 +91,11 @@ private[spark] class StreamingPythonRunner( new

Re: [PR] [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark [spark]

rangadi commented on code in PR #46125: URL: https://github.com/apache/spark/pull/46125#discussion_r1575186174 ## python/pyspark/sql/tests/connect/streaming/test_parity_foreach_batch.py: ## @@ -66,6 +66,27 @@ def func(df, _): q =