raphaelauv commented on code in PR #60041:
URL: https://github.com/apache/airflow/pull/60041#discussion_r2681648166
##########
providers/apache/spark/tests/system/apache/spark/example_pyspark.py:
##########
@@ -46,7 +46,7 @@ def example_pyspark():
# [START task_pyspark]
@task.pyspark(conn_id="spark-local")
- def spark_task(spark: SparkSession) -> pd.DataFrame:
+ def spark_task(spark: SparkSession, sc) -> pd.DataFrame:
Review Comment:
yes the sc argument become mandatory
but I'm in favor to remove it , since
"sc" is just
spark.sparkContext
so the user can do it in the python function if necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
