This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 5496e150c13 Improve error handling in edge worker on 405 (#60425)
add 417698fce9c PysparkOperator and not only a decorator (#60041)
No new revisions were added by this update.
Summary of changes:
providers/apache/spark/docs/decorators/pyspark.rst | 2 +-
providers/apache/spark/docs/operators.rst | 25 +++++++
providers/apache/spark/provider.yaml | 1 +
.../providers/apache/spark/decorators/pyspark.py | 80 ++++------------------
.../providers/apache/spark/get_provider_info.py | 1 +
.../pyspark.py => operators/spark_pyspark.py} | 59 +++-------------
.../tests/system/apache/spark/example_spark_dag.py | 11 +++
.../unit/apache/spark/decorators/test_pyspark.py | 11 ++-
.../apache/spark/operators/test_spark_pyspark.py} | 27 ++++----
9 files changed, 80 insertions(+), 137 deletions(-)
copy
providers/apache/spark/src/airflow/providers/apache/spark/{decorators/pyspark.py
=> operators/spark_pyspark.py} (65%)
copy providers/{redis/tests/unit/redis/sensors/test_redis_key.py =>
apache/spark/tests/unit/apache/spark/operators/test_spark_pyspark.py} (62%)
