This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 74d6d1d Add note about using dag_run.conf in BashOperator (#9143)
new 863bed7 Fix --forward-credentials flag in Breeze (#8554)
new 0d8a238 Fixed optimistions of non-py-code builds (#8601)
new ab10f48 Fix the process of requirements generations (#8648)
new f8bf546 Fixed test-target command (#8795)
new af6aed4 Add comments to breeze scripts (#8797)
new ff5b33e Useful help information in test-target and docker-compose
commands (#8796)
new ffbde53 The librabbitmq library stopped installing for python3.7
(#8853)
new 8d8d03e Use Debian's provided JRE from Buster (#8919)
new 431b52a Re-run all tests when Dockerfile or Github worflow change
(#8924)
new 519401b Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work
(#8938)
new 40d650b Python base images are stored in cache (#8943)
new 51a215c Add ADDITIONAL_PYTHON_DEPS (#9031)
new ab5e2ea Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
new 62c97ed Additional python extras and deps can be set in breeze (#9035)
new daca507 detect incompatible docker server version in breeze (#9042)
new 374a3e8 Adds hive as extra in pyhive (#9075)
new 4c6a82a Prevents failure on fixing permissions for files with space
in it (#9076)
new 64194ee Enable configurable git sync depth (#9094)
new b66a706 Don't reuse MY_DIR in breeze to mean different folder from
ci/_utils.sh (#9098)
new eed7d33 You can push with Breeze as separate command and to cache
(#8976)
new fbe1fe3 Produce less verbose output when building docker mount
options (#9103)
new 060cced Display docs errors summary (#8392)
new d4a00b7 Remove Hive/Hadoop/Java dependency from unit tests (#9029)
new 8dacc27 Kubernetes Cluster is started on host not in the container
(#8265)
new c34cd6d Fixes a bug where `build-image` command did not calculate md5
(#9130)
new 945b0e6 Fix INTEGRATIONS[*]: unbound variable error in breeze (#9135)
new 7236616 Cope with multiple processes get_remote_image_info in
parallel (#9105)
new 4e7c48e Remove remnant kubernetes stuff from breeze scripts (#9138)
new 86eeb00 Restrict google-cloud-texttospeach to <v2 (#9137)
new 410bcd4 Use static binary linked docker client in CI image (#9126)
new f16cae3 Remove vendored nvd3 and slugify libraries (#9136)
new c0534d8 Replaces cloud-provider CLIs in CI image with scripts running
containers (#9129)
new a4dfb98 Split utils sh (#9132)
new ec12334 Add PR/issue note in Contribution Workflow Example (#9177)
new e5162b0 Fixes failure of the build scripts when remote repo does not
exist (#9188)
new 4b8fbf2 Improved cloud tool available in the trimmed down CI
container (#9167)
The 36 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.coveragerc | 1 -
.flake8 | 2 +-
.github/workflows/ci.yml | 69 +-
.gitignore | 4 +-
.pre-commit-config.yaml | 40 +-
.rat-excludes | 5 +-
BREEZE.rst | 445 ++++--
CI.rst | 33 +-
CONTRIBUTING.rst | 10 +-
Dockerfile | 10 +-
Dockerfile.ci | 120 +-
IMAGES.rst | 27 +-
LICENSE | 2 +-
MANIFEST.in | 1 -
README.md | 2 +
TESTING.rst | 241 +++-
airflow/_vendor/README | 13 -
airflow/_vendor/__init__.py | 18 -
airflow/_vendor/nvd3/LICENSE | 24 -
airflow/_vendor/nvd3/NVD3Chart.py | 506 -------
airflow/_vendor/nvd3/__init__.py | 29 -
airflow/_vendor/nvd3/cumulativeLineChart.py | 104 --
airflow/_vendor/nvd3/discreteBarChart.py | 91 --
airflow/_vendor/nvd3/ipynb.py | 91 --
airflow/_vendor/nvd3/lineChart.py | 120 --
airflow/_vendor/nvd3/linePlusBarChart.py | 131 --
airflow/_vendor/nvd3/lineWithFocusChart.py | 105 --
airflow/_vendor/nvd3/multiBarChart.py | 95 --
airflow/_vendor/nvd3/multiBarHorizontalChart.py | 100 --
airflow/_vendor/nvd3/pieChart.py | 101 --
airflow/_vendor/nvd3/scatterChart.py | 121 --
airflow/_vendor/nvd3/stackedAreaChart.py | 99 --
airflow/_vendor/nvd3/templates/base.html | 35 -
airflow/_vendor/nvd3/templates/content.html | 123 --
.../nvd3/templates/cumulativelinechart.html | 10 -
.../_vendor/nvd3/templates/discretebarchart.html | 31 -
.../_vendor/nvd3/templates/linebarwfocuschart.html | 60 -
airflow/_vendor/nvd3/templates/linechart.html | 46 -
.../_vendor/nvd3/templates/lineplusbarchart.html | 43 -
.../_vendor/nvd3/templates/linewfocuschart.html | 10 -
airflow/_vendor/nvd3/templates/multibarchart.html | 10 -
.../nvd3/templates/multibarcharthorizontal.html | 10 -
airflow/_vendor/nvd3/templates/page.html | 12 -
airflow/_vendor/nvd3/templates/piechart.html | 80 -
airflow/_vendor/nvd3/templates/scatterchart.html | 52 -
.../_vendor/nvd3/templates/stackedareachart.html | 7 -
airflow/_vendor/nvd3/translator.py | 71 -
airflow/_vendor/slugify/LICENSE | 21 -
airflow/_vendor/slugify/__init__.py | 6 -
airflow/_vendor/slugify/slugify.py | 185 ---
airflow/config_templates/config.yml | 8 +
airflow/config_templates/default_airflow.cfg | 4 +
airflow/executors/kubernetes_executor.py | 2 +
airflow/hooks/hive_hooks.py | 4 +-
airflow/kubernetes/refresh_config.py | 1 -
airflow/kubernetes/volume_mount.py | 36 +-
airflow/kubernetes/worker_configuration.py | 2 +-
airflow/operators/hive_operator.py | 15 +
airflow/operators/hive_to_samba_operator.py | 14 +-
airflow/www/views.py | 7 +-
breeze | 629 +++++---
breeze-complete | 61 +-
docs/Makefile | 194 ---
docs/build | 265 ++++
docs/build.sh | 108 --
.../kubernetes => kubernetes_tests}/__init__.py | 0
.../test_kubernetes_executor.py | 101 +-
.../test_kubernetes_pod_operator.py | 432 ++++--
requirements/requirements-python3.6.txt | 4 +-
requirements/requirements-python3.7.txt | 4 +-
scripts/ci/_all_libs.sh | 49 +
scripts/ci/_script_init.sh | 9 +-
scripts/ci/_utils.sh | 1525 --------------------
scripts/ci/ci_build_dockerhub.sh | 33 +-
scripts/ci/ci_count_changed_files.sh | 6 +-
...ic_checks.sh => ci_deploy_app_to_kubernetes.sh} | 35 +-
scripts/ci/ci_docs.sh | 7 +-
scripts/ci/ci_fix_ownership.sh | 12 +
scripts/ci/ci_flake8.sh | 6 +
...oduction_images.sh => ci_load_image_to_kind.sh} | 17 +-
scripts/ci/ci_mypy.sh | 3 +
...ges.sh => ci_perform_kind_cluster_operation.sh} | 7 +-
scripts/ci/ci_run_airflow_testing.sh | 39 +-
scripts/ci/ci_run_kubernetes_tests.sh | 105 ++
scripts/ci/docker-compose/backend-sqlite.yml | 2 +-
scripts/ci/docker-compose/base.yml | 1 -
scripts/ci/docker-compose/forward-credentials.yml | 9 +-
scripts/ci/docker-compose/local-prod.yml | 3 +
scripts/ci/docker-compose/local.yml | 10 +-
scripts/ci/docker-compose/runtime-kubernetes.yml | 25 -
scripts/ci/in_container/_in_container_utils.sh | 101 +-
scripts/ci/in_container/check_environment.sh | 61 +-
scripts/ci/in_container/entrypoint_ci.sh | 85 +-
.../ci/in_container/kubernetes/app/deploy_app.sh | 210 ---
.../app/templates/init_git_sync.template.yaml | 36 -
.../kubernetes/docker/airflow-test-env-init.sh | 23 -
.../ci/in_container/kubernetes/docker/bootstrap.sh | 27 -
.../kubernetes/docker/rebuild_airflow_image.sh | 74 -
.../in_container/kubernetes/setup_kind_cluster.sh | 187 ---
scripts/ci/in_container/run_ci_tests.sh | 7 +-
scripts/ci/in_container/run_docs_build.sh | 2 +-
.../ci/in_container/run_generate_requirements.sh | 39 +-
scripts/ci/in_container/run_system_tests.sh | 1 -
.../kubernetes/app/postgres.yaml | 0
.../{in_container => }/kubernetes/app/secrets.yaml | 0
.../kubernetes/app/templates/airflow.template.yaml | 66 +-
.../app/templates/configmaps.template.yaml | 1 +
.../app/templates/init_git_sync.template.yaml | 40 +-
.../{in_container => }/kubernetes/app/volumes.yaml | 0
.../docker/airflow-test-env-init-dags.sh} | 19 +-
.../kubernetes/docker/airflow-test-env-init-db.sh | 26 +-
scripts/ci/kubernetes/docker/bootstrap.sh | 74 +
.../kubernetes/kind-cluster-conf.yaml | 0
scripts/ci/libraries/_build_images.sh | 771 ++++++++++
scripts/ci/libraries/_initialization.sh | 235 +++
scripts/ci/libraries/_kind.sh | 450 ++++++
scripts/ci/libraries/_local_mounts.sh | 66 +
scripts/ci/libraries/_md5sum.sh | 137 ++
scripts/ci/libraries/_parameters.sh | 58 +
scripts/ci/libraries/_permissions.sh | 59 +
scripts/ci/libraries/_push_pull_remove_images.sh | 183 +++
scripts/ci/{ci_docs.sh => libraries/_runs.sh} | 39 +-
scripts/ci/libraries/_sanity_checks.sh | 166 +++
scripts/ci/libraries/_spinner.sh | 53 +
scripts/ci/libraries/_start_end.sh | 79 +
.../{ci_bat_tests.sh => libraries/_verbosity.sh} | 29 +-
scripts/ci/pre_commit_local_yml_mounts.sh | 32 +-
scripts/ci/prepare_tool_scripts.sh | 64 +
setup.cfg | 28 +-
setup.py | 11 +-
tests/bats/bats_utils.bash | 4 +-
tests/bats/test_local_mounts.bats | 79 +-
tests/conftest.py | 37 -
tests/contrib/operators/test_hive_stats.py | 340 +++++
tests/contrib/operators/test_hive_to_mysql.py | 154 ++
tests/contrib/operators/test_hive_to_samba.py | 88 ++
.../sensors/test_hive_partition.py} | 39 +-
tests/contrib/sensors/test_metastore_partition.py | 43 +
tests/contrib/sensors/test_webhdfs.py | 42 +
.../test_snowflake_operator.py => hive.py} | 50 +-
tests/hooks/test_hive_hook.py | 613 +++++---
tests/operators/test_hive_operator.py | 411 +++---
tests/operators/test_operators.py | 249 +++-
.../kubernetes/operators/test_kubernetes_pod.py | 134 ++
tests/runtime/__init__.py | 18 -
tests/sensors/test_named_hive_partition_sensor.py | 107 +-
tests/test_utils/mock_hooks.py | 96 ++
tests/test_utils/mock_operators.py | 9 +
tests/test_utils/mock_process.py | 87 ++
149 files changed, 6662 insertions(+), 6843 deletions(-)
delete mode 100644 airflow/_vendor/README
delete mode 100644 airflow/_vendor/__init__.py
delete mode 100644 airflow/_vendor/nvd3/LICENSE
delete mode 100644 airflow/_vendor/nvd3/NVD3Chart.py
delete mode 100755 airflow/_vendor/nvd3/__init__.py
delete mode 100644 airflow/_vendor/nvd3/cumulativeLineChart.py
delete mode 100644 airflow/_vendor/nvd3/discreteBarChart.py
delete mode 100644 airflow/_vendor/nvd3/ipynb.py
delete mode 100644 airflow/_vendor/nvd3/lineChart.py
delete mode 100644 airflow/_vendor/nvd3/linePlusBarChart.py
delete mode 100644 airflow/_vendor/nvd3/lineWithFocusChart.py
delete mode 100644 airflow/_vendor/nvd3/multiBarChart.py
delete mode 100644 airflow/_vendor/nvd3/multiBarHorizontalChart.py
delete mode 100644 airflow/_vendor/nvd3/pieChart.py
delete mode 100644 airflow/_vendor/nvd3/scatterChart.py
delete mode 100644 airflow/_vendor/nvd3/stackedAreaChart.py
delete mode 100644 airflow/_vendor/nvd3/templates/base.html
delete mode 100644 airflow/_vendor/nvd3/templates/content.html
delete mode 100644 airflow/_vendor/nvd3/templates/cumulativelinechart.html
delete mode 100644 airflow/_vendor/nvd3/templates/discretebarchart.html
delete mode 100644 airflow/_vendor/nvd3/templates/linebarwfocuschart.html
delete mode 100644 airflow/_vendor/nvd3/templates/linechart.html
delete mode 100644 airflow/_vendor/nvd3/templates/lineplusbarchart.html
delete mode 100644 airflow/_vendor/nvd3/templates/linewfocuschart.html
delete mode 100644 airflow/_vendor/nvd3/templates/multibarchart.html
delete mode 100644 airflow/_vendor/nvd3/templates/multibarcharthorizontal.html
delete mode 100644 airflow/_vendor/nvd3/templates/page.html
delete mode 100644 airflow/_vendor/nvd3/templates/piechart.html
delete mode 100644 airflow/_vendor/nvd3/templates/scatterchart.html
delete mode 100644 airflow/_vendor/nvd3/templates/stackedareachart.html
delete mode 100644 airflow/_vendor/nvd3/translator.py
delete mode 100644 airflow/_vendor/slugify/LICENSE
delete mode 100644 airflow/_vendor/slugify/__init__.py
delete mode 100644 airflow/_vendor/slugify/slugify.py
delete mode 100644 docs/Makefile
create mode 100755 docs/build
delete mode 100755 docs/build.sh
rename {tests/runtime/kubernetes => kubernetes_tests}/__init__.py (100%)
rename {tests/runtime/kubernetes =>
kubernetes_tests}/test_kubernetes_executor.py (71%)
rename {tests/runtime/kubernetes =>
kubernetes_tests}/test_kubernetes_pod_operator.py (64%)
create mode 100755 scripts/ci/_all_libs.sh
delete mode 100644 scripts/ci/_utils.sh
copy scripts/ci/{ci_run_static_checks.sh => ci_deploy_app_to_kubernetes.sh}
(61%)
copy scripts/ci/{ci_push_production_images.sh => ci_load_image_to_kind.sh}
(61%)
copy scripts/ci/{ci_build_production_images.sh =>
ci_perform_kind_cluster_operation.sh} (89%)
create mode 100755 scripts/ci/ci_run_kubernetes_tests.sh
delete mode 100644 scripts/ci/docker-compose/runtime-kubernetes.yml
delete mode 100755 scripts/ci/in_container/kubernetes/app/deploy_app.sh
delete mode 100644
scripts/ci/in_container/kubernetes/app/templates/init_git_sync.template.yaml
delete mode 100755
scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh
delete mode 100755 scripts/ci/in_container/kubernetes/docker/bootstrap.sh
delete mode 100755
scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
delete mode 100755 scripts/ci/in_container/kubernetes/setup_kind_cluster.sh
rename scripts/ci/{in_container => }/kubernetes/app/postgres.yaml (100%)
rename scripts/ci/{in_container => }/kubernetes/app/secrets.yaml (100%)
rename scripts/ci/{in_container =>
}/kubernetes/app/templates/airflow.template.yaml (70%)
rename scripts/ci/{in_container =>
}/kubernetes/app/templates/configmaps.template.yaml (99%)
copy airflow/api/auth/backend/deny_all.py =>
scripts/ci/kubernetes/app/templates/init_git_sync.template.yaml (55%)
rename scripts/ci/{in_container => }/kubernetes/app/volumes.yaml (100%)
copy scripts/ci/{pre_commit_update_extras.sh =>
kubernetes/docker/airflow-test-env-init-dags.sh} (72%)
copy hooks/push => scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh
(64%)
create mode 100755 scripts/ci/kubernetes/docker/bootstrap.sh
rename scripts/ci/{in_container => }/kubernetes/kind-cluster-conf.yaml (100%)
create mode 100644 scripts/ci/libraries/_build_images.sh
create mode 100644 scripts/ci/libraries/_initialization.sh
create mode 100644 scripts/ci/libraries/_kind.sh
create mode 100644 scripts/ci/libraries/_local_mounts.sh
create mode 100644 scripts/ci/libraries/_md5sum.sh
create mode 100644 scripts/ci/libraries/_parameters.sh
create mode 100644 scripts/ci/libraries/_permissions.sh
create mode 100644 scripts/ci/libraries/_push_pull_remove_images.sh
copy scripts/ci/{ci_docs.sh => libraries/_runs.sh} (51%)
mode change 100755 => 100644
create mode 100644 scripts/ci/libraries/_sanity_checks.sh
create mode 100644 scripts/ci/libraries/_spinner.sh
create mode 100644 scripts/ci/libraries/_start_end.sh
copy scripts/ci/{ci_bat_tests.sh => libraries/_verbosity.sh} (53%)
mode change 100755 => 100644
create mode 100755 scripts/ci/prepare_tool_scripts.sh
create mode 100644 tests/contrib/operators/test_hive_stats.py
create mode 100644 tests/contrib/operators/test_hive_to_mysql.py
create mode 100644 tests/contrib/operators/test_hive_to_samba.py
copy tests/{sensors/test_timedelta_sensor.py =>
contrib/sensors/test_hive_partition.py} (50%)
create mode 100644 tests/contrib/sensors/test_metastore_partition.py
create mode 100644 tests/contrib/sensors/test_webhdfs.py
copy tests/{contrib/operators/test_snowflake_operator.py => hive.py} (50%)
create mode 100644
tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
delete mode 100644 tests/runtime/__init__.py
create mode 100644 tests/test_utils/mock_hooks.py
create mode 100644 tests/test_utils/mock_process.py
