This is an automated email from the ASF dual-hosted git repository.
onikolas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new a2cbaecf625 Keycloak Breeze integration (#43240)
a2cbaecf625 is described below
commit a2cbaecf625e4e9bea22cea66b29f485ea53c964
Author: Niko Oliveira <[email protected]>
AuthorDate: Wed Oct 30 15:00:38 2024 -0700
Keycloak Breeze integration (#43240)
* Keycloak Breeze integration
Work is starting on multi-team Airflow, and this project has many
dependencies on a auth manager that can support authn and authz and also
support the changes to the auth manager api (upcoming). An option for
this is Keycloak.
This PR adds a Breeze integartion for Keycloak which creates a container
running keycloak, which uses the existing Postgres container as the DB
and disables the requirements for ssl/https/certificates to use the
Keycloak admin console (since this is only for development purposes not
production).
---
.pre-commit-config.yaml | 2 +
contributing-docs/testing/integration_tests.rst | 68 +++++++++++-----------
dev/breeze/doc/images/output-commands.svg | 4 +-
dev/breeze/doc/images/output_shell.svg | 4 +-
dev/breeze/doc/images/output_shell.txt | 2 +-
dev/breeze/doc/images/output_start-airflow.svg | 4 +-
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
.../images/output_testing_integration-tests.svg | 4 +-
.../images/output_testing_integration-tests.txt | 2 +-
dev/breeze/doc/images/output_testing_tests.svg | 4 +-
dev/breeze/doc/images/output_testing_tests.txt | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 18 ++++--
.../src/airflow_breeze/params/shell_params.py | 13 +++++
scripts/ci/docker-compose/integration-keycloak.yml | 62 ++++++++++++++++++++
.../ci/docker-compose/keycloak/init-keycloak-db.sh | 27 +++++++++
.../docker-compose/keycloak/keycloak-entrypoint.sh | 45 ++++++++++++++
16 files changed, 211 insertions(+), 52 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index ad4b2529b86..78a5b3645c0 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -658,6 +658,8 @@ repos:
^docs/exts/removemarktransform.py$|
^newsfragments/41761.significant.rst$|
^scripts/ci/pre_commit/vendor_k8s_json_schema.py$|
+ ^scripts/ci/docker-compose/integration-keycloak.yml$|
+ ^scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh$|
^tests/|
^providers/tests/|
^.pre-commit-config\.yaml$|
diff --git a/contributing-docs/testing/integration_tests.rst
b/contributing-docs/testing/integration_tests.rst
index 322298d4f00..ea9dfb7e952 100644
--- a/contributing-docs/testing/integration_tests.rst
+++ b/contributing-docs/testing/integration_tests.rst
@@ -49,39 +49,41 @@ The following integrations are available:
.. BEGIN AUTO-GENERATED INTEGRATION LIST
-+--------------+----------------------------------------------------+
-| Identifier | Description |
-+==============+====================================================+
-| cassandra | Integration required for Cassandra hooks. |
-+--------------+----------------------------------------------------+
-| celery | Integration required for Celery executor tests. |
-+--------------+----------------------------------------------------+
-| drill | Integration required for drill operator and hook. |
-+--------------+----------------------------------------------------+
-| kafka | Integration required for Kafka hooks. |
-+--------------+----------------------------------------------------+
-| kerberos | Integration that provides Kerberos authentication. |
-+--------------+----------------------------------------------------+
-| mongo | Integration required for MongoDB hooks. |
-+--------------+----------------------------------------------------+
-| mssql | Integration required for mssql hooks. |
-+--------------+----------------------------------------------------+
-| openlineage | Integration required for Openlineage hooks. |
-+--------------+----------------------------------------------------+
-| otel | Integration required for OTEL/opentelemetry hooks. |
-+--------------+----------------------------------------------------+
-| pinot | Integration required for Apache Pinot hooks. |
-+--------------+----------------------------------------------------+
-| qdrant | Integration required for Qdrant tests. |
-+--------------+----------------------------------------------------+
-| redis | Integration required for Redis tests. |
-+--------------+----------------------------------------------------+
-| statsd | Integration required for Statsd hooks. |
-+--------------+----------------------------------------------------+
-| trino | Integration required for Trino hooks. |
-+--------------+----------------------------------------------------+
-| ydb | Integration required for YDB tests. |
-+--------------+----------------------------------------------------+
++--------------+-------------------------------------------------------+
+| Identifier | Description |
++==============+=======================================================+
+| cassandra | Integration required for Cassandra hooks. |
++--------------+-------------------------------------------------------+
+| celery | Integration required for Celery executor tests. |
++--------------+-------------------------------------------------------+
+| drill | Integration required for drill operator and hook. |
++--------------+-------------------------------------------------------+
+| kafka | Integration required for Kafka hooks. |
++--------------+-------------------------------------------------------+
+| kerberos | Integration that provides Kerberos authentication. |
++--------------+-------------------------------------------------------+
+| keycloak | Integration for manual testing of multi-team Airflow. |
++--------------+-------------------------------------------------------+
+| mongo | Integration required for MongoDB hooks. |
++--------------+-------------------------------------------------------+
+| mssql | Integration required for mssql hooks. |
++--------------+-------------------------------------------------------+
+| openlineage | Integration required for Openlineage hooks. |
++--------------+-------------------------------------------------------+
+| otel | Integration required for OTEL/opentelemetry hooks. |
++--------------+-------------------------------------------------------+
+| pinot | Integration required for Apache Pinot hooks. |
++--------------+-------------------------------------------------------+
+| qdrant | Integration required for Qdrant tests. |
++--------------+-------------------------------------------------------+
+| redis | Integration required for Redis tests. |
++--------------+-------------------------------------------------------+
+| statsd | Integration required for Statsd hooks. |
++--------------+-------------------------------------------------------+
+| trino | Integration required for Trino hooks. |
++--------------+-------------------------------------------------------+
+| ydb | Integration required for YDB tests. |
++--------------+-------------------------------------------------------+
.. END AUTO-GENERATED INTEGRATION LIST'
diff --git a/dev/breeze/doc/images/output-commands.svg
b/dev/breeze/doc/images/output-commands.svg
index f80a72a9fbc..f67cd5795e4 100644
--- a/dev/breeze/doc/images/output-commands.svg
+++ b/dev/breeze/doc/images/output-commands.svg
@@ -305,8 +305,8 @@
</text><text class="breeze-help-r5" x="0" y="142" textLength="12.2"
clip-path="url(#breeze-help-line-5)">│</text><text class="breeze-help-r7"
x="414.8" y="142" textLength="732"
clip-path="url(#breeze-help-line-5)">(>3.9< | 3.10 | 3.11 | 3.12)                                </text><text
class="br [...]
</text><text class="breeze-help-r5" x="0" y="166.4" textLength="12.2"
clip-path="url(#breeze-help-line-6)">│</text><text class="breeze-help-r5"
x="414.8" y="166.4" textLength="732"
clip-path="url(#breeze-help-line-6)">[default: 3.9]                                          
[...]
</text><text class="breeze-help-r5" x="0" y="190.8" textLength="12.2"
clip-path="url(#breeze-help-line-7)">│</text><text class="breeze-help-r4"
x="24.4" y="190.8" textLength="158.6"
clip-path="url(#breeze-help-line-7)">--integration</text><text
class="breeze-help-r1" x="414.8" y="190.8" textLength="1024.8"
clip-path="url(#breeze-help-line-7)">Integration(s) to enable when running (can be more than one).       
[...]
-</text><text class="breeze-help-r5" x="0" y="215.2" textLength="12.2"
clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r7"
x="414.8" y="215.2" textLength="1024.8"
clip-path="url(#breeze-help-line-8)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | mongo | mssql </text><text
class="breeze-help-r5" x="1451.8" y="215.2" textLength="12.2"
clip-path="url(#breeze-help [...]
-</text><text class="breeze-help-r5" x="0" y="239.6" textLength="12.2"
clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r7"
x="414.8" y="239.6" textLength="1024.8"
clip-path="url(#breeze-help-line-9)">| openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb)               </text><text
class="breeze-help-r5" x [...]
+</text><text class="breeze-help-r5" x="0" y="215.2" textLength="12.2"
clip-path="url(#breeze-help-line-8)">│</text><text class="breeze-help-r7"
x="414.8" y="215.2" textLength="1024.8"
clip-path="url(#breeze-help-line-8)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | keycloak |    </text><text
class="breeze-help-r5" x="1451.8" y="215.2" textLength="12.2"
clip-path="url(#b [...]
+</text><text class="breeze-help-r5" x="0" y="239.6" textLength="12.2"
clip-path="url(#breeze-help-line-9)">│</text><text class="breeze-help-r7"
x="414.8" y="239.6" textLength="1024.8"
clip-path="url(#breeze-help-line-9)">mongo | mssql | openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb) </text><text
class="breeze-help-r5" x="1451.8" y="239.6" textLength="12.2"
clip-path="url(#b [...]
</text><text class="breeze-help-r5" x="0" y="264" textLength="12.2"
clip-path="url(#breeze-help-line-10)">│</text><text class="breeze-help-r4"
x="24.4" y="264" textLength="317.2"
clip-path="url(#breeze-help-line-10)">--standalone-dag-processor</text><text
class="breeze-help-r1" x="414.8" y="264" textLength="573.4"
clip-path="url(#breeze-help-line-10)">Run standalone dag processor for start-airflow.</text><text
class="breeze-help-r5" x="1451.8" y="264" textLength= [...]
</text><text class="breeze-help-r5" x="0" y="288.4" textLength="12.2"
clip-path="url(#breeze-help-line-11)">│</text><text class="breeze-help-r4"
x="24.4" y="288.4" textLength="244"
clip-path="url(#breeze-help-line-11)">--database-isolation</text><text
class="breeze-help-r1" x="414.8" y="288.4" textLength="475.8"
clip-path="url(#breeze-help-line-11)">Run airflow in database isolation mode.</text><text
class="breeze-help-r5" x="1451.8" y="288.4" textLength="12.2" c [...]
</text><text class="breeze-help-r5" x="0" y="312.8" textLength="1464"
clip-path="url(#breeze-help-line-12)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-help-r1" x="1464" y="312.8" textLength="12.2"
clip-path="url(#breeze-help-line-12)">
diff --git a/dev/breeze/doc/images/output_shell.svg
b/dev/breeze/doc/images/output_shell.svg
index b8eb1713c7f..58c359bd96b 100644
--- a/dev/breeze/doc/images/output_shell.svg
+++ b/dev/breeze/doc/images/output_shell.svg
@@ -544,8 +544,8 @@
</text><text class="breeze-shell-r5" x="0" y="190.8" textLength="12.2"
clip-path="url(#breeze-shell-line-7)">│</text><text class="breeze-shell-r7"
x="414.8" y="190.8" textLength="732"
clip-path="url(#breeze-shell-line-7)">(>3.9< | 3.10 | 3.11 | 3.12)                                </text><text
c [...]
</text><text class="breeze-shell-r5" x="0" y="215.2" textLength="12.2"
clip-path="url(#breeze-shell-line-8)">│</text><text class="breeze-shell-r5"
x="414.8" y="215.2" textLength="732"
clip-path="url(#breeze-shell-line-8)">[default: 3.9]                                          
[...]
</text><text class="breeze-shell-r5" x="0" y="239.6" textLength="12.2"
clip-path="url(#breeze-shell-line-9)">│</text><text class="breeze-shell-r4"
x="24.4" y="239.6" textLength="158.6"
clip-path="url(#breeze-shell-line-9)">--integration</text><text
class="breeze-shell-r1" x="414.8" y="239.6" textLength="1024.8"
clip-path="url(#breeze-shell-line-9)">Integration(s) to enable when running (can be more than one).      
[...]
-</text><text class="breeze-shell-r5" x="0" y="264" textLength="12.2"
clip-path="url(#breeze-shell-line-10)">│</text><text class="breeze-shell-r7"
x="414.8" y="264" textLength="1024.8"
clip-path="url(#breeze-shell-line-10)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | mongo | mssql </text><text
class="breeze-shell-r5" x="1451.8" y="264" textLength="12.2"
clip-path="url(#breeze-she [...]
-</text><text class="breeze-shell-r5" x="0" y="288.4" textLength="12.2"
clip-path="url(#breeze-shell-line-11)">│</text><text class="breeze-shell-r7"
x="414.8" y="288.4" textLength="1024.8"
clip-path="url(#breeze-shell-line-11)">| openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb)               </text><text
class="breeze-shel [...]
+</text><text class="breeze-shell-r5" x="0" y="264" textLength="12.2"
clip-path="url(#breeze-shell-line-10)">│</text><text class="breeze-shell-r7"
x="414.8" y="264" textLength="1024.8"
clip-path="url(#breeze-shell-line-10)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | keycloak |    </text><text
class="breeze-shell-r5" x="1451.8" y="264" textLength="12.2" clip-path="url(#
[...]
+</text><text class="breeze-shell-r5" x="0" y="288.4" textLength="12.2"
clip-path="url(#breeze-shell-line-11)">│</text><text class="breeze-shell-r7"
x="414.8" y="288.4" textLength="1024.8"
clip-path="url(#breeze-shell-line-11)">mongo | mssql | openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb) </text><text
class="breeze-shell-r5" x="1451.8" y="288.4" textLength="12.2" clip-path= [...]
</text><text class="breeze-shell-r5" x="0" y="312.8" textLength="12.2"
clip-path="url(#breeze-shell-line-12)">│</text><text class="breeze-shell-r4"
x="24.4" y="312.8" textLength="317.2"
clip-path="url(#breeze-shell-line-12)">--standalone-dag-processor</text><text
class="breeze-shell-r1" x="414.8" y="312.8" textLength="573.4"
clip-path="url(#breeze-shell-line-12)">Run standalone dag processor for start-airflow.</text><text
class="breeze-shell-r5" x="1451.8" y="312 [...]
</text><text class="breeze-shell-r5" x="0" y="337.2" textLength="12.2"
clip-path="url(#breeze-shell-line-13)">│</text><text class="breeze-shell-r4"
x="24.4" y="337.2" textLength="244"
clip-path="url(#breeze-shell-line-13)">--database-isolation</text><text
class="breeze-shell-r1" x="414.8" y="337.2" textLength="475.8"
clip-path="url(#breeze-shell-line-13)">Run airflow in database isolation mode.</text><text
class="breeze-shell-r5" x="1451.8" y="337.2" textLength=" [...]
</text><text class="breeze-shell-r5" x="0" y="361.6" textLength="1464"
clip-path="url(#breeze-shell-line-14)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-shell-r1" x="1464" y="361.6" textLength="12.2"
clip-path="url(#breeze-shell-line-14)">
diff --git a/dev/breeze/doc/images/output_shell.txt
b/dev/breeze/doc/images/output_shell.txt
index 051dd34cd10..348aac661fa 100644
--- a/dev/breeze/doc/images/output_shell.txt
+++ b/dev/breeze/doc/images/output_shell.txt
@@ -1 +1 @@
-fd70e0f17940f32fbc0579e8f77fc6c4
+fef5a76133d85a06a67932b9c394b0e1
diff --git a/dev/breeze/doc/images/output_start-airflow.svg
b/dev/breeze/doc/images/output_start-airflow.svg
index e4336a4f931..4cdfd78ad47 100644
--- a/dev/breeze/doc/images/output_start-airflow.svg
+++ b/dev/breeze/doc/images/output_start-airflow.svg
@@ -408,8 +408,8 @@
</text><text class="breeze-start-airflow-r5" x="0" y="239.6" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-9)">│</text><text
class="breeze-start-airflow-r5" x="414.8" y="239.6" textLength="732"
clip-path="url(#breeze-start-airflow-line-9)">[default: 3.9]                                    
[...]
</text><text class="breeze-start-airflow-r5" x="0" y="264" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-10)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="264" textLength="122"
clip-path="url(#breeze-start-airflow-line-10)">--platform</text><text
class="breeze-start-airflow-r1" x="414.8" y="264" textLength="329.4"
clip-path="url(#breeze-start-airflow-line-10)">Platform for Airflow image.</text><text
class="breeze-start-airflow-r7" x="756.4" y="26 [...]
</text><text class="breeze-start-airflow-r5" x="0" y="288.4" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-11)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="288.4" textLength="158.6"
clip-path="url(#breeze-start-airflow-line-11)">--integration</text><text
class="breeze-start-airflow-r1" x="414.8" y="288.4" textLength="1024.8"
clip-path="url(#breeze-start-airflow-line-11)">Integration(s) to enable when running (can be more 
[...]
-</text><text class="breeze-start-airflow-r5" x="0" y="312.8" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-12)">│</text><text
class="breeze-start-airflow-r7" x="414.8" y="312.8" textLength="1024.8"
clip-path="url(#breeze-start-airflow-line-12)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | mongo | mssql </text><text
class="breeze-start-airflow-r5" x="1451.8" y="312.8 [...]
-</text><text class="breeze-start-airflow-r5" x="0" y="337.2" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-13)">│</text><text
class="breeze-start-airflow-r7" x="414.8" y="337.2" textLength="1024.8"
clip-path="url(#breeze-start-airflow-line-13)">| openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb)               
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="312.8" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-12)">│</text><text
class="breeze-start-airflow-r7" x="414.8" y="312.8" textLength="1024.8"
clip-path="url(#breeze-start-airflow-line-12)">(all | all-testable | cassandra | celery | drill | kafka | kerberos | keycloak |    </text><text
class="breeze-start-airflow-r5" x="1451.8 [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="337.2" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-13)">│</text><text
class="breeze-start-airflow-r7" x="414.8" y="337.2" textLength="1024.8"
clip-path="url(#breeze-start-airflow-line-13)">mongo | mssql | openlineage | otel | pinot | qdrant | redis | statsd | trino | ydb) </text><text
class="breeze-start-airflow-r5" x="1451.8 [...]
</text><text class="breeze-start-airflow-r5" x="0" y="361.6" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-14)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="361.6" textLength="317.2"
clip-path="url(#breeze-start-airflow-line-14)">--standalone-dag-processor</text><text
class="breeze-start-airflow-r1" x="414.8" y="361.6" textLength="573.4"
clip-path="url(#breeze-start-airflow-line-14)">Run standalone dag processor for start-airflow.</text
[...]
</text><text class="breeze-start-airflow-r5" x="0" y="386" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-15)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="386" textLength="244"
clip-path="url(#breeze-start-airflow-line-15)">--database-isolation</text><text
class="breeze-start-airflow-r1" x="414.8" y="386" textLength="475.8"
clip-path="url(#breeze-start-airflow-line-15)">Run airflow in database isolation mode.</text><text
class="breeze-s [...]
</text><text class="breeze-start-airflow-r5" x="0" y="410.4" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-16)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="410.4" textLength="231.8"
clip-path="url(#breeze-start-airflow-line-16)">--load-example-dags</text><text
class="breeze-start-airflow-r6" x="366" y="410.4" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-16)">-e</text><text
class="breeze-start-airflow-r1" x="414.8" y="410.4" textLength="780.8" cl [...]
diff --git a/dev/breeze/doc/images/output_start-airflow.txt
b/dev/breeze/doc/images/output_start-airflow.txt
index 5811c7ec666..4618c5d3b44 100644
--- a/dev/breeze/doc/images/output_start-airflow.txt
+++ b/dev/breeze/doc/images/output_start-airflow.txt
@@ -1 +1 @@
-e63a3289a1be34b82c28b606dee0c472
+02160e5d799a77830ac522c628e90aed
diff --git a/dev/breeze/doc/images/output_testing_integration-tests.svg
b/dev/breeze/doc/images/output_testing_integration-tests.svg
index 07a2ef88b13..7bac13970e4 100644
--- a/dev/breeze/doc/images/output_testing_integration-tests.svg
+++ b/dev/breeze/doc/images/output_testing_integration-tests.svg
@@ -218,8 +218,8 @@
</text><text class="breeze-testing-integration-tests-r5" x="0" y="361.6"
textLength="24.4"
clip-path="url(#breeze-testing-integration-tests-line-14)">╭─</text><text
class="breeze-testing-integration-tests-r5" x="24.4" y="361.6"
textLength="219.6"
clip-path="url(#breeze-testing-integration-tests-line-14)"> Test environment </text><text
class="breeze-testing-integration-tests-r5" x="244" y="361.6"
textLength="1195.6" clip-path="url(#breeze-testing-integration-tests-line-14)"
[...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="386"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-15)">│</text><text
class="breeze-testing-integration-tests-r4" x="24.4" y="386" textLength="158.6"
clip-path="url(#breeze-testing-integration-tests-line-15)">--integration</text><text
class="breeze-testing-integration-tests-r1" x="610" y="386" textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-15)">Integration(s) to 
[...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="410.4"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-16)">│</text><text
class="breeze-testing-integration-tests-r6" x="610" y="410.4"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-16)">(all | all-testable | cassandra | celery | drill | kafka | kerberos </text><text
class="breeze-testing-integration-tests-r5 [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="434.8"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-17)">│</text><text
class="breeze-testing-integration-tests-r6" x="610" y="434.8"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-17)">| mongo | mssql | openlineage | otel | pinot | qdrant | redis |     </text><text
class="breez [...]
-</text><text class="breeze-testing-integration-tests-r5" x="0" y="459.2"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-18)">│</text><text
class="breeze-testing-integration-tests-r6" x="610" y="459.2"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-18)">statsd | trino | ydb)                        
[...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="434.8"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-17)">│</text><text
class="breeze-testing-integration-tests-r6" x="610" y="434.8"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-17)">| keycloak | mongo | mssql | openlineage | otel | pinot | qdrant |  </text><text
class="breeze-testing-integ [...]
+</text><text class="breeze-testing-integration-tests-r5" x="0" y="459.2"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-18)">│</text><text
class="breeze-testing-integration-tests-r6" x="610" y="459.2"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-18)">redis | statsd | trino | ydb)                     
[...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="483.6"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-19)">│</text><text
class="breeze-testing-integration-tests-r4" x="24.4" y="483.6"
textLength="109.8"
clip-path="url(#breeze-testing-integration-tests-line-19)">--backend</text><text
class="breeze-testing-integration-tests-r7" x="561.2" y="483.6"
textLength="24.4"
clip-path="url(#breeze-testing-integration-tests-line-19)">-b</text><text
class="br [...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="508"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-20)">│</text><text
class="breeze-testing-integration-tests-r1" x="610" y="508" textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-20)">an invalid database configuration, meaning there will be no database</text><text
class="breeze-testing-integration-tests-r5" x="1451.8" y="508" tex [...]
</text><text class="breeze-testing-integration-tests-r5" x="0" y="532.4"
textLength="12.2"
clip-path="url(#breeze-testing-integration-tests-line-21)">│</text><text
class="breeze-testing-integration-tests-r1" x="610" y="532.4"
textLength="829.6"
clip-path="url(#breeze-testing-integration-tests-line-21)">available, and any attempts to connect to the Airflow database will </text><text
class="breeze-testing-integration-tests-r5" x="1451. [...]
diff --git a/dev/breeze/doc/images/output_testing_integration-tests.txt
b/dev/breeze/doc/images/output_testing_integration-tests.txt
index 81d45b246b5..3304c7f7602 100644
--- a/dev/breeze/doc/images/output_testing_integration-tests.txt
+++ b/dev/breeze/doc/images/output_testing_integration-tests.txt
@@ -1 +1 @@
-5436131180cd928292c8234d15e0496f
+633eca64e9397259c7400814ce4c2877
diff --git a/dev/breeze/doc/images/output_testing_tests.svg
b/dev/breeze/doc/images/output_testing_tests.svg
index c20e2ef16b2..ac7e81bcbd5 100644
--- a/dev/breeze/doc/images/output_testing_tests.svg
+++ b/dev/breeze/doc/images/output_testing_tests.svg
@@ -497,8 +497,8 @@
</text><text class="breeze-testing-tests-r5" x="0" y="1020.4"
textLength="24.4" clip-path="url(#breeze-testing-tests-line-41)">╭─</text><text
class="breeze-testing-tests-r5" x="24.4" y="1020.4" textLength="219.6"
clip-path="url(#breeze-testing-tests-line-41)"> Test environment </text><text
class="breeze-testing-tests-r5" x="244" y="1020.4" textLength="1195.6"
clip-path="url(#breeze-testing-tests-line-41)">────────────────────────────────────────────────────────────────────
[...]
</text><text class="breeze-testing-tests-r5" x="0" y="1044.8"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-42)">│</text><text
class="breeze-testing-tests-r4" x="24.4" y="1044.8" textLength="158.6"
clip-path="url(#breeze-testing-tests-line-42)">--integration</text><text
class="breeze-testing-tests-r1" x="610" y="1044.8" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-42)">Integration(s) to enable when running (can be more 
[...]
</text><text class="breeze-testing-tests-r5" x="0" y="1069.2"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-43)">│</text><text
class="breeze-testing-tests-r7" x="610" y="1069.2" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-43)">(all | all-testable | cassandra | celery | drill | kafka | kerberos </text><text
class="breeze-testing-tests-r5" x="1451.8" y="1069.2" textLength="12.2"
clip-path="url(# [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="1093.6"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-44)">│</text><text
class="breeze-testing-tests-r7" x="610" y="1093.6" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-44)">| mongo | mssql | openlineage | otel | pinot | qdrant | redis |     </text><text
class="breeze-testing-tests-r5" x="1451.8" y="1093.6" text [...]
-</text><text class="breeze-testing-tests-r5" x="0" y="1118" textLength="12.2"
clip-path="url(#breeze-testing-tests-line-45)">│</text><text
class="breeze-testing-tests-r7" x="610" y="1118" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-45)">statsd | trino | ydb)                                 
[...]
+</text><text class="breeze-testing-tests-r5" x="0" y="1093.6"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-44)">│</text><text
class="breeze-testing-tests-r7" x="610" y="1093.6" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-44)">| keycloak | mongo | mssql | openlineage | otel | pinot | qdrant |  </text><text
class="breeze-testing-tests-r5" x="1451.8" y="1093.6" textLength="12.2" c [...]
+</text><text class="breeze-testing-tests-r5" x="0" y="1118" textLength="12.2"
clip-path="url(#breeze-testing-tests-line-45)">│</text><text
class="breeze-testing-tests-r7" x="610" y="1118" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-45)">redis | statsd | trino | ydb)                              
[...]
</text><text class="breeze-testing-tests-r5" x="0" y="1142.4"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-46)">│</text><text
class="breeze-testing-tests-r4" x="24.4" y="1142.4" textLength="109.8"
clip-path="url(#breeze-testing-tests-line-46)">--backend</text><text
class="breeze-testing-tests-r6" x="561.2" y="1142.4" textLength="24.4"
clip-path="url(#breeze-testing-tests-line-46)">-b</text><text
class="breeze-testing-tests-r1" x="610" y="1142.4" textLength="829.6" clip-pat
[...]
</text><text class="breeze-testing-tests-r5" x="0" y="1166.8"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-47)">│</text><text
class="breeze-testing-tests-r1" x="610" y="1166.8" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-47)">an invalid database configuration, meaning there will be no database</text><text
class="breeze-testing-tests-r5" x="1451.8" y="1166.8" textLength="12.2"
clip-path="url(#breeze-testing-tests [...]
</text><text class="breeze-testing-tests-r5" x="0" y="1191.2"
textLength="12.2" clip-path="url(#breeze-testing-tests-line-48)">│</text><text
class="breeze-testing-tests-r1" x="610" y="1191.2" textLength="829.6"
clip-path="url(#breeze-testing-tests-line-48)">available, and any attempts to connect to the Airflow database will </text><text
class="breeze-testing-tests-r5" x="1451.8" y="1191.2" textLength="12.2"
clip-path="url(#breeze-tes [...]
diff --git a/dev/breeze/doc/images/output_testing_tests.txt
b/dev/breeze/doc/images/output_testing_tests.txt
index d19f6cf5abd..bc12c1fb42a 100644
--- a/dev/breeze/doc/images/output_testing_tests.txt
+++ b/dev/breeze/doc/images/output_testing_tests.txt
@@ -1 +1 @@
-15002aa129ce25039921f800fb1cf744
+4e128855ff2df624e0fc59e229b0973d
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py
b/dev/breeze/src/airflow_breeze/global_constants.py
index 24a61dc651a..03fbe55a2e2 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -62,10 +62,14 @@ APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.9", "3.10", "3.11", "3.12"]
DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
ALLOWED_ARCHITECTURES = [Architecture.X86_64, Architecture.ARM]
-# Database Backends used when starting Breeze. The "none" value means that
invalid configuration
-# Is set and no database started - access to a database will fail.
-ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "none"]
-ALLOWED_PROD_BACKENDS = ["mysql", "postgres"]
+# Database Backends used when starting Breeze. The "none" value means that the
configuration is invalid.
+# No database will be started - access to a database will fail.
+SQLITE_BACKEND = "sqlite"
+MYSQL_BACKEND = "mysql"
+POSTGRES_BACKEND = "postgres"
+NONE_BACKEND = "none"
+ALLOWED_BACKENDS = [SQLITE_BACKEND, MYSQL_BACKEND, POSTGRES_BACKEND,
NONE_BACKEND]
+ALLOWED_PROD_BACKENDS = [MYSQL_BACKEND, POSTGRES_BACKEND]
DEFAULT_BACKEND = ALLOWED_BACKENDS[0]
CELERY_INTEGRATION = "celery"
TESTABLE_INTEGRATIONS = [
@@ -85,7 +89,11 @@ TESTABLE_INTEGRATIONS = [
DISABLE_TESTABLE_INTEGRATIONS_FROM_CI = [
"mssql",
]
-OTHER_INTEGRATIONS = ["statsd", "otel", "openlineage"]
+KEYCLOAK_INTEGRATION = "keycloak"
+STATSD_INTEGRATION = "statsd"
+OTEL_INTEGRATION = "otel"
+OPENLINEAGE_INTEGRATION = "openlineage"
+OTHER_INTEGRATIONS = [STATSD_INTEGRATION, OTEL_INTEGRATION,
OPENLINEAGE_INTEGRATION, KEYCLOAK_INTEGRATION]
ALLOWED_DEBIAN_VERSIONS = ["bookworm"]
ALL_INTEGRATIONS = sorted(
[
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py
b/dev/breeze/src/airflow_breeze/params/shell_params.py
index 2d3cadbf197..0c602643751 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -43,6 +43,7 @@ from airflow_breeze.global_constants import (
EDGE_EXECUTOR,
FASTAPI_API_HOST_PORT,
FLOWER_HOST_PORT,
+ KEYCLOAK_INTEGRATION,
MOUNT_ALL,
MOUNT_PROVIDERS_AND_TESTS,
MOUNT_REMOVE,
@@ -50,6 +51,7 @@ from airflow_breeze.global_constants import (
MOUNT_TESTS,
MSSQL_HOST_PORT,
MYSQL_HOST_PORT,
+ POSTGRES_BACKEND,
POSTGRES_HOST_PORT,
REDIS_HOST_PORT,
SSH_PORT,
@@ -665,3 +667,14 @@ class ShellParams:
self.airflow_constraints_reference =
self.default_constraints_branch
if self.providers_constraints_reference == "default":
self.providers_constraints_reference =
self.default_constraints_branch
+
+ if (
+ self.backend
+ and self.integration
+ and KEYCLOAK_INTEGRATION in self.integration
+ and not self.backend == POSTGRES_BACKEND
+ ):
+ get_console().print(
+ "[error]When using the Keycloak integration the backend must
be Postgres![/]\n"
+ )
+ sys.exit(2)
diff --git a/scripts/ci/docker-compose/integration-keycloak.yml
b/scripts/ci/docker-compose/integration-keycloak.yml
new file mode 100644
index 00000000000..7373c5fb617
--- /dev/null
+++ b/scripts/ci/docker-compose/integration-keycloak.yml
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+services:
+ keycloak:
+ image: quay.io/keycloak/keycloak:23.0.6
+ labels:
+ breeze.description: "Integration for manual testing of multi-team
Airflow."
+ entrypoint: /opt/keycloak/keycloak-entrypoint.sh
+ environment:
+ KC_HOSTNAME: localhost
+ KC_HOSTNAME_PORT: 48080
+ KC_HOSTNAME_STRICT_BACKCHANNEL: false
+ KC_HTTP_ENABLED: true
+ KC_HOSTNAME_STRICT: true
+
+ KEYCLOAK_ADMIN: admin
+ KEYCLOAK_ADMIN_PASSWORD: admin
+
+ KC_DB: postgres
+ KC_DB_URL: jdbc:postgresql://postgres/keycloak
+ KC_DB_USERNAME: keycloak
+ KC_DB_PASSWORD: keycloak
+ ports:
+ - 48080:48080
+ restart: always
+ depends_on:
+ postgres:
+ condition: service_healthy
+ volumes:
+ - ./keycloak/keycloak-entrypoint.sh:/opt/keycloak/keycloak-entrypoint.sh
+
+ postgres:
+ volumes:
+ -
./keycloak/init-keycloak-db.sh:/docker-entrypoint-initdb.d/init-keycloak-db.sh
+ environment:
+ KC_POSTGRES_DB: keycloak
+ KC_POSTGRES_USER: keycloak
+ KC_POSTGRES_PASSWORD: keycloak
+ healthcheck:
+ test: ["CMD", "psql", "-h", "localhost", "-U", "keycloak"]
+ interval: 10s
+ timeout: 10s
+ retries: 5
+
+ airflow:
+ depends_on:
+ - keycloak
diff --git a/scripts/ci/docker-compose/keycloak/init-keycloak-db.sh
b/scripts/ci/docker-compose/keycloak/init-keycloak-db.sh
new file mode 100755
index 00000000000..47df6aede20
--- /dev/null
+++ b/scripts/ci/docker-compose/keycloak/init-keycloak-db.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eu
+
+psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" > /dev/null <<-EOSQL
+ CREATE USER ${KC_POSTGRES_USER};
+ ALTER USER ${KC_POSTGRES_USER} WITH PASSWORD '${KC_POSTGRES_PASSWORD}';
+ CREATE DATABASE ${KC_POSTGRES_DB};
+ GRANT ALL PRIVILEGES ON DATABASE ${KC_POSTGRES_DB} TO ${KC_POSTGRES_USER};
+EOSQL
diff --git a/scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh
b/scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh
new file mode 100755
index 00000000000..e699d858346
--- /dev/null
+++ b/scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# We exit in case cd fails
+cd /opt/keycloak/bin/ || exit
+
+http_port="${KC_HOSTNAME_PORT}"
+
+# Start Keycloak in the background
+./kc.sh start-dev --http-port="$http_port" &
+
+# Wait for Keycloak to be ready
+echo "Waiting for Keycloak to start on port $http_port..."
+while ! (echo > /dev/tcp/localhost/"$http_port") 2>/dev/null; do
+ echo "keycloak still not started"
+ sleep 5
+done
+sleep 3
+echo "Keycloak is running (probably...)"
+
+# The below commands are used to disable the ssl requirement to use the admin
panel of keycloak
+echo "Configuring admin console access without ssl/https"
+# Get credentials to make the below update to the realm settings
+./kcadm.sh config credentials --server http://localhost:"$http_port"; --realm
master --user admin --password admin
+./kcadm.sh update realms/master -s sslRequired=NONE --server
http://localhost:"$http_port";
+echo "Configuring complete!"
+
+# Keep the container running
+wait
