This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1d0a8f7cdac add exasol operators doc and example & refactor trino
(#49679)
1d0a8f7cdac is described below
commit 1d0a8f7cdac0f420e292955fb846fc8306e54ade
Author: Aaron Chen <[email protected]>
AuthorDate: Wed Apr 30 00:49:23 2025 -0700
add exasol operators doc and example & refactor trino (#49679)
* add exasol operators doc, example and refactor trino
* fix lint issue
* fix lint issue #2
* fix code duplicate issue
* fix lint issue #3
* fix CI Static Checks
---
providers/exasol/docs/index.rst | 14 ++++
providers/exasol/docs/operators.rst | 79 +++++++++++++++++++
providers/exasol/tests/system/__init__.py | 16 ++++
providers/exasol/tests/system/exasol/__init__.py | 16 ++++
.../exasol/tests/system/exasol/example_exasol.py | 91 ++++++++++++++++++++++
.../docs/{operators/transfer => }/gcs_to_trino.rst | 0
providers/trino/docs/index.rst | 4 +-
.../docs/{operators/trino.rst => operators.rst} | 0
providers/trino/docs/operators/index.rst | 25 ------
9 files changed, 218 insertions(+), 27 deletions(-)
diff --git a/providers/exasol/docs/index.rst b/providers/exasol/docs/index.rst
index d7755960f01..cf33cd9c50e 100644
--- a/providers/exasol/docs/index.rst
+++ b/providers/exasol/docs/index.rst
@@ -29,6 +29,13 @@
Changelog <changelog>
Security <security>
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Guides
+
+ Operators <operators>
+
.. toctree::
:hidden:
:maxdepth: 1
@@ -44,6 +51,13 @@
PyPI Repository <https://pypi.org/project/apache-airflow-providers-exasol/>
Installing from sources <installing-providers-from-sources>
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: System tests
+
+ System Tests <_api/tests/system/exasol/index>
+
.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE
OVERWRITTEN AT RELEASE TIME!
diff --git a/providers/exasol/docs/operators.rst
b/providers/exasol/docs/operators.rst
new file mode 100644
index 00000000000..4d8e6f2c0fb
--- /dev/null
+++ b/providers/exasol/docs/operators.rst
@@ -0,0 +1,79 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:ExasolOperator:
+
+SQLExecuteQueryOperator to connect to Exasol
+====================================================
+
+Use the
:class:`SQLExecuteQueryOperator<airflow.providers.common.sql.operators.sql>` to
execute
+SQL commands in an `Exasol <https://www.exasol.com/>`__ database.
+
+.. note::
+ Previously, an ``ExasolOperator`` was used to perform this kind of
operation.
+ After deprecation this has been removed. Please use
``SQLExecuteQueryOperator`` instead.
+
+.. note::
+ Make sure you have installed the ``apache-airflow-providers-exasol``
package
+ and its dependency ``pyexasol`` to enable Exasol support.
+
+Using the Operator
+^^^^^^^^^^^^^^^^^^
+
+Use the ``conn_id`` argument to connect to your Exasol instance where
+the connection metadata is structured as follows:
+
+.. list-table:: Exasol Airflow Connection Metadata
+ :widths: 25 25
+ :header-rows: 1
+
+ * - Parameter
+ - Input
+ * - Host: string
+ - Exasol hostname, container alias or IP address
+ * - Schema: string
+ - Default schema name (e.g. ``TEST``) (optional)
+ * - Login: string
+ - Exasol username (e.g. ``TEST`` or ``sys``)
+ * - Password: string
+ - Exasol password
+ * - Port: int
+ - Exasol port (default: 8563)
+ * - Extra: JSON
+ - Additional connection configuration passed to *pyexasol*, such as:
+ ``{"encryption": false}`` or
+ ``{"encryption": true, "websocket_sslopt": {"cert_reqs": 0}}``
+
+An example usage of the SQLExecuteQueryOperator to connect to Exasol is as
follows:
+
+.. exampleinclude:: /../../exasol/tests/system/exasol/example_exasol.py
+ :language: python
+ :start-after: [START howto_operator_exasol]
+ :end-before: [END howto_operator_exasol]
+
+Reference
+^^^^^^^^^
+For further information, look at:
+
+* `Exasol Documentation <https://docs.exasol.com/>`__
+
+.. note::
+
+ Parameters provided directly via SQLExecuteQueryOperator() take precedence
+ over those specified in the Airflow connection metadata (such as ``schema``,
``login``, ``password``, etc).
diff --git a/providers/exasol/tests/system/__init__.py
b/providers/exasol/tests/system/__init__.py
new file mode 100644
index 00000000000..13a83393a91
--- /dev/null
+++ b/providers/exasol/tests/system/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/providers/exasol/tests/system/exasol/__init__.py
b/providers/exasol/tests/system/exasol/__init__.py
new file mode 100644
index 00000000000..13a83393a91
--- /dev/null
+++ b/providers/exasol/tests/system/exasol/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/providers/exasol/tests/system/exasol/example_exasol.py
b/providers/exasol/tests/system/exasol/example_exasol.py
new file mode 100644
index 00000000000..c477aec01f4
--- /dev/null
+++ b/providers/exasol/tests/system/exasol/example_exasol.py
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example DAG for the use of the SQLExecuteQueryOperator with Exasol.
+"""
+
+from __future__ import annotations
+
+import datetime
+
+from airflow import DAG
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.exasol.hooks.exasol import exasol_fetch_all_handler
+
+DAG_ID = "example_exasol"
+
+with DAG(
+ dag_id=DAG_ID,
+ start_date=datetime.datetime(2025, 1, 1),
+ default_args={"conn_id": "my_exasol_conn", "handler":
exasol_fetch_all_handler},
+ schedule="@once",
+ catchup=False,
+) as dag:
+ # [START howto_operator_exasol]
+ create_table_exasol = SQLExecuteQueryOperator(
+ task_id="create_table_exasol",
+ sql="""
+ CREATE OR REPLACE TABLE exasol_example (
+ a VARCHAR(100),
+ b DECIMAL(18,0)
+ );
+ """,
+ )
+
+ alter_table_exasol = SQLExecuteQueryOperator(
+ task_id="alter_table_exasol",
+ sql="ALTER TABLE exasol_example ADD COLUMN c DECIMAL(18,0);",
+ )
+
+ insert_data_exasol = SQLExecuteQueryOperator(
+ task_id="insert_data_exasol",
+ sql="""
+ INSERT INTO exasol_example (a, b, c)
+ VALUES
+ ('a', 1, 1),
+ ('a', 2, 1),
+ ('b', 3, 1);
+ """,
+ )
+
+ select_data_exasol = SQLExecuteQueryOperator(
+ task_id="select_data_exasol",
+ sql="SELECT * FROM exasol_example;",
+ )
+
+ drop_table_exasol = SQLExecuteQueryOperator(
+ task_id="drop_table_exasol",
+ sql="DROP TABLE exasol_example;",
+ )
+ # [END howto_operator_exasol]
+
+ (
+ create_table_exasol
+ >> alter_table_exasol
+ >> insert_data_exasol
+ >> select_data_exasol
+ >> drop_table_exasol
+ )
+
+ from tests_common.test_utils.watcher import watcher
+
+ list(dag.tasks) >> watcher()
+
+from tests_common.test_utils.system_tests import get_test_run # noqa: E402
+
+# Needed to run the example DAG with pytest (see:
tests/system/README.md#run_via_pytest)
+test_run = get_test_run(dag)
diff --git a/providers/trino/docs/operators/transfer/gcs_to_trino.rst
b/providers/trino/docs/gcs_to_trino.rst
similarity index 100%
rename from providers/trino/docs/operators/transfer/gcs_to_trino.rst
rename to providers/trino/docs/gcs_to_trino.rst
diff --git a/providers/trino/docs/index.rst b/providers/trino/docs/index.rst
index 30cb1263cb1..d4563b46220 100644
--- a/providers/trino/docs/index.rst
+++ b/providers/trino/docs/index.rst
@@ -34,8 +34,8 @@
:maxdepth: 1
:caption: Guides
- TrinoTransferOperator types <operators/transfer/gcs_to_trino>
- TrinoOperator <operators/index>
+ TrinoTransferOperator types <gcs_to_trino>
+ Operators <operators>
Connection types <connections>
.. toctree::
diff --git a/providers/trino/docs/operators/trino.rst
b/providers/trino/docs/operators.rst
similarity index 100%
rename from providers/trino/docs/operators/trino.rst
rename to providers/trino/docs/operators.rst
diff --git a/providers/trino/docs/operators/index.rst
b/providers/trino/docs/operators/index.rst
deleted file mode 100644
index d51bca0f7ae..00000000000
--- a/providers/trino/docs/operators/index.rst
+++ /dev/null
@@ -1,25 +0,0 @@
- .. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- .. http://www.apache.org/licenses/LICENSE-2.0
-
- .. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-Trino operator
-==============
-
-.. toctree::
- :maxdepth: 1
- :glob:
-
- *
