This is an automated email from the ASF dual-hosted git repository.
nsivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new f5fe1dce2720 feat(lakehouse): add Superset and Zeppelin Spark Connect
integration (#20467)
f5fe1dce2720 is described below
commit f5fe1dce27206447d7bab2dc60d7735ac3cba80b
Author: Shuo Cheng <[email protected]>
AuthorDate: Fri Nov 6 12:36:11 2026 +0800
feat(lakehouse): add Superset and Zeppelin Spark Connect integration
(#20467)
Add an experimental SQLAlchemy dialect and Superset engine spec under
hudi-lakehouse/superset-spark-connect, backed by the Foundry Spark ADBC
driver, so BI tools can query and write Hudi tables via Spark Connect
instead of the Thrift server.
- Dialect supports reflection, temporal filters and offset pagination;
it is pinned to the spark ADBC driver (driver/URI overrides rejected)
and rejects parameterized execution.
- Dockerfile extending Superset 6.1.0 with the adapter and native driver;
package accepts PyArrow >=16.1,<19.
- Document Zeppelin 0.12 access via the %python interpreter, plus a
local Superset/Zeppelin quickstart.
- Offline dialect tests in CI (PyArrow 16.1 and 18.1); opt-in live tests
for the adapter, Superset SQL Lab and Zeppelin.
Closes #20466
---
.github/workflows/agent_gateway_ci.yml | 18 ++
hudi-lakehouse/README.md | 7 +-
hudi-lakehouse/charts/hudi-spark-connect/README.md | 70 ++++++--
hudi-lakehouse/superset-spark-connect/Dockerfile | 24 +++
.../superset-spark-connect/QUICKSTART.md | 163 +++++++++++++++++
hudi-lakehouse/superset-spark-connect/README.md | 146 +++++++++++++++
.../hudi_spark_connect/__init__.py | 15 ++
.../hudi_spark_connect/dialect.py | 180 +++++++++++++++++++
.../hudi_spark_connect/engine_spec.py | 72 ++++++++
.../superset-spark-connect/pyproject.toml | 33 ++++
.../superset-spark-connect/test_dialect.py | 137 ++++++++++++++
.../superset-spark-connect/test_engine_spec.py | 83 +++++++++
hudi-lakehouse/superset-spark-connect/test_live.py | 138 +++++++++++++++
.../superset-spark-connect/test_superset_live.py | 196 +++++++++++++++++++++
.../superset-spark-connect/test_zeppelin_live.py | 93 ++++++++++
15 files changed, 1362 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/agent_gateway_ci.yml
b/.github/workflows/agent_gateway_ci.yml
index 6e16575e3514..d2074bb76f37 100644
--- a/.github/workflows/agent_gateway_ci.yml
+++ b/.github/workflows/agent_gateway_ci.yml
@@ -43,6 +43,24 @@ permissions:
contents: read
jobs:
+ superset-spark-connect-tests:
+ name: Superset Spark Connect dialect tests (PyArrow ${{ matrix.pyarrow }})
+ runs-on: ubuntu-24.04
+ strategy:
+ matrix:
+ pyarrow: ['16.1.0', '18.1.0']
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: '3.11'
+ cache: pip
+ cache-dependency-path:
hudi-lakehouse/superset-spark-connect/pyproject.toml
+ - name: Install adapter
+ run: pip install ./hudi-lakehouse/superset-spark-connect 'pyarrow==${{
matrix.pyarrow }}'
+ - name: Run offline dialect tests
+ run: python -m unittest discover -s
hudi-lakehouse/superset-spark-connect -p 'test_dialect.py' -v
+
gateway-tests:
name: gateway pytest + lint (python ${{ matrix.python }})
runs-on: ubuntu-24.04
diff --git a/hudi-lakehouse/README.md b/hudi-lakehouse/README.md
index d362eec1f2ca..974bb0751f0a 100644
--- a/hudi-lakehouse/README.md
+++ b/hudi-lakehouse/README.md
@@ -179,8 +179,11 @@ The gateway selects the official PySpark Connect client
with
See the [BI client
notes](charts/hudi-spark-connect/README.md#spark-connect-client-tutorials)
for Superset and Zeppelin connection requirements and ADBC limitations.
-Gateway integration and BI adapters/examples for Superset and other tools will
follow
-in subsequent PRs, tracked in
[#20232](https://github.com/apache/hudi/issues/20232).
+The [experimental Superset adapter](superset-spark-connect/README.md) uses
ADBC over
+Spark Connect. It includes opt-in SQL Lab and chart-data API checks; browser
+dashboard refresh is documented as a separate manual acceptance step.
+Agent and MCP gateway integration is tracked separately in
+[#20232](https://github.com/apache/hudi/issues/20232).
## Version compatibility notes
diff --git a/hudi-lakehouse/charts/hudi-spark-connect/README.md
b/hudi-lakehouse/charts/hudi-spark-connect/README.md
index fd7c7d45c207..ebbadb1d7eb0 100644
--- a/hudi-lakehouse/charts/hudi-spark-connect/README.md
+++ b/hudi-lakehouse/charts/hudi-spark-connect/README.md
@@ -314,17 +314,65 @@ do not require an LLM. Override the provider as described
in the gateway README.
## BI clients
-BI adapters/examples for Superset and other tools are tracked
-in [#20232](https://github.com/apache/hudi/issues/20232) and will be delivered
in follow-up PRs.
-The initial SQLAlchemy adapter is planned under `hudi-lakehouse`; upstreaming
can
-be evaluated separately after the integration stabilizes.
-
-- **Superset 5.0:** requires a custom SQLAlchemy dialect and engine spec for
ADBC;
- neither is included here. The built-in PyHive connector uses Thrift, not
Connect.
- See [Superset engine
specs](https://github.com/apache/superset/blob/5.0.0/superset/db_engine_specs/README.md).
-- **Zeppelin 0.12:** use ADBC from the `%python` interpreter, with the client
- packages above installed in its Python environment. See the
- [Python interpreter
guide](https://zeppelin.apache.org/docs/0.12.0/interpreter/python.html).
+For a local, step-by-step Superset and Zeppelin example, see the
+[BI quickstart](../../superset-spark-connect/QUICKSTART.md).
+
+The [experimental Superset adapter](../../superset-spark-connect/README.md)
+provides a SQLAlchemy dialect and Superset engine spec for Foundry ADBC. Follow
+its installation instructions and application-level checks. Those checks are
+separate from the endpoint acceptance tests above.
+Upstreaming the adapter can be evaluated separately after the integration
stabilizes.
+
+For Zeppelin 0.12, install the ADBC packages and Foundry driver into the Python
+environment used by the `%python` interpreter. Its official 0.12.0 image uses
+Python 3.9, so the manager 1.12.0 requirements above do not apply to that
image.
+The following tested versions are installed as that interpreter's user (or at
+system level where they are readable by that user):
+
+```bash
+python3 -m pip install adbc-driver-manager==1.8.0 pyarrow==18.1.0 dbc==0.3.1
+dbc install 'spark=0.2.1'
+```
+
+In a notebook, connect to the endpoint and render a query (replace the table
with
+an isolated test table):
+
+```python
+%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+with dbapi.connect(driver="spark", db_kwargs={"uri":
"spark://localhost:15002?api=connect&auth_type=none&tls=false"},
autocommit=True) as conn:
+ with conn.cursor() as cur:
+ cur.execute("SELECT region, SUM(amount) AS total FROM
connect_smoke_example.orders GROUP BY region ORDER BY region")
+ rows = cur.fetchall()
+ z.show(pd.DataFrame(rows, columns=["region", "total"]))
+```
+
+The hostname is relative to the Zeppelin interpreter process. Check the
rendered
+rows against the test fixture, then run INSERT, UPDATE and DELETE and SELECT
after
+each mutation in a disposable table. This uses `%python`, not native `%jdbc`.
+The exact rendered output depends on the installed interpreter. A local run of
+Zeppelin 0.12.0's x86_64 image (emulated on ARM), Python 3.9.15, ADBC manager
+1.8.0, PyArrow 18.1.0 and Foundry Spark ADBC 0.2.1 executed this path against a
+local Spark 4.1.3/Hudi 1.2.0 endpoint. It checked INSERT, UPDATE, DELETE and
+readback and rendered the final aggregation.
+
+For a repeatable API-driven check, install `requests` in the test environment,
+set `HUDI_ZEPPELIN_URL` and `HUDI_ZEPPELIN_CONNECT_URI` (the latter must be
+reachable from the Zeppelin interpreter), and run the CRUD and MERGE test:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p
'test_zeppelin_live.py' -v
+```
+
+Use only a disposable endpoint and notebook server. The Connect URI is stored
in
+the notebook source, so the test requires `auth_type=none` and rejects
credentials
+and secret-bearing options. The test creates a unique notebook and table and
+removes them; if execution is interrupted or the outcome
+is ambiguous, wait for the notebook to stop before manually cleaning up the
+printed table name. API-created paragraphs must include a table-results config;
+without it this Zeppelin version returned a table result via API but its
browser
+UI failed to render the paragraph.
[Foundry Spark ADBC 0.2.1](https://adbc-drivers.org/drivers/spark/) uses
autocommit.
diff --git a/hudi-lakehouse/superset-spark-connect/Dockerfile
b/hudi-lakehouse/superset-spark-connect/Dockerfile
new file mode 100644
index 000000000000..6f3d03e5d3a0
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/Dockerfile
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM apache/superset:6.1.0
+USER root
+COPY . /tmp/hudi-spark-connect-superset
+RUN uv pip install --python /app/.venv/bin/python
/tmp/hudi-spark-connect-superset 'dbc==0.3.1' 'pyarrow==18.1.0' \
+ && /app/.venv/bin/dbc install --level system 'spark=0.2.1' \
+ && mkdir -p /app/superset_home/.config \
+ && chown -R superset:superset /app/superset_home/.config \
+ && rm -rf /tmp/hudi-spark-connect-superset
+USER superset
diff --git a/hudi-lakehouse/superset-spark-connect/QUICKSTART.md
b/hudi-lakehouse/superset-spark-connect/QUICKSTART.md
new file mode 100644
index 000000000000..5ae9d4073393
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/QUICKSTART.md
@@ -0,0 +1,163 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+associated with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Try Superset and Zeppelin with Spark Connect
+
+This local example uses a disposable Hudi table. It targets Docker Desktop on
+macOS, Superset 6.1.0 and Zeppelin 0.12.0. Run shell commands from the root of
+an Apache Hudi checkout containing this adapter. You need Docker Desktop,
+minikube, kubectl, Helm and network access for images and dependencies.
Allocate
+at least 4 CPUs and 10 GiB of memory to minikube, plus resources for the BI
+containers. First startup can take several minutes. These settings are for a
+trusted local environment, not production.
+
+## 1. Start Spark Connect
+
+Start Docker Desktop, then run:
+
+```bash
+./hudi-lakehouse/scripts/quickstart.sh --spark-connect
+kubectl -n hudi-lakehouse rollout status deployment/hudi-spark-connect
--timeout=300s
+```
+
+In a separate terminal, leave the port-forward running:
+
+```bash
+kubectl -n hudi-lakehouse port-forward svc/hudi-spark-connect 15002:15002
+```
+
+If minikube reports a missing `minikube` Docker container while the script says
+it is running, the profile may be stale. Check `minikube status` and Docker
+Desktop's current context; repair or recreate the profile only after checking
+whether its existing data is needed. See the [endpoint
guide](../charts/hudi-spark-connect/README.md)
+for other deployment options.
+
+## 2. Start Superset and connect
+
+Build and initialize a disposable local Superset instance:
+
+```bash
+docker build -t hudi-superset-connect:local
hudi-lakehouse/superset-spark-connect
+docker run -d --name hudi-bi-superset -p 127.0.0.1:8088:8088 \
+ -e SUPERSET_SECRET_KEY=local-demo-secret-change-for-real-use \
+ -e SUPERSET_HOME=/tmp/superset-home \
+ --entrypoint /bin/sh hudi-superset-connect:local \
+ -c 'mkdir -p "$SUPERSET_HOME" &&
+ superset db upgrade &&
+ superset fab create-admin --username admin --firstname Local --lastname
Demo \
+ --email [email protected] --password admin &&
+ superset init &&
+ superset run -h 0.0.0.0 -p 8088'
+docker logs -f hudi-bi-superset
+```
+
+Once startup completes, press Ctrl-C to stop following the logs; this does not
+stop the container. Open <http://localhost:8088> and sign in with `admin` /
`admin`.
+Under **Settings → Database Connections**, click **+ Database**, and select
+**Apache Spark Connect (ADBC)**. On the **Basic** tab, enter a display name and
+the SQLAlchemy URI below. Before clicking **Connect**, open **Advanced → SQL
Lab**
+and enable **Allow DDL and DML** so the example can create and write its table.
+Use a disposable connection.
+
+```text
+sparkconnect://host.docker.internal:15002/default?api=connect&auth_type=none&tls=false
+```
+
+`host.docker.internal` routes from a Docker Desktop container to the Mac; it is
+not the Kubernetes service name. Test the connection and open **SQL Lab**.
+Run the following statements one at a time, selecting this connection and the
+`default` schema. The table name is reserved for this walkthrough: use a unique
+name if it already exists.
+
+```sql
+CREATE TABLE default.bi_connect_demo (id INT, region STRING, amount INT, ts
BIGINT)
+USING hudi TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
+```
+
+```sql
+INSERT INTO default.bi_connect_demo VALUES (1, 'East', 100, 1), (2, 'West',
200, 1);
+```
+
+```sql
+MERGE INTO default.bi_connect_demo t
+USING (SELECT 1 AS id, 'East' AS region, 175 AS amount, CAST(3 AS BIGINT) AS ts
+ UNION ALL SELECT 3, 'North', 300, CAST(3 AS BIGINT)) s
+ON t.id = s.id
+WHEN MATCHED THEN UPDATE SET *
+WHEN NOT MATCHED THEN INSERT *;
+```
+
+```sql
+SELECT id, region, amount, ts FROM default.bi_connect_demo ORDER BY id;
+```
+
+Expected rows: `(1, East, 175, 3)`, `(2, West, 200, 1)` and
+`(3, North, 300, 3)`. Create a dataset from `bi_connect_demo` and a chart
grouped
+by `region` to explore the data. If a write times out or disconnects, check the
+server and table before retrying: the write may still commit.
+
+## 3. Try Zeppelin
+
+Start Zeppelin and install the client in its Python environment:
+
+```bash
+docker run -d --name hudi-bi-zeppelin -p 127.0.0.1:8080:8080
apache/zeppelin:0.12.0
+docker exec --user root hudi-bi-zeppelin bash -c \
+ "python3 -m pip install adbc-driver-manager==1.8.0 pyarrow==18.1.0
dbc==0.3.1 && dbc install --level system 'spark=0.2.1' && chown -R 1000:0
/opt/zeppelin/.config"
+```
+
+The final `chown` makes the configuration directory created by `dbc` accessible
+to Zeppelin's user in the official image. If the driver registry download
fails,
+check network access and rerun the `docker exec` command before opening a
notebook.
+
+Open <http://localhost:8080>, create a notebook and run this `%python`
paragraph.
+It reads the same table created in Superset and renders the three rows:
+
+```python
+%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+
+uri = 'spark://host.docker.internal:15002?api=connect&auth_type=none&tls=false'
+with dbapi.connect(driver='spark', db_kwargs={'uri': uri}, autocommit=True) as
conn:
+ with conn.cursor() as cur:
+ cur.execute('SELECT id, region, amount, ts FROM
default.bi_connect_demo ORDER BY id')
+ z.show(pd.DataFrame(cur.fetchall(), columns=['id', 'region', 'amount',
'ts']))
+```
+
+The adapter tests also exercise Zeppelin writes and MERGE; see the
+[client guide](README.md) and [Zeppelin test
instructions](../charts/hudi-spark-connect/README.md#bi-clients).
+Zeppelin uses ADBC from `%python`; this example does not use `%jdbc`.
+
+## 4. Clean up
+
+In Superset SQL Lab, after all queries have completed, run:
+
+```sql
+DROP TABLE default.bi_connect_demo PURGE;
+```
+
+Remove only the two containers created above, and stop the port-forward with
+Ctrl-C in its terminal:
+
+```bash
+docker rm -f hudi-bi-superset hudi-bi-zeppelin
+```
+
+The minikube deployment remains available. See the [local-dev teardown
guide](../local-dev/README.md)
+before removing the cluster or its data.
diff --git a/hudi-lakehouse/superset-spark-connect/README.md
b/hudi-lakehouse/superset-spark-connect/README.md
new file mode 100644
index 000000000000..c2023dcfa6dc
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/README.md
@@ -0,0 +1,146 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Spark Connect adapter for Superset
+
+This experimental package provides a SQLAlchemy dialect and a Superset database
+engine spec for the Foundry Spark ADBC driver. It uses Spark Connect, not the
+Spark Thrift Server. It is independent of Hudi-specific SQL and can query other
+Spark tables. The repository's Connect deployment instructions are in
+[the chart README](../charts/hudi-spark-connect/README.md).
+
+For a step-by-step local walkthrough, including Superset and Zeppelin, see the
+[BI quickstart](QUICKSTART.md).
+
+Install the package and native driver in **every** Superset web and worker
+runtime. The driver must be accessible to the user running Superset, not just
+installed in the build user's home. The initial target is Superset 6.1.0,
+Python 3.11, SQLAlchemy 1.4.54, ADBC manager 1.12.0, PyArrow 18.1.0 and
+Foundry Spark ADBC 0.2.1 (the official Superset 6.1.0 container used
+Python 3.10 in the SQL Lab API check):
+
+```bash
+pip install ./hudi-lakehouse/superset-spark-connect 'dbc==0.3.1'
+dbc install 'spark=0.2.1'
+```
+
+The supplied Dockerfile extends the official Superset 6.1.0 image, installs the
+adapter into its existing Python environment and installs the native driver at
+system level for the Superset user:
+
+```bash
+docker build -t hudi-superset-connect:local
hudi-lakehouse/superset-spark-connect
+```
+
+Use that image for Superset web and worker processes. Configure a Superset
secret
+and metadata database using the normal Superset deployment instructions. The
image
+is a local integration example, not a hardened production deployment. It
replaces
+the base image's PyArrow 16.1.0 with 18.1.0; compatibility with unrelated
Superset
+features and connectors has not been established. The adapter package accepts
+`pyarrow>=16.1,<19`, so installing it into an existing Superset environment
does
+not require upgrading PyArrow 16.1.0. The Dockerfile pins 18.1.0 explicitly for
+the reproducible example.
+PyArrow 16.1.0 was also checked with ADBC manager 1.12.0 and Foundry Spark ADBC
+0.2.1: dialect and engine spec tests, Spark/Hudi live tests, and Superset SQL
Lab
+CRUD/MERGE and chart-data tests passed. This does not establish compatibility
+with every PyArrow version in the range or unrelated Superset connectors.
+
+Restart Superset after installation. Configure the database with a URI such as:
+
+```text
+sparkconnect://localhost:15002/default?api=connect&auth_type=none&tls=false
+```
+
+`localhost` is relative to the Superset process. For an in-cluster client use
+`hudi-spark-connect.hudi-lakehouse.svc:15002`; for a host client, use the
+port-forward in the chart README. This URI is only for a trusted local network.
+The adapter currently rejects credentials in the URL; authenticated endpoints
and
+catalog selection have not been validated. Its URI path selects the Spark
schema.
+The adapter always loads the `spark` driver installed by the Superset operator.
+The URI query parameter `driver` is rejected, including `driver=spark`;
database
+connections cannot select another native library. Install drivers only in
trusted,
+operator-controlled locations. Do not put credentials in query
+parameters, which can appear in logs.
+
+Use `superset test-db -c '{}'` with the URI to check dialect compatibility. In
SQL Lab,
+select the schema and table, run a grouped query, then create a dataset and
chart.
+To test direct writes, enable **Allow DML** on a disposable database
connection,
+execute INSERT, UPDATE and DELETE against a uniquely named test table, and run
a
+SELECT after each mutation. Refresh the chart after the writes. Disable Allow
DML
+when finished. The agent gateway remains read-only; these direct writes bypass
it.
+
+Parameterized execution and executemany are deliberately rejected: the tested
+Foundry 0.2.1 combination did not support parameter binding. Do not interpolate
+untrusted values into SQL. Driver errors or disconnects can occur after a write
+commits; inspect the table and timeline before retrying. Unsupported complex
Spark
+types are reported as unknown SQLAlchemy types. Query cancellation, large
results,
+authentication, concurrency and dashboard behaviour require separate
validation.
+
+Offline contract tests and an opt-in live test:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p
'test_dialect.py' -v
+HUDI_SUPERSET_CONNECT_TEST_URI='sparkconnect://localhost:15002/default?tls=false'
\
+ python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p
'test_live.py' -v
+```
+
+CI runs only the offline dialect tests, with PyArrow 16.1.0 and 18.1.0.
+The Superset engine spec tests are manual
+and require Superset 6.1.0. After building the example
+image above, run them against the checkout with:
+
+```bash
+docker run --rm -e SUPERSET_SECRET_KEY=local-test-only -e PYTHONPATH=/work \
+ -v "$PWD/hudi-lakehouse/superset-spark-connect:/work:ro" \
+ --entrypoint /app/.venv/bin/python hudi-superset-connect:local \
+ -m unittest discover -s /work -p 'test_engine_spec.py' -v
+```
+
+The live test creates a uniquely named database and Hudi table, checks
discovery,
+queries and CRUD, and drops only its own table and database. It needs a
disposable
+server with Hudi extensions and write permissions. A failed cleanup may leave
the
+named resources behind. Offline tests do not prove Superset UI or Spark
behaviour.
+
+To exercise Superset's SQL Lab API, install `requests` in the test environment,
+configure a disposable Superset database connection with Allow DML enabled, and
+set `HUDI_SUPERSET_URL`, `HUDI_SUPERSET_USER`, `HUDI_SUPERSET_PASSWORD` and
+`HUDI_SUPERSET_DATABASE_ID`. Optionally set `HUDI_SUPERSET_SCHEMA` (default
+`default`). Then run:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p
'test_superset_live.py' -v
+```
+
+The SQL Lab test creates a unique table and Superset dataset, checks MERGE
updates
+and inserts with readback, verifies chart data
+before and after a forced refresh following an update, then removes both
resources;
+it reports their names if cleanup fails. Use a disposable environment and
+verify database permissions before enabling DML. The test covers the
application
+API; it does not automate browser rendering or dashboard refresh.
+
+A local functional run used Superset 6.1.0, Spark Connect 4.1.3, Hudi 1.2.0
+and Foundry ADBC 0.2.1 against both a local container and the Helm-deployed
+endpoint. The SQL Lab API test passed queries, filtering and
+INSERT/UPDATE/DELETE with readback; the chart data API returned updated
aggregation
+results after a forced refresh. The SQL Lab interface displayed a simple
+query result. In a separate local-container run, a Superset dashboard table
showed
+East 100 and West 200; after an independent ADBC writer changed East to 150,
+clicking **Refresh dashboard** displayed East 150 and West 200.
+
+These checks used a single local-mode Spark driver and do not establish
distributed
+execution or production readiness.
diff --git
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py
new file mode 100644
index 000000000000..c5a5d9fae9de
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
new file mode 100644
index 000000000000..74fd2ad88d8f
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
@@ -0,0 +1,180 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""SQLAlchemy dialect for Spark Connect using the Foundry ADBC driver."""
+
+import re
+from urllib.parse import unquote, urlencode
+
+from sqlalchemy import types
+from sqlalchemy.engine import default
+from sqlalchemy.sql.compiler import SQLCompiler
+from sqlalchemy.sql.compiler import IdentifierPreparer
+
+
+class SparkIdentifierPreparer(IdentifierPreparer):
+ def __init__(self, dialect):
+ super().__init__(dialect, initial_quote="`", escape_quote="`")
+
+
+class SparkConnectCompiler(SQLCompiler):
+ def limit_clause(self, select, **kw):
+ text = ""
+ if select._limit_clause is not None:
+ text += "\n LIMIT " + self.process(select._limit_clause, **kw)
+ if select._offset_clause is not None:
+ text += "\n OFFSET " + self.process(select._offset_clause, **kw)
+ return text
+
+
+def spark_type(name):
+ name = name.strip().lower()
+ if name.startswith("decimal("):
+ match = re.fullmatch(r"decimal\((\d+),\s*(\d+)\)", name)
+ if match:
+ return types.Numeric(int(match.group(1)), int(match.group(2)))
+ base = name.split("(", 1)[0]
+ return {
+ "boolean": types.Boolean, "tinyint": types.SmallInteger,
+ "smallint": types.SmallInteger, "int": types.Integer,
+ "integer": types.Integer, "bigint": types.BigInteger,
+ "float": types.Float, "double": types.Float,
+ "string": types.String, "varchar": types.String, "char": types.String,
+ "date": types.Date, "timestamp": types.DateTime,
+ "timestamp_ntz": types.DateTime, "binary": types.LargeBinary,
+ }.get(base, types.NullType)()
+
+
+class SparkConnectDialect(default.DefaultDialect):
+ name = "sparkconnect"
+ driver = "adbc"
+ preparer = SparkIdentifierPreparer
+ statement_compiler = SparkConnectCompiler
+ supports_statement_cache = False
+ supports_native_boolean = True
+ default_paramstyle = "qmark"
+
+ @classmethod
+ def dbapi(cls):
+ from adbc_driver_manager import dbapi
+ return dbapi
+
+ def create_connect_args(self, url):
+ if not url.host:
+ raise ValueError("Spark Connect host is required")
+ if url.username or url.password:
+ raise ValueError("Credentials in SQLAlchemy URL are not supported")
+ options = dict(url.query)
+ if "driver" in options:
+ raise ValueError("The driver URI parameter is not supported;
install the 'spark' driver on the server")
+ options.setdefault("api", "connect")
+ options.setdefault("auth_type", "none")
+ if options["api"] != "connect":
+ raise ValueError("Only api=connect is supported")
+ host = f"[{url.host}]" if ":" in url.host else url.host
+ authority = f"{host}:{url.port or 15002}"
+ uri = f"spark://{authority}?{urlencode(options)}"
+ return [], {"driver": "spark", "db_kwargs": {"uri": uri},
"autocommit": True,
+ "_schema": unquote(url.database) if url.database else None}
+
+ def connect(self, *args, **kwargs):
+ if kwargs.pop("driver", "spark") != "spark":
+ raise ValueError("Only the 'spark' ADBC driver is supported")
+ schema = kwargs.pop("_schema", None)
+ connection = self.dbapi.connect(*args, driver="spark", **kwargs)
+ if schema:
+ try:
+ with connection.cursor() as cursor:
+ cursor.execute("USE " +
self.identifier_preparer.quote_identifier(schema))
+ except BaseException:
+ connection.close()
+ raise
+ return connection
+
+ def do_execute(self, cursor, statement, parameters, context=None):
+ if parameters:
+ raise NotImplementedError("Spark Connect ADBC parameter binding is
not supported")
+ cursor.execute(statement)
+
+ def do_executemany(self, cursor, statement, parameters, context=None):
+ raise NotImplementedError("Spark Connect ADBC parameter binding is not
supported")
+
+ def do_rollback(self, dbapi_connection):
+ # Spark does not support SQL transactions; connections use autocommit.
+ pass
+
+ def _get_default_schema_name(self, connection):
+ return connection.exec_driver_sql("SELECT current_database()").scalar()
+
+ def get_schema_names(self, connection, **kw):
+ return [row[0] for row in connection.exec_driver_sql("SHOW DATABASES")]
+
+ def _qualified(self, table, schema=None):
+ quote = self.identifier_preparer.quote_identifier
+ return ".".join(quote(part) for part in (schema, table) if part is not
None)
+
+ def get_table_names(self, connection, schema=None, **kw):
+ sql = "SHOW TABLES"
+ if schema:
+ sql += " IN " + self.identifier_preparer.quote_identifier(schema)
+ views = set(self.get_view_names(connection, schema=schema))
+ return [row[1] for row in connection.exec_driver_sql(sql) if not
row[2] and row[1] not in views]
+
+ def get_view_names(self, connection, schema=None, **kw):
+ sql = "SHOW VIEWS"
+ if schema:
+ sql += " IN " + self.identifier_preparer.quote_identifier(schema)
+ return [row[1] for row in connection.exec_driver_sql(sql) if not
row[2]]
+
+ def get_columns(self, connection, table_name, schema=None, **kw):
+ rows = connection.exec_driver_sql("DESCRIBE TABLE " +
self._qualified(table_name, schema))
+ columns = []
+ for row in rows:
+ name = row[0]
+ # DESCRIBE separates metadata sections with rows that have no type.
+ # A real, quoted column can contain whitespace or start with '#'.
+ if not name or not row[1]:
+ break
+ columns.append({"name": name, "type": spark_type(row[1]),
"nullable": True, "default": None})
+ return columns
+
+ def has_table(self, connection, table_name, schema=None, **kw):
+ if schema is None:
+ # Local temporary views are session-scoped and cannot be qualified
+ # with a database name, even though SHOW TABLES IN lists them.
+ if any(row[1] == table_name and row[2]
+ for row in connection.exec_driver_sql("SHOW TABLES")):
+ return True
+ return (table_name in self.get_table_names(connection, schema=schema)
+ or table_name in self.get_view_names(connection,
schema=schema))
+
+ def get_pk_constraint(self, connection, table_name, schema=None, **kw):
+ # Spark does not expose enforced SQL primary key constraints.
+ return {"constrained_columns": [], "name": None}
+
+ def get_foreign_keys(self, connection, table_name, schema=None, **kw):
+ return []
+
+ def get_indexes(self, connection, table_name, schema=None, **kw):
+ return []
+
+ def get_unique_constraints(self, connection, table_name, schema=None,
**kw):
+ return []
+
+ def get_check_constraints(self, connection, table_name, schema=None, **kw):
+ return []
+
+ def get_table_comment(self, connection, table_name, schema=None, **kw):
+ return {"text": None}
diff --git
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
new file mode 100644
index 000000000000..2e34a2f74cf2
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Superset integration for Spark SQL over ADBC Connect."""
+
+from urllib.parse import quote, unquote
+
+from superset.db_engine_specs.base import BaseEngineSpec
+from superset.db_engine_specs.spark import time_grain_expressions
+from superset.sql.parse import SQLGLOT_DIALECTS
+
+# Superset maps SQL parser dialects by engine name.
+# This internal registry is tested with Superset 6.1.0; revalidate on upgrades.
+SQLGLOT_DIALECTS["sparkconnect"] = SQLGLOT_DIALECTS["spark"]
+
+
+class SparkConnectEngineSpec(BaseEngineSpec):
+ engine = "sparkconnect"
+ engine_name = "Apache Spark Connect (ADBC)"
+ supports_dynamic_schema = True
+ supports_file_upload = False
+ _time_grain_expressions = time_grain_expressions
+ metadata = {
+ "description": "Apache Spark SQL over Spark Connect using ADBC.",
+ "logo": "apache-spark.png",
+ "homepage_url": "https://spark.apache.org/sql/",
+ "connection_string":
"sparkconnect://{hostname}:{port}/{database}?api=connect",
+ "default_port": 15002,
+ }
+
+ @classmethod
+ def convert_dttm(cls, target_type, dttm, db_extra=None):
+ target = target_type.strip().upper()
+ if target == "DATE":
+ return f"CAST('{dttm.date().isoformat()}' AS DATE)"
+ if target in ("DATETIME", "TIMESTAMP", "TIMESTAMP_NTZ"):
+ value = dttm.isoformat(sep=" ", timespec="microseconds")
+ spark_type = "TIMESTAMP" if target == "DATETIME" else target
+ return f"CAST('{value}' AS {spark_type})"
+ return None
+
+ @classmethod
+ def epoch_to_dttm(cls):
+ return "TIMESTAMP_SECONDS({col})"
+
+ @classmethod
+ def epoch_ms_to_dttm(cls):
+ return "TIMESTAMP_MILLIS({col})"
+
+ @classmethod
+ def adjust_engine_params(cls, uri, connect_args, catalog=None,
schema=None):
+ if catalog:
+ raise ValueError("Selecting a catalog is not supported")
+ if schema:
+ uri = uri.set(database=quote(schema, safe=""))
+ return uri, connect_args
+
+ @classmethod
+ def get_schema_from_engine_params(cls, sqlalchemy_uri, connect_args):
+ return unquote(sqlalchemy_uri.database) if sqlalchemy_uri.database
else None
diff --git a/hudi-lakehouse/superset-spark-connect/pyproject.toml
b/hudi-lakehouse/superset-spark-connect/pyproject.toml
new file mode 100644
index 000000000000..cbd9594fae5f
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/pyproject.toml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hudi-spark-connect-superset"
+version = "0.1.0"
+requires-python = ">=3.9"
+dependencies = ["SQLAlchemy>=1.4,<2", "adbc-driver-manager==1.12.0",
"pyarrow>=16.1,<19"]
+
+[project.entry-points."sqlalchemy.dialects"]
+sparkconnect = "hudi_spark_connect.dialect:SparkConnectDialect"
+
+[project.entry-points."superset.db_engine_specs"]
+sparkconnect = "hudi_spark_connect.engine_spec:SparkConnectEngineSpec"
+
+[tool.setuptools]
+packages = ["hudi_spark_connect"]
diff --git a/hudi-lakehouse/superset-spark-connect/test_dialect.py
b/hudi-lakehouse/superset-spark-connect/test_dialect.py
new file mode 100644
index 000000000000..253676eeb06b
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_dialect.py
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import unittest
+from unittest.mock import MagicMock, patch
+from urllib.parse import urlencode
+
+from sqlalchemy.engine.url import make_url
+from sqlalchemy import create_engine, types
+from sqlalchemy.dialects import registry
+from sqlalchemy import column, select
+from hudi_spark_connect.dialect import SparkConnectDialect, spark_type
+
+
+class DialectTest(unittest.TestCase):
+ def setUp(self):
+ self.dialect = SparkConnectDialect()
+
+ def test_connect_url_and_schema(self):
+ args, options = self.dialect.create_connect_args(make_url(
+ "sparkconnect://localhost:15002/a%60b?tls=false"))
+ self.assertEqual(args, [])
+ self.assertEqual(options["db_kwargs"]["uri"],
+
"spark://localhost:15002?tls=false&api=connect&auth_type=none")
+ self.dialect.dbapi = MagicMock()
+ conn = self.dialect.dbapi.connect.return_value
+ self.dialect.connect(*args, **options)
+
conn.cursor.return_value.__enter__.return_value.execute.assert_called_once_with("USE
`a``b`")
+ self.dialect.dbapi.connect.assert_called_once_with(
+ driver="spark", db_kwargs={"uri":
"spark://localhost:15002?tls=false&api=connect&auth_type=none"},
autocommit=True)
+
+ def test_driver_uri_override_rejected(self):
+ for driver in ("spark", "other_manifest", "/tmp/x.so", "../x",
r"C:\x.dll"):
+ with self.subTest(driver=driver),
self.assertRaisesRegex(ValueError, "driver URI parameter"):
+ self.dialect.create_connect_args(make_url(
+ "sparkconnect://localhost:15002/default?" +
urlencode({"driver": driver})))
+
+ def test_engine_connect_args_cannot_override_driver(self):
+ from adbc_driver_manager import dbapi
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ for options in ({"driver": "/tmp/evil.so"}, {"driver": "other"}):
+ with self.subTest(options=options), patch.object(dbapi, "connect")
as connect:
+ engine = create_engine("sparkconnect://localhost/default",
connect_args=options, module=dbapi)
+ with self.assertRaises(ValueError):
+ engine.raw_connection()
+ connect.assert_not_called()
+ engine.dispose()
+ with patch.object(dbapi, "connect",
side_effect=RuntimeError("connection attempted")) as connect:
+ engine = create_engine("sparkconnect://localhost/default",
module=dbapi)
+ with self.assertRaisesRegex(RuntimeError, "connection attempted"):
+ engine.raw_connection()
+ self.assertEqual(connect.call_args.kwargs["driver"], "spark")
+ engine.dispose()
+
+ def test_credentials_and_protocol_rejected(self):
+ for url in ("sparkconnect://u:p@host",
"sparkconnect://host?api=thrift"):
+ with self.subTest(url=url), self.assertRaises(ValueError):
+ self.dialect.create_connect_args(make_url(url))
+
+ def test_failed_schema_selection_closes_connection(self):
+ self.dialect.dbapi = MagicMock()
+ conn = self.dialect.dbapi.connect.return_value
+ conn.cursor.return_value.__enter__.return_value.execute.side_effect =
RuntimeError("bad schema")
+ with self.assertRaisesRegex(RuntimeError, "bad schema"):
+ self.dialect.connect(_schema="missing")
+ conn.close.assert_called_once()
+
+ def test_execute_never_binds_or_retries(self):
+ cursor = MagicMock()
+ self.dialect.do_execute(cursor, "SELECT 1", ())
+ cursor.execute.assert_called_once_with("SELECT 1")
+ with self.assertRaises(NotImplementedError):
+ self.dialect.do_execute(cursor, "SELECT ?", (1,))
+ with self.assertRaises(NotImplementedError):
+ self.dialect.do_executemany(cursor, "INSERT INTO t VALUES (?)",
[(1,)])
+ self.assertEqual(cursor.execute.call_count, 1)
+
+ def test_offset_without_limit_does_not_add_negative_limit(self):
+ stmt = select(column("id")).offset(5)
+ sql = str(stmt.compile(dialect=self.dialect,
compile_kwargs={"literal_binds": True}))
+ self.assertIn("OFFSET 5", sql)
+ self.assertNotIn("LIMIT", sql)
+ sql = str(stmt.limit(10).compile(
+ dialect=self.dialect, compile_kwargs={"literal_binds": True}))
+ self.assertIn("LIMIT 10", sql)
+ self.assertIn("OFFSET 5", sql)
+
+ def test_reflection_quoting_and_types(self):
+ conn = MagicMock()
+ conn.exec_driver_sql.side_effect = [
+ [("db", "v", False)], [("db", "t", False), ("db", "v", False),
("db", "tmp", True)],
+ [("id", "int", ""), ("amount", "decimal(18, 2)", ""), ("", "",
""), ("# Partition", "", "")],
+ ]
+ self.assertEqual(self.dialect.get_table_names(conn, 'a`b'), ["t"])
+ columns = self.dialect.get_columns(conn, "table", 'a`b')
+ self.assertEqual([x["name"] for x in columns], ["id", "amount"])
+ self.assertIsInstance(columns[0]["type"], types.Integer)
+ self.assertEqual((columns[1]["type"].precision,
columns[1]["type"].scale), (18, 2))
+ self.assertIn("`a``b`.`table`", conn.exec_driver_sql.call_args.args[0])
+ self.assertIsInstance(spark_type("struct<a:int>"), types.NullType)
+
+
+ def test_reflection_preserves_quoted_column_names(self):
+ conn = MagicMock()
+ conn.exec_driver_sql.return_value = [
+ (" leading ", "int", ""), ("#metric", "string", ""),
+ ("", "", ""), ("# Partition Information", "", ""),
+ ]
+ columns = self.dialect.get_columns(conn, "t")
+ self.assertEqual([column["name"] for column in columns], [" leading ",
"#metric"])
+
+ def test_has_table_finds_session_temporary_view(self):
+ conn = MagicMock()
+ conn.exec_driver_sql.return_value = [("", "temporary_view", True)]
+ self.assertTrue(self.dialect.has_table(conn, "temporary_view"))
+ conn.exec_driver_sql.assert_called_once_with("SHOW TABLES")
+
+ def test_has_table_does_not_bind_temporary_view_to_schema(self):
+ conn = MagicMock()
+ conn.exec_driver_sql.return_value = [("", "temporary_view", True)]
+ self.assertFalse(self.dialect.has_table(conn, "temporary_view",
schema="other"))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/hudi-lakehouse/superset-spark-connect/test_engine_spec.py
b/hudi-lakehouse/superset-spark-connect/test_engine_spec.py
new file mode 100644
index 000000000000..eb0bfbc2a1c4
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_engine_spec.py
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run with Superset installed; exercises its actual filter handling path."""
+import importlib.util
+import unittest
+from datetime import datetime
+
+
[email protected](importlib.util.find_spec('superset'), 'requires Superset')
+class SupersetEngineSpecTest(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ from superset.app import create_app
+ cls.app = create_app()
+
+ def test_schema_round_trip(self):
+ from unittest.mock import MagicMock
+ from sqlalchemy.engine.url import make_url
+ from hudi_spark_connect.dialect import SparkConnectDialect
+ from hudi_spark_connect.engine_spec import SparkConnectEngineSpec as
spec
+
+ for schema in ("a/b", "a`b", "a b", "a/b` c", "a%2Fb"):
+ with self.subTest(schema=schema):
+ uri, connect_args = spec.adjust_engine_params(
+
make_url("sparkconnect://localhost:15002/default?tls=false"), {}, schema=schema)
+ uri = make_url(str(uri))
+ self.assertEqual(spec.get_schema_from_engine_params(uri,
connect_args), schema)
+ dialect = SparkConnectDialect()
+ args, options = dialect.create_connect_args(uri)
+ self.assertEqual(options["_schema"], schema)
+ dialect.dbapi = MagicMock()
+ dialect.connect(*args, **options)
+ cursor =
dialect.dbapi.connect.return_value.cursor.return_value.__enter__.return_value
+ cursor.execute.assert_called_once_with("USE `" +
schema.replace("`", "``") + "`")
+
+ def test_temporal_filter_values_compile(self):
+ import sqlalchemy as sa
+ from superset.models.helpers import ExploreMixin
+ from superset.utils.core import FilterOperator, GenericDataType
+ from hudi_spark_connect.engine_spec import SparkConnectEngineSpec
+
+ for native, expected in [('DATE', "CAST('2023-11-14' AS DATE)"),
+ ('DATETIME', "CAST('2023-11-14
22:13:20.000000' AS TIMESTAMP)"),
+ ('TIMESTAMP', "CAST('2023-11-14
22:13:20.000000' AS TIMESTAMP)"),
+ ('TIMESTAMP_NTZ', "CAST('2023-11-14
22:13:20.000000' AS TIMESTAMP_NTZ)")]:
+ with self.subTest(native=native):
+ value = ExploreMixin.filter_values_handler(
+ [1700000000000], FilterOperator.EQUALS,
GenericDataType.TEMPORAL,
+ target_native_type=native,
db_engine_spec=SparkConnectEngineSpec)
+ sql = str(sa.select([sa.column('ts')]).where(sa.column('ts')
== value).compile(
+ compile_kwargs={'literal_binds': True}))
+ self.assertIn(expected, sql)
+
+ def test_epoch_expressions_and_unknown_type(self):
+ import sqlalchemy as sa
+ from hudi_spark_connect.engine_spec import SparkConnectEngineSpec as
spec
+ self.assertEqual(spec.epoch_to_dttm(), 'TIMESTAMP_SECONDS({col})')
+ self.assertEqual(spec.epoch_ms_to_dttm(), 'TIMESTAMP_MILLIS({col})')
+ self.assertIsNone(spec.convert_dttm('STRING', datetime(2026, 1, 1)))
+ with self.app.app_context():
+ for fmt, function in [('epoch_s', 'TIMESTAMP_SECONDS'),
+ ('epoch_ms', 'TIMESTAMP_MILLIS')]:
+ with self.subTest(fmt=fmt):
+ expr = spec.get_timestamp_expr(sa.column('ts'), fmt, 'P1D')
+ sql =
str(sa.select([expr]).compile(compile_kwargs={'literal_binds': True}))
+ self.assertIn("date_trunc('day', " + function + '(ts))',
sql)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/hudi-lakehouse/superset-spark-connect/test_live.py
b/hudi-lakehouse/superset-spark-connect/test_live.py
new file mode 100644
index 000000000000..ff270716eb56
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_live.py
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in live test: creates and removes a unique database on the target
server."""
+import os
+from datetime import datetime, timezone
+from decimal import Decimal
+import unittest
+import uuid
+
+from sqlalchemy import MetaData, Table, column, create_engine, inspect,
select, text
+from sqlalchemy.dialects import registry
+
+URI = os.environ.get("HUDI_SUPERSET_CONNECT_TEST_URI")
+
+
[email protected](URI, "set HUDI_SUPERSET_CONNECT_TEST_URI to a disposable
endpoint")
+class LiveDialectTest(unittest.TestCase):
+ def test_offset_only_compiled_sql(self):
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ engine = create_engine(URI)
+ try:
+ query = select(column("id")).select_from(text("range(10)"))\
+ .order_by(column("id")).offset(5)
+ sql = str(query.compile(dialect=engine.dialect,
compile_kwargs={"literal_binds": True}))
+ with engine.connect() as conn:
+ self.assertEqual(conn.exec_driver_sql(sql).fetchall(), [(5,),
(6,), (7,), (8,), (9,)])
+ finally:
+ engine.dispose()
+
+ def test_epoch_seconds_and_milliseconds(self):
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ engine = create_engine(URI)
+ try:
+ with engine.connect() as conn:
+ row = conn.exec_driver_sql(
+ "SELECT unix_micros(timestamp_seconds(1700000000)), "
+ "unix_micros(timestamp_millis(1700000000123)), "
+ "unix_micros(timestamp_millis(-1))"
+ ).fetchone()
+ self.assertEqual(tuple(row), (1700000000000000,
1700000000123000, -1000))
+ finally:
+ engine.dispose()
+
+ def test_hudi_read_write_and_reflection(self):
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ engine = create_engine(URI)
+ database = "connect_adapter_" + uuid.uuid4().hex
+ table = database + ".orders"
+ try:
+ with engine.connect() as conn:
+ conn.exec_driver_sql("CREATE DATABASE " + database)
+ try:
+ conn.exec_driver_sql(f"""CREATE TABLE {table} (id INT,
region STRING, amount INT, ts BIGINT, day DATE)
+ USING hudi TBLPROPERTIES (type='cow', primaryKey='id',
preCombineField='ts')""")
+ conn.exec_driver_sql(f"INSERT INTO {table} VALUES (1,
'East', 100, 1, DATE '2026-10-01'), (2, 'West', 200, 1, DATE '2026-10-01')")
+ self.assertIn(database, inspect(conn).get_schema_names())
+ self.assertIn('orders',
inspect(conn).get_table_names(database))
+ reflected = Table('orders', MetaData(), schema=database,
autoload_with=conn)
+ self.assertIn('amount', reflected.c)
+ columns = inspect(conn).get_columns('orders', database)
+ self.assertTrue({'id', 'region', 'amount', 'ts'} <=
{c['name'] for c in columns})
+ self.assertEqual(conn.exec_driver_sql(f"SELECT SUM(amount)
FROM {table}").scalar(), 300)
+ metadata = conn.exec_driver_sql(
+ f"SELECT _hoodie_commit_time, _hoodie_record_key FROM
{table} ORDER BY id"
+ ).fetchall()
+ self.assertEqual([row[1] for row in metadata], ['1', '2'])
+ self.assertTrue(all(row[0] for row in metadata))
+ self.assertEqual(conn.exec_driver_sql(
+ f"SELECT region, SUM(amount) FROM {table} WHERE
region='East' GROUP BY region"
+ ).fetchall(), [('East', 100)])
+ self.assertEqual(conn.exec_driver_sql(
+ f"SELECT CAST(date_trunc('day', day) AS STRING),
SUM(amount) "
+ f"FROM {table} GROUP BY date_trunc('day', day)"
+ ).fetchall(), [('2026-10-01 00:00:00', 300)])
+ conn.exec_driver_sql(f"UPDATE {table} SET amount=150, ts=2
WHERE id=1")
+ self.assertEqual(conn.exec_driver_sql(f"SELECT id, amount
FROM {table} ORDER BY id").fetchall(), [(1, 150), (2, 200)])
+ conn.exec_driver_sql(f"DELETE FROM {table} WHERE id=2")
+ self.assertEqual(conn.exec_driver_sql(f"SELECT id, amount
FROM {table}").fetchall(), [(1, 150)])
+ finally:
+ conn.exec_driver_sql(f"DROP TABLE IF EXISTS {table} PURGE")
+ conn.exec_driver_sql(f"DROP DATABASE IF EXISTS {database}")
+ finally:
+ engine.dispose()
+
+ def test_session_view_and_quoted_columns(self):
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ engine = create_engine(URI)
+ name = "connect_view_" + uuid.uuid4().hex
+ try:
+ with engine.connect() as conn:
+ try:
+ conn.exec_driver_sql(f"CREATE TEMP VIEW {name} AS SELECT 1
AS ` leading `, 2 AS `#metric`")
+ inspector = inspect(conn)
+ self.assertTrue(inspector.has_table(name))
+ self.assertFalse(inspector.has_table(name,
schema="default"))
+ self.assertEqual([column["name"] for column in
inspector.get_columns(name)],
+ [" leading ", "#metric"])
+ self.assertEqual(conn.exec_driver_sql(f"SELECT ` leading
`, `#metric` FROM {name}").fetchall(), [(1, 2)])
+ finally:
+ conn.exec_driver_sql(f"DROP VIEW IF EXISTS {name}")
+ finally:
+ engine.dispose()
+
+ def test_arrow_type_mapping(self):
+ registry.register("sparkconnect", "hudi_spark_connect.dialect",
"SparkConnectDialect")
+ engine = create_engine(URI)
+ try:
+ with engine.connect() as conn:
+ with conn.connection.cursor() as cursor:
+ cursor.execute("SET TIME ZONE 'UTC'")
+ cursor.execute("""SELECT CAST('2026-01-02 03:04:05.123456'
AS TIMESTAMP) AS ts,
+ CAST('2026-01-02 03:04:05.123456' AS TIMESTAMP_NTZ) AS
ntz,
+ CAST(123.456 AS DECIMAL(38,18)) AS d,
+ array(1, CAST(NULL AS INT)) AS a, named_struct('x', 7)
AS s""")
+ arrow = cursor.fetch_arrow_table()
+ self.assertEqual(str(arrow.schema.field('ts').type),
'timestamp[us, tz=UTC]')
+ self.assertEqual(str(arrow.schema.field('ntz').type),
'timestamp[us]')
+ self.assertEqual(str(arrow.schema.field('d').type),
'decimal128(38, 18)')
+ self.assertEqual(arrow.to_pylist(), [{
+ 'ts': datetime(2026, 1, 2, 3, 4, 5, 123456,
tzinfo=timezone.utc),
+ 'ntz': datetime(2026, 1, 2, 3, 4, 5, 123456),
+ 'd': Decimal('123.456000000000000000'),
+ 'a': [1, None], 's': {'x': 7}}])
+ finally:
+ engine.dispose()
diff --git a/hudi-lakehouse/superset-spark-connect/test_superset_live.py
b/hudi-lakehouse/superset-spark-connect/test_superset_live.py
new file mode 100644
index 000000000000..e7106ae1ec9c
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_superset_live.py
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in SQL Lab API test. The target connection must allow DML."""
+import os
+import unittest
+import uuid
+
+URL = os.environ.get("HUDI_SUPERSET_URL")
+USER = os.environ.get("HUDI_SUPERSET_USER")
+PASSWORD = os.environ.get("HUDI_SUPERSET_PASSWORD")
+DATABASE = os.environ.get("HUDI_SUPERSET_DATABASE_ID")
+SCHEMA = os.environ.get("HUDI_SUPERSET_SCHEMA", "default")
+
+
[email protected](all((URL, USER, PASSWORD, DATABASE)), "set Superset URL,
user, password and database ID")
+class SupersetLiveTest(unittest.TestCase):
+ def test_temporal_chart_filters_and_epoch_grouping(self):
+ import requests
+
+ session = requests.Session()
+ base = URL.rstrip("/")
+ response = session.post(base + "/api/v1/security/login", json={
+ "username": USER, "password": PASSWORD, "provider": "db"},
timeout=30)
+ response.raise_for_status()
+ session.headers["Authorization"] = "Bearer " +
response.json()["access_token"]
+ response = session.get(base + "/api/v1/security/csrf_token/",
timeout=30)
+ response.raise_for_status()
+ session.headers.update({"X-CSRFToken": response.json()["result"],
"Referer": base + "/"})
+ table = "temporal_adapter_" + uuid.uuid4().hex
+ dataset_id = None
+ created = False
+
+ def run(sql):
+ response = session.post(base + "/api/v1/sqllab/execute/", json={
+ "database_id": int(DATABASE), "sql": sql, "schema": SCHEMA,
+ "runAsync": False, "client_id": uuid.uuid4().hex[:11]},
timeout=180)
+ response.raise_for_status()
+ result = response.json()
+ self.assertEqual(result.get("status"), "success", result)
+ return result["data"]
+
+ def chart(query):
+ response = session.post(base + "/api/v1/chart/data", json={
+ "datasource": {"id": dataset_id, "type": "table"}, "queries":
[query],
+ "result_format": "json", "result_type": "full", "force":
True}, timeout=120)
+ response.raise_for_status()
+ result = response.json()["result"][0]
+ self.assertEqual(result["status"], "success", result)
+ return result
+
+ try:
+ run(f"CREATE TABLE {table} (id INT, day DATE, event_ts TIMESTAMP, "
+ "epoch_s BIGINT, epoch_ms BIGINT) USING parquet")
+ created = True
+ run(f"INSERT INTO {table} VALUES "
+ "(1, DATE '2023-11-14', TIMESTAMP '2023-11-14 22:13:20',
1700000000, 1700000000000), "
+ "(2, DATE '2023-11-15', TIMESTAMP '2023-11-15 22:13:20',
1700086400, 1700086400000)")
+ response = session.post(base + "/api/v1/dataset/", json={
+ "database": int(DATABASE), "schema": SCHEMA, "table_name":
table}, timeout=120)
+ response.raise_for_status()
+ dataset_id = response.json()["id"]
+ for name in ("day", "event_ts"):
+ result = chart({"columns": ["id"], "filters": [
+ {"col": name, "op": "==", "val": 1700000000000}],
"row_limit": 100})
+ self.assertEqual(result["data"], [{"id": 1}])
+
+ response = session.get(base + f"/api/v1/dataset/{dataset_id}",
timeout=30)
+ response.raise_for_status()
+ columns = response.json()["result"]["columns"]
+ for column in columns:
+ if column["column_name"] in ("epoch_s", "epoch_ms"):
+ column["is_dttm"] = True
+ column["python_date_format"] = column["column_name"]
+ keys = ("id", "column_name", "is_dttm", "python_date_format",
"type", "groupby", "filterable")
+ response = session.put(base + f"/api/v1/dataset/{dataset_id}",
json={
+ "columns": [{key: column[key] for key in keys} for column in
columns]}, timeout=30)
+ response.raise_for_status()
+ for name in ("epoch_s", "epoch_ms"):
+ result = chart({"columns": [{"columnType": "BASE_AXIS",
"sqlExpression": name,
+ "label": name, "timeGrain":
"P1D"}],
+ "metrics": [{"expressionType": "SQL",
"sqlExpression": "COUNT(*)",
+ "label": "count"}], "row_limit":
100})
+ self.assertEqual(sorted((row[name], row["count"]) for row in
result["data"]),
+ [(1699920000000, 1), (1700006400000, 1)])
+ result = chart({"columns": ["id"], "orderby": [["id", True]],
+ "row_offset": 1, "row_limit": None})
+ self.assertEqual(result["data"], [{"id": 2}])
+ finally:
+ try:
+ if dataset_id is not None:
+ session.delete(base + f"/api/v1/dataset/{dataset_id}",
timeout=30).raise_for_status()
+ finally:
+ if created:
+ run(f"DROP TABLE IF EXISTS {table} PURGE")
+ session.close()
+
+ def test_sql_lab_queries_and_writes(self):
+ import requests
+
+ session = requests.Session()
+ base = URL.rstrip("/")
+ response = session.post(base + "/api/v1/security/login", json={
+ "username": USER, "password": PASSWORD, "provider": "db"},
timeout=30)
+ response.raise_for_status()
+ session.headers["Authorization"] = "Bearer " +
response.json()["access_token"]
+ response = session.get(base + "/api/v1/security/csrf_token/",
timeout=30)
+ response.raise_for_status()
+ session.headers.update({"X-CSRFToken": response.json()["result"],
"Referer": base + "/"})
+ table = "connect_adapter_" + uuid.uuid4().hex
+
+ def run(sql):
+ response = session.post(base + "/api/v1/sqllab/execute/", json={
+ "database_id": int(DATABASE), "sql": sql, "schema": SCHEMA,
+ "runAsync": False, "client_id": uuid.uuid4().hex[:11]},
timeout=180)
+ response.raise_for_status()
+ result = response.json()
+ self.assertEqual(result.get("status"), "success", result)
+ return result["data"]
+
+ created = False
+ dataset_id = None
+ attempted = False
+ try:
+ attempted = True
+ run(f"CREATE TABLE {table} (id INT, region STRING, amount INT, ts
BIGINT) USING hudi "
+ "TBLPROPERTIES(type='cow', primaryKey='id',
preCombineField='ts')")
+ created = True
+ run(f"INSERT INTO {table} VALUES (1,'East',100,1),
(2,'West',200,1)")
+ self.assertEqual(run(f"SELECT SUM(amount) AS total FROM {table}"),
[{"total": 300}])
+ self.assertEqual(run(f"SELECT id FROM {table} WHERE
region='East'"), [{"id": 1}])
+ response = session.post(base + "/api/v1/dataset/", json={
+ "database": int(DATABASE), "schema": SCHEMA, "table_name":
table}, timeout=120)
+ response.raise_for_status()
+ dataset_id = response.json()["id"]
+ payload = {"datasource": {"id": dataset_id, "type": "table"},
+ "queries": [{"metrics": [{"expressionType": "SQL",
+ "sqlExpression": "SUM(amount)", "label": "total"}],
+ "columns": ["region"], "row_limit": 100}],
+ "result_format": "json", "result_type": "full",
"force": True}
+
+ def chart_data():
+ response = session.post(base + "/api/v1/chart/data",
json=payload, timeout=120)
+ response.raise_for_status()
+ result = response.json()["result"][0]
+ self.assertEqual(result["status"], "success", result)
+ return {row["region"]: row["total"] for row in result["data"]}
+
+ self.assertEqual(chart_data(), {"East": 100, "West": 200})
+ run(f"UPDATE {table} SET amount=150, ts=2 WHERE id=1")
+ self.assertEqual(run(f"SELECT id, amount FROM {table} WHERE
id=1"), [{"id": 1, "amount": 150}])
+ self.assertEqual(chart_data(), {"East": 150, "West": 200})
+ run(f"MERGE INTO {table} t USING ("
+ "SELECT 1 AS id, 'East' AS region, 175 AS amount, CAST(3 AS
BIGINT) AS ts "
+ "UNION ALL SELECT 3, 'North', 300, CAST(3 AS BIGINT)) s "
+ "ON t.id=s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED
THEN INSERT *")
+ self.assertEqual(run(f"SELECT id, region, amount, ts FROM {table}
ORDER BY id"), [
+ {"id": 1, "region": "East", "amount": 175, "ts": 3},
+ {"id": 2, "region": "West", "amount": 200, "ts": 1},
+ {"id": 3, "region": "North", "amount": 300, "ts": 3},
+ ])
+ run(f"DELETE FROM {table} WHERE id=2")
+ self.assertEqual(run(f"SELECT count(*) AS n FROM {table}"), [{"n":
2}])
+ finally:
+ try:
+ if dataset_id is not None:
+ try:
+ response = session.delete(base +
f"/api/v1/dataset/{dataset_id}", timeout=30)
+ response.raise_for_status()
+ except Exception:
+ print(f"Cleanup failed; remove test dataset
{dataset_id} manually.")
+ raise
+ finally:
+ try:
+ if created:
+ try:
+ run(f"DROP TABLE IF EXISTS {table} PURGE")
+ except Exception:
+ print(f"Cleanup failed; remove test table
{SCHEMA}.{table} manually.")
+ raise
+ elif attempted:
+ print(f"CREATE did not report success; check for test
table {SCHEMA}.{table} before cleanup.")
+ finally:
+ session.close()
diff --git a/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
b/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
new file mode 100644
index 000000000000..57e3b12a936b
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in Zeppelin notebook test against an isolated Connect endpoint."""
+import os
+import unittest
+import uuid
+from urllib.parse import parse_qsl, urlsplit
+
+URL = os.environ.get('HUDI_ZEPPELIN_URL')
+CONNECT_URI = os.environ.get('HUDI_ZEPPELIN_CONNECT_URI')
+
+
[email protected](URL and CONNECT_URI, 'set HUDI_ZEPPELIN_URL and
HUDI_ZEPPELIN_CONNECT_URI')
+class ZeppelinLiveTest(unittest.TestCase):
+ def test_notebook_crud_and_table_output(self):
+ import requests
+
+ parsed = urlsplit(CONNECT_URI)
+ options = dict(parse_qsl(parsed.query, keep_blank_values=True))
+ allowed = {'api', 'auth_type', 'tls', 'validateservercertificate'}
+ if (parsed.scheme != 'spark' or parsed.username or parsed.password
+ or options.get('auth_type') != 'none' or options.get('api') !=
'connect'
+ or set(options) - allowed):
+ self.fail('Use an unauthenticated test URI without credentials: it
is stored in the notebook')
+ base = URL.rstrip('/')
+ table = 'zeppelin_test_' + uuid.uuid4().hex
+ name = 'Hudi Connect test ' + uuid.uuid4().hex
+ # The URI is rendered as a Python string, not interpolated into SQL.
+ code = f'''%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+with dbapi.connect(driver="spark", db_kwargs={{"uri": {CONNECT_URI!r}}},
autocommit=True) as conn:
+ with conn.cursor() as cur:
+ cur.execute("CREATE TABLE default.{table} (id INT, region STRING,
amount INT, ts BIGINT) USING hudi TBLPROPERTIES(type='cow', primaryKey='id',
preCombineField='ts')")
+ try:
+ cur.execute("INSERT INTO default.{table} VALUES (1,'East',100,1),
(2,'West',200,1)")
+ cur.execute("SELECT region, SUM(amount) FROM default.{table} GROUP
BY region ORDER BY region")
+ assert cur.fetchall() == [('East',100),('West',200)]
+ cur.execute("UPDATE default.{table} SET amount=150, ts=2 WHERE
id=1")
+ cur.execute("SELECT id, amount FROM default.{table} WHERE id=1")
+ assert cur.fetchall() == [(1,150)]
+ cur.execute("MERGE INTO default.{table} t USING (SELECT 1 AS id,
'East' AS region, 175 AS amount, CAST(3 AS BIGINT) AS ts UNION ALL SELECT 3,
'North', 300, CAST(3 AS BIGINT)) s ON t.id=s.id WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *")
+ cur.execute("SELECT id, region, amount, ts FROM default.{table}
ORDER BY id")
+ assert cur.fetchall() ==
[(1,'East',175,3),(2,'West',200,1),(3,'North',300,3)]
+ cur.execute("DELETE FROM default.{table} WHERE id=2")
+ cur.execute("SELECT region, SUM(amount) FROM default.{table} GROUP
BY region ORDER BY region")
+ rows = cur.fetchall()
+ assert rows == [('East',175),('North',300)]
+ z.show(pd.DataFrame(rows, columns=['region','total']))
+ finally:
+ cur.execute("DROP TABLE IF EXISTS default.{table} PURGE")
+'''
+ response = requests.post(base + '/api/notebook', json={'name': name,
'paragraphs': [{
+ 'title': 'ADBC Hudi CRUD', 'text': code,
+ 'config': {'colWidth': 12, 'results': [{'graph': {'mode':
'table'}}]}}]}, timeout=30)
+ response.raise_for_status()
+ note = response.json()['body']
+ succeeded = False
+ try:
+ response = requests.get(base + '/api/notebook/' + note, timeout=30)
+ response.raise_for_status()
+ paragraph = response.json()['body']['paragraphs'][0]['id']
+ response = requests.post(base +
f'/api/notebook/run/{note}/{paragraph}', timeout=240)
+ response.raise_for_status()
+ result = response.json()['body']
+ self.assertEqual(result['code'], 'SUCCESS', result)
+ self.assertTrue(any(item['type'] == 'TABLE' and item['data'] ==
'region\ttotal\nEast\t175\nNorth\t300\n'
+ for item in result['msg']), result)
+ succeeded = True
+ finally:
+ try:
+ response = requests.delete(base + '/api/notebook/' + note,
timeout=30)
+ response.raise_for_status()
+ finally:
+ if not succeeded:
+ print(f'Check test table default.{table} after the
notebook stops before cleaning it up.')
+
+
+if __name__ == '__main__':
+ unittest.main()