This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino-playground.git
The following commit(s) were added to refs/heads/main by this push:
new d1e85b3 Fix Iceberg JDBC catalog init failure in Trino; seed Iceberg
demo data; persist catalog data; playground.sh and README fixes (#157)
d1e85b3 is described below
commit d1e85b3efdc4f6f2380e130c710603d000e0c5af
Author: Mark Hoerth <[email protected]>
AuthorDate: Tue Jul 14 19:08:16 2026 -0700
Fix Iceberg JDBC catalog init failure in Trino; seed Iceberg demo data;
persist catalog data; playground.sh and README fixes (#157)
Trino's Iceberg connector initializes JdbcCatalog with
init-catalog-tables disabled, so querying catalog_iceberg from Trino
before any Spark activity fails with `Cannot check and eventually update
SQL schema`, caused by `Table 'db.iceberg_tables' doesn't exist`. Spark
auto-creates the tables, which masks the issue when Spark runs first.
### What changes were proposed in this pull request?
**Bug fix.** Add `CREATE TABLE IF NOT EXISTS` statements for
`iceberg_tables` and `iceberg_namespace_properties` in the `db` database
to `init/mysql/init.sql`, so the Iceberg JDBC catalog backing tables
exist as soon as MySQL initializes. Also adds `CREATE DATABASE IF NOT
EXISTS db` for robustness, though the compose file already creates it
via `MYSQL_DATABASE`.
**Iceberg seed data.** The playground seeds Hive and PostgreSQL with
demo data but left the Iceberg catalog empty. A new startup step
(`init/spark/iceberg-seed.sh`, hooked into the Spark entrypoint) seeds
`analytics.orders`, a region-partitioned Iceberg table written in two
commits so snapshot history and time travel are demonstrable
immediately. The seed waits for the Iceberg REST service, is idempotent
(skips when the namespace exists), runs with the same identity that
initializes the metalake so it works with and without `--enable-auth`,
and logs to `/tmp/iceberg-seed.log` in the Spark container.
**Persist catalog data across stops.** MySQL, PostgreSQL, and the hive
container's HDFS tree and internal metastore database now use named
Docker volumes, so `./playground.sh stop` followed by `start` preserves
catalog contents. Previously these lived in anonymous volumes and were
silently lost on every stop, while Gravitino's own metadata
(bind-mounted) survived, leaving Gravitino aware of catalogs whose
contents had vanished. The full reset (`docker compose -p
gravitino-playground down -v` plus removing the `data` directory)
remains the way to start fresh, and the README's Stop section now states
the behavior accurately.
**playground.sh fixes.** Robust hello-world cleanup after the Docker
preflight test (the previous image-listing parse produced `No such
image` errors), corrected the Postgres port in the preflight check
(`15342` -> `15432`), added Spark's `14040`, and aligned the RAM
requirement with the README (8 GB).
**README overhaul.** While fixing the bug required reverse-engineering
the environment, the README got a full pass:
- New Environment configuration section stating what the playground
actually runs: authentication and authorization posture, plain HTTP
transport, the H2 entity store, the MySQL-backed Iceberg JDBC catalog,
HDFS table storage, and demo credentials
- Usage section rework: HTTPS clone URL instead of SSH, the two install
paths presented as explicit alternatives, a shared Playground management
subsection, a pointer to the `--enable-auth` and `--enable-ranger`
flags, and documented full-reset instructions (`docker compose down -v`
plus removing the `data` directory, neither of which was documented)
- Documentation of the seeded `analytics.orders` table with showcase
queries including `$snapshots` and `FOR VERSION AS OF`
- Structure: code blocks indented under their numbered list items, the
two step-labeled demos converted to the same numbered convention as the
rest of the document, single-child headings collapsed, gerund headings
converted to noun phrases, the Examples subsections reordered so the two
Iceberg REST sections are adjacent and the governance demos form a
contiguous block
- Accuracy: removed the stale ASF Incubator disclaimer (Gravitino is a
TLP), removed version-gated claims (1.0+, 1.1), corrected the access
control intro to claim catalog/schema/table level privileges rather than
fine-grained authorization, and replaced the vague security remediation
with concrete guidance (OAuth2 token validation per the Gravitino
security docs)
- Correctness fixes: a broken four-backtick code fence, `//` comments in
a Spark SQL block (invalid syntax), the intro service list completed to
match the compose stack, and a note that `catalog_iceberg` in the role
definitions is the same catalog as `catalog_rest` in Spark
- Style: consistent imperative second person, "Log in to" throughout,
filler and marketing phrasing removed, normalized notebook pointers
Reviewer note: the README diff is large but much of the line count is
list re-indentation; the hide-whitespace toggle on the Files changed
view shows the substance.
### Why are the changes needed?
Fixes a bug. The playground currently works only if Spark touches
`catalog_iceberg` before Trino does, because Spark's JdbcCatalog
auto-creates its backing tables and Trino's does not. A user who starts
the playground and queries the Iceberg catalog from Trino first gets a
`GENERIC_INTERNAL_ERROR` with no indication of the cause. Creating the
tables at MySQL init time removes the engine-ordering dependency.
Repro: start the playground per the README, then run `SHOW SCHEMAS FROM
catalog_iceberg` in Trino before running anything in Spark.
The seed data brings the Iceberg catalog to parity with the Hive and
PostgreSQL catalogs, which ship populated, and gives the Iceberg REST
demo something to show before the user creates anything, including
Iceberg-specific capabilities (snapshots, time travel) that a
single-insert table cannot demonstrate.
The README changes make the environment describable and the document
consistent: before this change the README could not answer basic
questions about the environment it documents (is authorization on, where
is metadata stored, how do I reset it), carried an obsolete incubation
disclaimer, and overstated the access control capability.
### Does this PR introduce any user-facing change?
No API or property key changes. The Trino-first query path now works
instead of failing, the Iceberg catalog comes up seeded with
`analytics.orders`, catalog data now persists across stop/start, the
preflight check validates the correct ports and RAM figure, and the
README is restructured as described above.
### How was this patch tested?
Reproduced the failure on a fresh playground (Gravitino 1.3.0, Trino,
MySQL 8.0). All verification was performed on clean clones of this
branch with fresh volumes and a removed `data` directory:
- Default start (`./playground.sh start`): both catalog tables present
in MySQL at startup; `SHOW SCHEMAS FROM catalog_iceberg` succeeds in
Trino with no prior Spark activity; the seed completes in the background
within a few minutes of startup; `analytics.orders` returns 12 rows with
correct regional aggregates; `orders$snapshots` lists two snapshots;
`FOR VERSION AS OF` the first snapshot returns 10 rows.
- Auth-enabled start (`./playground.sh start --enable-auth`): the seed
succeeds with the init identity and the same queries return the same
results. (An earlier revision seeded as a named user, which the Iceberg
REST server rejects under auth because the user does not exist in the
metalake; the seed now uses the same identity that initializes the
metalake.)
- Restart without reset (`./playground.sh stop` then `start`): with the
named volumes, a marker table created in MySQL before the stop is
present after it, the seed detects the existing `analytics` namespace
and skips, and `orders$snapshots` returns the same snapshot ids as
before the stop, proving MySQL and HDFS survived coherently. The seed's
namespace check also fails safe: if the check itself errors, it skips
rather than risking duplicate inserts.
- The fixed preflight was observed during startup: corrected port list,
no image-removal error after the Docker test.
- All statements in `init.sql` and the seed are idempotent (`IF NOT
EXISTS` plus the namespace check), so they are harmless when Spark has
already created objects or on restarts against existing volumes.
- README verified in GitHub's rendered view on the branch.
---------
Co-authored-by: Mark Hoerth <[email protected]>
---
README.md | 907 ++++++++++++++++++++++----------------------
docker-compose.yaml | 10 +
init/mysql/init.sql | 20 +
init/spark/iceberg-seed.sh | 53 +++
init/spark/iceberg-seed.sql | 35 ++
init/spark/init.sh | 1 +
playground.sh | 12 +-
7 files changed, 589 insertions(+), 449 deletions(-)
diff --git a/README.md b/README.md
old mode 100644
new mode 100755
index 3ab072a..d4dfd7b
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
## Playground introduction
-The playground is a complete Apache Gravitino Docker runtime environment with
`Hive`, `HDFS`, `Trino`, `MySQL`, `PostgreSQL`, `Jupyter`, and a `Gravitino`
server.
+The playground is a complete Apache Gravitino Docker runtime environment with
`Hive`, `HDFS`, `Trino`, `Spark`, `MySQL`, `PostgreSQL`, `Ranger`, `Jupyter`,
`Prometheus`, `Grafana`, and a `Gravitino` server.
Depending on your network and computer, startup time may take 3-5 minutes.
Once the playground environment has started, you can open
[http://localhost:8090](http://localhost:8090) in a browser to access the
Gravitino Web UI.
@@ -29,7 +29,7 @@ Install Git (optional), Docker, Docker Compose.
## System Resource Requirements
-2 CPU cores, 8 GB RAM, 25 GB disk storage, MacOS or Linux OS (Verified
Ubuntu22.04 Ubuntu24.04 AmazonLinux).
+2 CPU cores, 8 GB RAM, 25 GB disk storage, macOS or Linux (verified on Ubuntu
22.04, Ubuntu 24.04, and Amazon Linux).
## TCP ports used
@@ -48,53 +48,84 @@ The playground runs several services. The TCP ports used
may clash with existing
| playground-prometheus | 19090 |
| playground-grafana | 13000 |
+## Environment configuration
+
+The playground is preconfigured for local evaluation and is not a production
reference architecture. The defaults reflect that:
+
+| Aspect | Configuration | Notes
|
+| ------------------------ | ----------------------------------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| Authentication | None | Gravitino
trusts the username presented by clients. See the security note in the access
control demo below. |
+| Authorization | Disabled | Enable
Gravitino native access control with `--enable-auth`, or Ranger enforcement for
Hive with `--enable-ranger`. |
+| Transport | Plain HTTP | No TLS on
any service, including Trino and the Gravitino API.
|
+| Gravitino metadata store | Embedded H2 in the `data` directory | Only the
Gravitino server accesses this store, so an embedded database suffices. Wiped
by the full reset. |
+| Iceberg catalog backend | JDBC, MySQL `db` database | Shared by
`catalog_iceberg` (Gravitino, Trino) and `catalog_rest` (Spark, through the
Iceberg REST service), so it needs a database all three can reach. |
+| Table storage | HDFS in the `hive` container |
`hdfs://hive:9000` for both Hive and Iceberg warehouses. No object storage is
involved. |
+| Credentials | Hardcoded demo values | For
example, MySQL uses `mysql`/`mysql`.
|
+
+The playground has no authentication or TLS, so any reachable port grants full
access to that service. If you run the playground on a remote host, control who
can reach the ports, for example with firewall rules scoped to your address or
an SSH tunnel.
+
## Playground usage
-### One curl command launch playground
-```shell
+There are two ways to get the playground. Use one or the other, not both.
+
+### Option 1: One-command install and launch
+
+Downloads the playground and starts it in a single step:
+
+```bash
/bin/bash -c "$(curl -fsSL
https://raw.githubusercontent.com/apache/gravitino-playground/HEAD/install.sh)"
```
-### Use git to download and launch playground
+### Option 2: Clone and launch with git
-```shell
-git clone [email protected]:apache/gravitino-playground.git
+```bash
+git clone https://github.com/apache/gravitino-playground.git
cd gravitino-playground
+./playground.sh start
```
-### Start
+The start command accepts optional flags for the access control demos
described below: `--enable-auth` or `--enable-ranger` (the two cannot be
combined).
-```shell
-./playground.sh start
-```
+### Playground management
-### Check status
-```shell
+Run these from the playground directory, whichever option you used to install
it (`gravitino-playground` for git, `gravitino-playground-main` for the
installer).
+
+#### Check status
+
+```bash
./playground.sh status
```
-### Stop
-```shell
+When all containers are healthy, open the Gravitino Web UI at
<http://localhost:8090>.
+
+#### Stop
+
+```bash
./playground.sh stop
```
-## Experiencing Apache Gravitino with Trino SQL
+Stopping keeps all data: Gravitino metadata in the `data` directory of this
repo, and MySQL, PostgreSQL, and HDFS contents in named Docker volumes. To
remove everything and start completely fresh:
-### Using Trino CLI in Docker Container
+```bash
+docker compose -p gravitino-playground down -v
+rm -rf data
+```
-1. Login to the Gravitino playground Trino Docker container using the
following command:
+## Trino CLI
-```shell
-docker exec -it playground-trino bash
-```
+1. Log in to the Trino container:
-2. Open the Trino CLI in the container.
+ ```shell
+ docker exec -it playground-trino bash
+ ```
-```shell
-trino
-```
+2. Open the Trino CLI:
-## Using Jupyter Notebook
+ ```shell
+ trino
+ ```
+
+## Jupyter Notebook
1. Open the Jupyter Notebook in the browser at
[http://localhost:18888](http://localhost:18888).
@@ -102,33 +133,33 @@ trino
3. Start the notebook and run the cells.
-## Using Spark client
+## Spark SQL client
-1. Login to the Gravitino playground Spark Docker container using the
following command:
+1. Log in to the Spark container:
-```shell
-docker exec -it playground-spark bash
-````
+ ```shell
+ docker exec -it playground-spark bash
+ ```
-2. Open the Spark SQL client in the container.
+2. Open the Spark SQL client:
-```shell
-cd /opt/spark && /bin/bash bin/spark-sql
-```
+ ```shell
+ cd /opt/spark && /bin/bash bin/spark-sql
+ ```
-## Monitoring Gravitino
+## Grafana dashboards
-1. Open the Grafana in the browser at
[http://localhost:13000](http://localhost:13000).
+1. Open Grafana in the browser at
[http://localhost:13000](http://localhost:13000).
2. In the navigation menu, click **Dashboards** -> **Gravitino Playground**.
3. Experiment with the default template.
-## Example
+## Examples
### Simple Trino queries
-You can use simple queries to test in the Trino CLI.
+Test the setup with simple queries in the Trino CLI.
```SQL
SHOW CATALOGS;
@@ -160,9 +191,9 @@ SHOW TABLES from catalog_hive.company;
### Cross-catalog queries
-In a company, there may be different departments using different data stacks.
In this example, the HR department uses Apache Hive to store its data, and the
sales department uses PostgreSQL. You can run some interesting queries by
joining the two departments' data together with Gravitino.
+Different departments often run different data stacks. In this example, HR
stores its data in Hive and sales uses PostgreSQL. Gravitino lets you join data
across both.
-To know which employee has the largest sales amount, run this SQL:
+To find the employee with the largest sales amount:
```SQL
SELECT given_name, family_name, job_title, sum(total_amount) AS total_sales
@@ -174,7 +205,7 @@ ORDER BY total_sales DESC
LIMIT 1;
```
-To know the top customers who bought the most by state, run this SQL:
+To find the top customers by state:
```SQL
SELECT customer_name, location, SUM(total_amount) AS total_spent
@@ -186,7 +217,7 @@ GROUP BY location, customer_name
ORDER BY location, SUM(total_amount) DESC;
```
-To know the employee's average performance rating and total sales, run this
SQL:
+To get each employee's average performance rating and total sales:
```SQL
SELECT e.employee_id, given_name, family_name, AVG(rating) AS average_rating,
SUM(total_amount) AS total_sales
@@ -197,47 +228,46 @@ WHERE e.employee_id = p.employee_id AND p.employee_id =
s.employee_id
GROUP BY e.employee_id, given_name, family_name;
```
-### Using Spark and Trino
+### Spark and Trino together
-You might also consider generating data with SparkSQL and then querying this
data using Trino. Give it a try with Gravitino:
+You can also generate data with Spark SQL and query it with Trino:
-1. Login Spark container and execute the SQLs:
+1. Log in to the Spark container and run the SQL:
-```sql
-// using Hive catalog to create Hive table
-USE catalog_hive;
-CREATE DATABASE product;
-USE product;
-
-CREATE TABLE IF NOT EXISTS employees (
- id INT,
- name STRING,
- age INT
-)
-PARTITIONED BY (department STRING)
-STORED AS PARQUET;
-DESC TABLE EXTENDED employees;
+ ```sql
+ -- using Hive catalog to create Hive table
+ USE catalog_hive;
+ CREATE DATABASE product;
+ USE product;
-INSERT OVERWRITE TABLE employees PARTITION(department='Engineering') VALUES
(1, 'John Doe', 30), (2, 'Jane Smith', 28);
-INSERT OVERWRITE TABLE employees PARTITION(department='Marketing') VALUES (3,
'Mike Brown', 32);
-```
+ CREATE TABLE IF NOT EXISTS employees (
+ id INT,
+ name STRING,
+ age INT
+ )
+ PARTITIONED BY (department STRING)
+ STORED AS PARQUET;
+ DESC TABLE EXTENDED employees;
-2. Login Trino container and execute SQLs:
+ INSERT OVERWRITE TABLE employees PARTITION(department='Engineering') VALUES
(1, 'John Doe', 30), (2, 'Jane Smith', 28);
+ INSERT OVERWRITE TABLE employees PARTITION(department='Marketing') VALUES
(3, 'Mike Brown', 32);
+ ```
-```sql
-SELECT * FROM catalog_hive.product.employees WHERE department = 'Engineering';
-```
+2. Log in to the Trino container and run the query:
-The demo is located in the `jupyter` folder, and you can open the
`gravitino-spark-trino-example.ipynb`
-demo via Jupyter Notebook by [http://localhost:18888](http://localhost:18888).
+ ```sql
+ SELECT * FROM catalog_hive.product.employees WHERE department =
'Engineering';
+ ```
-### Using Apache Iceberg REST service
+The demo is also available as `gravitino-spark-trino-example.ipynb` in Jupyter
at [http://localhost:18888](http://localhost:18888).
-Suppose you want to migrate your business from Hive to Iceberg. Some tables
will use Hive, and the other tables will use Iceberg.
-Gravitino provides an Iceberg REST catalog service, too. You can use Spark to
access the REST catalog to write the table data.
-Then, you can use Trino to read the data from the Hive table joining the
Iceberg table.
+### Iceberg REST service
-`spark-defaults.conf` is as follows (It's already configured in the
playground):
+A common migration scenario: some tables remain in Hive while others move to
Iceberg.
+Gravitino provides an Iceberg REST catalog service for exactly this. In the
example below, Spark writes
+table data through the REST catalog, and Trino joins the new Iceberg table
with an existing Hive table.
+
+The playground ships with the following `spark-defaults.conf`:
```text
spark.sql.extensions
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
@@ -247,201 +277,171 @@ spark.sql.catalog.catalog_rest.uri
http://gravitino:9001/iceberg/
spark.locality.wait.node 0
```
-Please note that `catalog_rest` in SparkSQL and `catalog_iceberg` in Gravitino
and Trino share the same Iceberg JDBC backend, implying they can access the
same dataset.
-
-1. Login Spark container and execute the steps.
-
-```shell
-docker exec -it playground-spark bash
-```
-
-```shell
-spark@container_id:/$ cd /opt/spark && /bin/bash bin/spark-sql
-```
+Note that `catalog_rest` in Spark and `catalog_iceberg` in Gravitino and Trino
share the same Iceberg JDBC backend, so they access the same dataset.
-```SQL
-use catalog_rest;
-create database sales;
-use sales;
-create table customers (customer_id int, customer_name varchar(100),
customer_email varchar(100));
-describe extended customers;
-insert into customers (customer_id, customer_name, customer_email) values
(11,'Rory Brown','[email protected]');
-insert into customers (customer_id, customer_name, customer_email) values
(12,'Jerry Washington','[email protected]');
-```
+1. Log in to the Spark container and run the steps:
-2. Login Trino container and execute the steps.
- You can get all the customers from both the Hive and Iceberg table.
+ ```shell
+ docker exec -it playground-spark bash
+ ```
-```shell
-docker exec -it playground-trino bash
-```
+ ```shell
+ spark@container_id:/$ cd /opt/spark && /bin/bash bin/spark-sql
+ ```
-```shell
-trino@container_id:/$ trino
-```
+ ```SQL
+ use catalog_rest;
+ create database sales;
+ use sales;
+ create table customers (customer_id int, customer_name varchar(100),
customer_email varchar(100));
+ describe extended customers;
+ insert into customers (customer_id, customer_name, customer_email) values
(11,'Rory Brown','[email protected]');
+ insert into customers (customer_id, customer_name, customer_email) values
(12,'Jerry Washington','[email protected]');
+ ```
-```SQL
-select * from catalog_hive.sales.customers
-union
-select * from catalog_iceberg.sales.customers;
-```
-
-The demo is located in the `jupyter` folder, you can open the
`gravitino-spark-trino-example.ipynb`
-demo via Jupyter Notebook by [http://localhost:18888](http://localhost:18888).
+2. Log in to the Trino container and query all customers across the Hive and
Iceberg tables:
-### Using Gravitino with LlamaIndex
+ ```shell
+ docker exec -it playground-trino bash
+ ```
-The Gravitino Playground also provides a simple RAG demo with LlamaIndex. This
demo will show you the
-the ability to use Gravitino to manage both tabular and non-tabular datasets,
connecting to
-LlamaIndex as a unified data source, then use LlamaIndex and LLM to query both
tabular and
-non-tabular data with one natural language query.
+ ```shell
+ trino@container_id:/$ trino
+ ```
-The demo is located in the `jupyter` folder, and you can open the
`gravitino_llama_index_demo.ipynb`
-demo via Jupyter Notebook by [http://localhost:18888](http://localhost:18888).
+ ```SQL
+ select * from catalog_hive.sales.customers
+ union
+ select * from catalog_iceberg.sales.customers;
+ ```
-The scenario of this demo is that basic structured city statistics data is
stored in MySQL, and
-detailed city introductions are stored in PDF files. The user wants to know
the answers to the
-cities both in the structured data and the PDF files.
+The demo is also available as `gravitino-spark-trino-example.ipynb` in Jupyter
at [http://localhost:18888](http://localhost:18888).
-In this demo, you will use Gravitino to manage the MySQL table using a
relational catalog, pdf
-files using a fileset catalog, treating Gravitino as a unified data source for
LlamaIndex to build
-indexes on both tabular and non-tabular data. Then you will use LLM to query
the data with natural
-language queries.
+The playground also seeds the Iceberg catalog with a demo table at startup:
`analytics.orders`,
+partitioned by region and written in two commits, so the table has snapshot
history from the
+first query you run:
-Note: to run this demo, you need to set `OPENAI_API_KEY` in the
`gravitino_llama_index_demo.ipynb`,
-like below, `OPENAI_API_BASE` is optional.
-
-```python
-import os
+```sql
+SELECT region, SUM(amount) FROM catalog_iceberg.analytics.orders GROUP BY
region;
-os.environ["OPENAI_API_KEY"] = ""
-os.environ["OPENAI_API_BASE"] = ""
+SELECT * FROM catalog_iceberg."analytics"."orders$snapshots";
```
-### Using Gravitino with Ranger authorization
+The second query lists the table's snapshots; pick a `snapshot_id` from it to
read the table
+as of an earlier commit:
-Gravitino supports to provide the ability of access control for Hive tables
using Ranger plugin.
+```sql
+SELECT COUNT(*) FROM catalog_iceberg.analytics.orders FOR VERSION AS OF
<snapshot_id>;
+```
-For example, there are a manager and staffs in your company. Manager creates a
Hive catalog and create different roles.
-The manager can give different roles to different staffs.
+### Iceberg REST server access control
-You can run the command
+Gravitino provides built-in access control for the Iceberg REST server,
enforcing catalog,
+schema, and table level privileges without requiring an external authorization
service like
+Ranger. You manage users, roles, and privileges through the Gravitino API, and
the Iceberg
+REST server enforces them.
-```shell
-./playground.sh start --enable-ranger
-```
+**Security note**: the examples below use HTTP Basic Authentication only to
pass a username.
+Gravitino does not verify the password and trusts the supplied username for
access control
+decisions, so any client that can reach the REST endpoint can act as any user.
That is
+acceptable for the playground and nothing else. Production deployments
configure real
+authentication, such as OAuth2 token validation, as described in the
+[Gravitino security
documentation](https://gravitino.apache.org/docs/latest/security/access-control).
-The demo is located in the `jupyter` folder, you can open the
`gravitino-access-control-example.ipynb`
-demo via Jupyter Notebook by [http://localhost:18888](http://localhost:18888).
+1. Start the playground with auth enabled:
-### Using Gravitino Iceberg REST Server with Access Control
+ ```shell
+ ./playground.sh start --enable-auth
+ ```
-Gravitino 1.1 introduced built-in access control for the Iceberg REST server,
enabling fine-grained
-authorization for Iceberg tables without requiring external authorization
services like Ranger.
-This feature allows you to manage user permissions through Gravitino's unified
API with native
-access control enforcement at the REST API level.
+ **Note**: The `--enable-auth` flag enables Gravitino's access control by
removing the PassThroughAuthorizer, which allows proper privilege enforcement
for the Iceberg REST catalog.
-**Security note (authentication)**: The Iceberg REST catalog examples shown
here use HTTP Basic Authentication only as a transport to pass the username
through the `Authorization` header. Gravitino currently **does not verify the
Basic Auth password** and instead fully trusts the username provided in the
header for access control decisions. As a result, this mechanism **does not
provide real authentication**: any client that can reach the REST endpoint
could impersonate any user by choos [...]
+2. Create users through Gravitino's REST API:
-This behavior is intended **for local/demo use only** (such as when running
the playground) and **must not be relied upon in production** or any
environment exposed to untrusted clients. For secure deployments, you must
front the Iceberg REST server with a real authentication mechanism (for
example, an authenticating reverse proxy, API gateway, or other identity
provider) and configure Gravitino to validate the authenticated identity,
rather than trusting arbitrary usernames from the `Au [...]
-#### Demo Steps
+ ```shell
+ # Add manager user
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"name":"manager"}' \
+ http://localhost:8090/api/metalakes/metalake_demo/users
-**Step 1: Start the Playground with Auth Enabled**
-
-```shell
-./playground.sh start --enable-auth
-```
+ # Add data_analyst user
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"name":"data_analyst"}' \
+ http://localhost:8090/api/metalakes/metalake_demo/users
-**Note**: The `--enable-auth` flag enables Gravitino's access control by
removing the PassThroughAuthorizer, which allows proper privilege enforcement
for the Iceberg REST catalog.
+ # Set manager as owner of the metalake
+ curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"name":"manager","type":"USER"}' \
+
http://localhost:8090/api/metalakes/metalake_demo/owners/metalake/metalake_demo
+ ```
-**Step 2: Create Users**
+3. Create a database and table as the manager:
-Create users through Gravitino's REST API:
+ Log in to the Spark container:
-```shell
-# Add manager user
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{"name":"manager"}' \
- http://localhost:8090/api/metalakes/metalake_demo/users
-
-# Add data_analyst user
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{"name":"data_analyst"}' \
- http://localhost:8090/api/metalakes/metalake_demo/users
-
-# Set manager as owner of the metalake
-curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{"name":"manager","type":"USER"}' \
-
http://localhost:8090/api/metalakes/metalake_demo/owners/metalake/metalake_demo
-```
+ ```shell
+ docker exec -it playground-spark bash
+ ```
-**Step 3: Create Database and Table with Manager**
+ Start spark-sql as manager:
-Login to Spark container:
+ ```shell
+ cd /opt/spark && /bin/bash bin/spark-sql --conf
spark.sql.catalog.catalog_rest.rest.auth.type=basic --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.username=manager --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
+ ```
-```shell
-docker exec -it playground-spark bash
-```
+ Create database and table:
-Start spark-sql as manager:
+ ```sql
+ USE catalog_rest;
+ CREATE DATABASE IF NOT EXISTS demo_db;
+ USE demo_db;
-```shell
-cd /opt/spark && /bin/bash bin/spark-sql --conf
spark.sql.catalog.catalog_rest.rest.auth.type=basic --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.username=manager --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
-```
+ CREATE TABLE IF NOT EXISTS employees (
+ employee_id INT,
+ name STRING,
+ department STRING,
+ salary DECIMAL(10,2)
+ ) USING iceberg;
-Create database and table:
+ INSERT INTO employees VALUES
+ (1, 'Alice Johnson', 'Engineering', 95000.00),
+ (2, 'Bob Smith', 'Sales', 75000.00);
-```sql
-USE catalog_rest;
-CREATE DATABASE IF NOT EXISTS demo_db;
-USE demo_db;
-
-CREATE TABLE IF NOT EXISTS employees (
- employee_id INT,
- name STRING,
- department STRING,
- salary DECIMAL(10,2)
-) USING iceberg;
-
-INSERT INTO employees VALUES
- (1, 'Alice Johnson', 'Engineering', 95000.00),
- (2, 'Bob Smith', 'Sales', 75000.00);
-
-SELECT * FROM employees;
-```
+ SELECT * FROM employees;
+ ```
-**Step 4: Test Access Control Before Granting Privileges**
+4. Test access control before granting privileges:
-Exit spark-sql and start a new session as data_analyst (without any privileges
yet):
+ Exit spark-sql and start a new session as data_analyst (without any
privileges yet):
-```shell
-export HADOOP_USER_NAME=data_analyst
-cd /opt/spark
-/bin/bash bin/spark-sql --conf
spark.sql.catalog.catalog_rest.rest.auth.type=basic --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.username=data_analyst --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
-```
+ ```shell
+ export HADOOP_USER_NAME=data_analyst
+ cd /opt/spark
+ /bin/bash bin/spark-sql --conf
spark.sql.catalog.catalog_rest.rest.auth.type=basic --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.username=data_analyst --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
+ ```
-Try to query the table (this should FAIL):
+ Try to query the table. The query should fail:
-```sql
-USE catalog_rest.demo_db;
+ ```sql
+ USE catalog_rest.demo_db;
--- This should FAIL - schema doesn't exist, because we don't have USE_SCHEMA
privilege
-```
+ -- Fails: the schema is not visible without the USE_SCHEMA privilege
+ ```
-**Step 5: Create Role with Privileges and Assign to User**
+5. Create a role with privileges and assign it to the user:
-Exit spark-sql and create a role with the necessary privileges:
+ Exit spark-sql and create a role with the necessary privileges. Note that
the role references `catalog_iceberg`, the catalog name in Gravitino;
`catalog_rest` in Spark is the same catalog exposed through the Iceberg REST
endpoint:
-```shell
-# Create role with all required privileges
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -u manager:123 \
- -d '{
+ ```shell
+ # Create role with all required privileges
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -u manager:123 \
+ -d '{
"name": "analyst_role",
"securableObjects": [
{
@@ -466,217 +466,229 @@ curl -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
]
}
]
- }' \
- http://localhost:8090/api/metalakes/metalake_demo/roles
-
-# Assign role to user
-curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -u manager:123 \
- -d '{
+ }' \
+ http://localhost:8090/api/metalakes/metalake_demo/roles
+
+ # Assign role to user
+ curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -u manager:123 \
+ -d '{
"roleNames": ["analyst_role"]
-}'
http://localhost:8090/api/metalakes/metalake_demo/permissions/users/data_analyst/grant
-```
+ }'
http://localhost:8090/api/metalakes/metalake_demo/permissions/users/data_analyst/grant
+ ```
-Start spark-sql as data_analyst again and test:
+ Start spark-sql as data_analyst again and test:
-```shell
-cd /opt/spark && /bin/bash bin/spark-sql \
- --conf spark.sql.catalog.catalog_rest.rest.auth.type=basic \
- --conf spark.sql.catalog.catalog_rest.rest.auth.basic.username=data_analyst \
- --conf spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
-```
+ ```shell
+ cd /opt/spark && /bin/bash bin/spark-sql \
+ --conf spark.sql.catalog.catalog_rest.rest.auth.type=basic \
+ --conf
spark.sql.catalog.catalog_rest.rest.auth.basic.username=data_analyst \
+ --conf spark.sql.catalog.catalog_rest.rest.auth.basic.password=123
+ ```
-Try to query the table again (this should SUCCEED now):
+ Try to query the table again. The query should succeed now:
-```sql
-USE catalog_rest.demo_db;
+ ```sql
+ USE catalog_rest.demo_db;
--- This should succeed - now has SELECT_TABLE privilege
-SELECT * FROM employees;
-```
+ -- Succeeds: the role now grants SELECT_TABLE
+ SELECT * FROM employees;
+ ```
-This demonstrates how Gravitino's access control works:
-- Before granting privileges: Access denied
-- After granting privileges: Access allowed
+The demo shows Gravitino's access control at work: access is denied before the
privileges are
+granted and allowed after.
For more details, refer to the [Gravitino
documentation](https://gravitino.apache.org/docs/latest/security/access-control).
-### Using Gravitino Policies, Statistics, and Jobs to Drop Unused Tables
+### Ranger authorization with Hive
+
+Gravitino provides access control for Hive tables using the Ranger plugin.
+
+For example, a company has a manager and several staff members. The manager
creates a Hive catalog and defines different roles,
+then assigns those roles to staff members.
+
+Start the playground with Ranger enabled:
+
+```shell
+./playground.sh start --enable-ranger
+```
+
+The demo notebook is `gravitino-access-control-example.ipynb` in Jupyter at
[http://localhost:18888](http://localhost:18888).
+
+### Unused table cleanup with policies, statistics, and jobs
-Gravitino 1.0+ provides a powerful combination of policies, statistics, and
jobs that enables automated data governance tasks. This demo shows how to
identify and drop tables that haven't been accessed for a long time, helping
you manage data lifecycle and reduce storage costs.
+Gravitino provides a powerful combination of policies, statistics, and jobs
that enables automated, metadata-driven data governance. The demo shows how to
identify and drop tables that haven't been accessed for a long time, reducing
storage costs.
**Workflow Overview:**
1. **Statistics** - Track table usage with custom statistics (e.g.,
`custom-lastAccessTime`)
2. **Policies** - Define rules for identifying unused tables (e.g., not
accessed for 90 days)
3. **Jobs** - Execute automated actions to drop unused tables
-#### Demo Steps
+1. Start the playground:
-**Step 1: Start the Playground**
+ ```shell
+ ./playground.sh start
+ ```
-```shell
-./playground.sh start
-```
+2. Update statistics for an existing table:
-**Step 2: Update Statistics for an Existing Table**
+ The playground already has tables in the Hive catalog. Update the
statistics of an existing table to simulate an old, unused table:
-The playground already has tables in the Hive catalog. We'll use one of the
existing tables and update its statistics to simulate an old, unused table:
-
-```shell
-# First, verify the existing table
-docker exec -it playground-trino trino --execute "SELECT * FROM
catalog_hive.sales.customers LIMIT 5"
+ ```shell
+ # First, verify the existing table
+ docker exec -it playground-trino trino --execute "SELECT * FROM
catalog_hive.sales.customers LIMIT 5"
-# Calculate a date 100 days ago (more than the 90-day threshold)
-OLD_DATE=$(date -u -d '100 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date
-u -v-100d +%Y-%m-%dT%H:%M:%SZ)
+ # Calculate a date 100 days ago (more than the 90-day threshold)
+ OLD_DATE=$(date -u -d '100 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null ||
date -u -v-100d +%Y-%m-%dT%H:%M:%SZ)
-# Update last access time for the table to make it appear unused
-curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d "{
+ # Update last access time for the table to make it appear unused
+ curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d "{
\"updates\": {
\"custom-lastAccessTime\": \"$OLD_DATE\",
\"custom-rowCount\": \"10\"
}
- }" \
-
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/statistics
-
-# Check statistics to verify they were set
-curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/statistics
-```
-
-You should see output like:
-```json
-{
- "statistics": {
+ }" \
+
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/statistics
+
+ # Check statistics to verify they were set
+ curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/statistics
+ ```
+
+ You should see output like:
+ ```json
+ {
+ "statistics": {
"custom-lastAccessTime": {
"value": "2024-09-08T10:30:00Z"
},
"custom-rowCount": {
"value": "10"
}
- }
-}
-```
-
-**Step 3: Create a Policy for Unused Tables**
-
-Create a custom policy to identify tables not accessed for more than 90 days:
-
-```shell
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "name": "unused_table_policy",
- "comment": "Policy to identify tables not accessed for 90+ days",
- "policyType": "custom",
- "enabled": true,
- "content": {
- "customRules": {
- "maxIdleDays": 90,
- "action": "drop"
- },
- "supportedObjectTypes": ["TABLE"],
- "properties": {
- "checkStatistic": "custom-lastAccessTime",
- "threshold": "90d"
- }
- }
- }' \
- http://localhost:8090/api/metalakes/metalake_demo/policies
-```
-
-**Step 4: Associate Policy with Tables**
-
-Associate the policy with the existing customers table:
-
-```shell
-# Associate policy with the customers table
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
+ }
+ }
+ ```
+
+3. Create a policy for unused tables:
+
+ Create a custom policy to identify tables not accessed for more than 90
days:
+
+ ```shell
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "name": "unused_table_policy",
+ "comment": "Policy to identify tables not accessed for 90+ days",
+ "policyType": "custom",
+ "enabled": true,
+ "content": {
+ "customRules": {
+ "maxIdleDays": 90,
+ "action": "drop"
+ },
+ "supportedObjectTypes": ["TABLE"],
+ "properties": {
+ "checkStatistic": "custom-lastAccessTime",
+ "threshold": "90d"
+ }
+ }
+ }' \
+ http://localhost:8090/api/metalakes/metalake_demo/policies
+ ```
+
+4. Associate the policy with tables:
+
+ Associate the policy with the existing customers table:
+
+ ```shell
+ # Associate policy with the customers table
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
"policiesToAdd": ["unused_table_policy"]
- }' \
-
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/policies
+ }' \
+
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/policies
-# Verify the policy was associated
-curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/policies
-```
+ # Verify the policy was associated
+ curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+
http://localhost:8090/api/metalakes/metalake_demo/objects/table/catalog_hive.sales.customers/policies
+ ```
-Alternatively, you can associate the policy with the entire schema to monitor
all tables:
+ Alternatively, you can associate the policy with the entire schema to
monitor all tables:
-```shell
-# Associate with the entire schema (will apply to all tables in sales)
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
+ ```shell
+ # Associate with the entire schema (will apply to all tables in sales)
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
"policiesToAdd": ["unused_table_policy"]
- }' \
-
http://localhost:8090/api/metalakes/metalake_demo/objects/schema/catalog_hive.sales/policies
-```
-
-**Step 5: Register a Job Template to Drop Unused Tables**
-
-Create a shell script job template to drop tables:
-
-```shell
-# First, create the drop script on the host
-cat > /tmp/drop_unused_tables.sh << 'EOF'
-#!/bin/bash
-# Script to drop unused tables based on policy evaluation
-CATALOG=$1
-SCHEMA=$2
-TABLE=$3
-
-echo "Checking if table ${CATALOG}.${SCHEMA}.${TABLE} should be dropped..."
-
-# Get table statistics (use localhost since script runs on host or in
container with port mapping)
-STATS=$(curl -s -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-
"http://localhost:8090/api/metalakes/metalake_demo/objects/table/${CATALOG}.${SCHEMA}.${TABLE}/statistics";)
-
-echo "Statistics response: $STATS"
-
-# Parse the statistics array to find custom-lastAccessTime
-LAST_ACCESS=$(echo $STATS | jq -r '.statistics[] |
select(.name=="custom-lastAccessTime") | .value')
-echo "Last access time: $LAST_ACCESS"
-
-# Calculate days since last access
-if [ -n "$LAST_ACCESS" ] && [ "$LAST_ACCESS" != "null" ]; then
- CURRENT_DATE=$(date +%s)
- LAST_ACCESS_DATE=$(date -d "$LAST_ACCESS" +%s 2>/dev/null || date -j -f
"%Y-%m-%dT%H:%M:%SZ" "$LAST_ACCESS" +%s)
- DAYS_IDLE=$(( ($CURRENT_DATE - $LAST_ACCESS_DATE) / 86400 ))
-
- echo "Days since last access: $DAYS_IDLE"
-
- if [ $DAYS_IDLE -gt 90 ]; then
+ }' \
+
http://localhost:8090/api/metalakes/metalake_demo/objects/schema/catalog_hive.sales/policies
+ ```
+
+5. Register a job template to drop unused tables:
+
+ Create a shell script job template to drop tables:
+
+ ```shell
+ # First, create the drop script on the host
+ cat > /tmp/drop_unused_tables.sh << 'EOF'
+ #!/bin/bash
+ # Script to drop unused tables based on policy evaluation
+ CATALOG=$1
+ SCHEMA=$2
+ TABLE=$3
+
+ echo "Checking if table ${CATALOG}.${SCHEMA}.${TABLE} should be dropped..."
+
+ # Get table statistics (use localhost since script runs on host or in
container with port mapping)
+ STATS=$(curl -s -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+
"http://localhost:8090/api/metalakes/metalake_demo/objects/table/${CATALOG}.${SCHEMA}.${TABLE}/statistics";)
+
+ echo "Statistics response: $STATS"
+
+ # Parse the statistics array to find custom-lastAccessTime
+ LAST_ACCESS=$(echo $STATS | jq -r '.statistics[] |
select(.name=="custom-lastAccessTime") | .value')
+ echo "Last access time: $LAST_ACCESS"
+
+ # Calculate days since last access
+ if [ -n "$LAST_ACCESS" ] && [ "$LAST_ACCESS" != "null" ]; then
+ CURRENT_DATE=$(date +%s)
+ LAST_ACCESS_DATE=$(date -d "$LAST_ACCESS" +%s 2>/dev/null || date -j -f
"%Y-%m-%dT%H:%M:%SZ" "$LAST_ACCESS" +%s)
+ DAYS_IDLE=$(( ($CURRENT_DATE - $LAST_ACCESS_DATE) / 86400 ))
+
+ echo "Days since last access: $DAYS_IDLE"
+
+ if [ $DAYS_IDLE -gt 90 ]; then
echo "Table has been idle for more than 90 days. Dropping table..."
# Drop table via Gravitino API
DROP_RESPONSE=$(curl -s -X DELETE -H "Accept:
application/vnd.gravitino.v1+json" \
"http://localhost:8090/api/metalakes/metalake_demo/catalogs/${CATALOG}/schemas/${SCHEMA}/tables/${TABLE}";)
echo "Drop response: $DROP_RESPONSE"
echo "Table ${CATALOG}.${SCHEMA}.${TABLE} dropped successfully"
- else
+ else
echo "Table is still active. No action needed."
- fi
-else
- echo "No last access time found. Skipping..."
-fi
-EOF
+ fi
+ else
+ echo "No last access time found. Skipping..."
+ fi
+ EOF
-chmod +x /tmp/drop_unused_tables.sh
+ chmod +x /tmp/drop_unused_tables.sh
-# Copy the script into the Gravitino container
-docker cp /tmp/drop_unused_tables.sh
playground-gravitino:/tmp/drop_unused_tables.sh
+ # Copy the script into the Gravitino container
+ docker cp /tmp/drop_unused_tables.sh
playground-gravitino:/tmp/drop_unused_tables.sh
-# Make it executable in the container
-docker exec playground-gravitino chmod +x /tmp/drop_unused_tables.sh
+ # Make it executable in the container
+ docker exec playground-gravitino chmod +x /tmp/drop_unused_tables.sh
-# Register the job template
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
+ # Register the job template
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
"jobTemplate": {
"name": "drop_unused_table_job",
"jobType": "shell",
@@ -687,91 +699,100 @@ curl -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
"customFields": {},
"scripts": []
}
- }' \
- http://localhost:8090/api/metalakes/metalake_demo/jobs/templates
-```
+ }' \
+ http://localhost:8090/api/metalakes/metalake_demo/jobs/templates
+ ```
-**Step 6: Run the Job to Drop Unused Tables**
+6. Run the job to drop unused tables:
-Execute the job for the customers table:
+ Execute the job for the customers table:
-```shell
-# Run job for the customers table (should drop it since it's > 90 days old)
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
+ ```shell
+ # Run job for the customers table (should drop it since it's > 90 days old)
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
"jobTemplateName": "drop_unused_table_job",
"jobConf": {
"catalog": "catalog_hive",
"schema": "sales",
"table": "customers"
}
- }' \
- http://localhost:8090/api/metalakes/metalake_demo/jobs/runs
-```
+ }' \
+ http://localhost:8090/api/metalakes/metalake_demo/jobs/runs
+ ```
-The response will contain a `jobRunId` that you can use to check the job
status.
+ The response will contain a `jobRunId` that you can use to check the job
status.
-**Step 7: Verify the Job Result**
+7. Verify the job result:
-Check the job execution status and result:
+ Check the job execution status and result:
-```shell
-# Get the job run details (replace {jobRunId} with the actual ID from Step 6
response)
-curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
- http://localhost:8090/api/metalakes/metalake_demo/jobs/runs/{jobRunId}
+ ```shell
+ # Get the job run details (replace {jobRunId} with the actual ID from the
response in the previous step)
+ curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+ http://localhost:8090/api/metalakes/metalake_demo/jobs/runs/{jobRunId}
-# Example: If jobRunId is "job-123"
-curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
- http://localhost:8090/api/metalakes/metalake_demo/jobs/runs/job-123
-```
+ # Example: If jobRunId is "job-123"
+ curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+ http://localhost:8090/api/metalakes/metalake_demo/jobs/runs/job-123
+ ```
-The response will show:
-- **status**: Job status (`QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`,
`CANCELLING`, `CANCELLED`)
-- **startTime**: When the job started
-- **endTime**: When the job completed
-- **output**: Job execution output/logs
+ The response will show:
+ - **status**: Job status (`QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`,
`CANCELLING`, `CANCELLED`)
+ - **startTime**: When the job started
+ - **endTime**: When the job completed
+ - **output**: Job execution output/logs
-You can also verify the table was actually dropped:
+ Verify the table was dropped:
-```shell
-# Check if the table still exists (should show it's gone)
-docker exec -it playground-trino trino --execute "SHOW TABLES FROM
catalog_hive.sales"
+ ```shell
+ # Check if the table still exists (should show it's gone)
+ docker exec -it playground-trino trino --execute "SHOW TABLES FROM
catalog_hive.sales"
-# Or try to query the dropped table (should fail with "Table not found")
-docker exec -it playground-trino trino --execute "SELECT * FROM
catalog_hive.sales.customers LIMIT 1"
-```
+ # Or try to query the dropped table (should fail with "Table not found")
+ docker exec -it playground-trino trino --execute "SELECT * FROM
catalog_hive.sales.customers LIMIT 1"
+ ```
-If the table was successfully dropped, you'll see an error like:
-```
-Query failed: line 1:15: Table 'hive.sales.customers' does not exist
-```
-
-**Key Concepts:**
-
-- **Statistics**: Track custom metrics like `custom-lastAccessTime` to monitor
table usage
-- **Policies**: Define governance rules to identify tables that meet certain
criteria (e.g., idle for 90+ days)
-- **Jobs**: Execute automated actions (drop tables) based on policy evaluation
-- **Metadata-driven actions**: Use Gravitino's metadata (statistics, policies)
to drive data governance decisions
-
-This approach enables:
-- ✅ Automated data lifecycle management
-- ✅ Cost reduction by removing unused data
-- ✅ Compliance with data retention policies
-- ✅ Centralized governance across multiple catalogs
+ If the table was successfully dropped, you'll see an error like:
+ ```
+ Query failed: line 1:15: Table 'hive.sales.customers' does not exist
+ ```
For more details, refer to:
- [Manage Statistics in
Gravitino](https://gravitino.apache.org/docs/latest/manage-statistics-in-gravitino)
- [Manage Policies in
Gravitino](https://gravitino.apache.org/docs/latest/manage-policies-in-gravitino)
- [Manage Jobs in
Gravitino](https://gravitino.apache.org/docs/latest/manage-jobs-in-gravitino)
-## NOTICE
+### Gravitino with LlamaIndex
+
+The Gravitino Playground also provides a simple RAG demo with LlamaIndex. The
demo shows the
+ability to use Gravitino to manage both tabular and non-tabular datasets,
connecting to
+LlamaIndex as a unified data source, then use LlamaIndex and LLM to query both
tabular and
+non-tabular data with one natural language query.
+
+The demo notebook is `gravitino_llama_index_demo.ipynb` in Jupyter at
[http://localhost:18888](http://localhost:18888).
+
+In the demo scenario, structured city statistics live in MySQL and detailed
city introductions
+live in PDF files. A single natural language question needs answers drawn from
both.
-If you want to clean cache files, you can delete the directory `data` of this
repo.
+You will manage the MySQL table with a relational catalog and the PDF files
with a fileset
+catalog, treating Gravitino as a unified data source for LlamaIndex to index
both. An LLM then
+answers natural language queries over the combined data.
-## ASF Incubator disclaimer
+Note: the demo requires `OPENAI_API_KEY` to be set in
`gravitino_llama_index_demo.ipynb` as shown
+below. `OPENAI_API_BASE` is optional.
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = ""
+os.environ["OPENAI_API_BASE"] = ""
+```
+
+## NOTICE
-Apache Gravitino is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of
all newly accepted projects until a further review indicates that the
infrastructure, communications, and decision making process have stabilized in
a manner consistent with other successful ASF projects. While incubation status
is not necessarily a reflection of the completeness or stability of the code,
it does indicate that the proje [...]
+The playground stores state in Docker volumes and in the `data` directory of
this repo. See the Stop section above for how to reset the playground
completely.
<sub>Apache®, Apache Gravitino™, Apache Hive™, Apache
Iceberg™, and Apache Spark™ are either registered trademarks or
trademarks of the Apache Software Foundation in the United States and/or other
countries.</sub>
diff --git a/docker-compose.yaml b/docker-compose.yaml
index 4df8f7d..603006f 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -30,6 +30,8 @@ services:
entrypoint: /bin/bash /tmp/hive/init.sh
volumes:
- ./init/hive:/tmp/hive
+ - hive-hdfs:/tmp/hadoop-root
+ - hive-metastore-db:/var/lib/mysql
healthcheck:
test: ["CMD", "/tmp/check-status.sh"]
interval: 10s
@@ -121,6 +123,7 @@ services:
- "15432:5432"
volumes:
- ./init/postgres:/docker-entrypoint-initdb.d/
+ - postgres-data:/var/lib/postgresql/data
mysql:
image: mysql:${MYSQL_IMAGE_TAG}
@@ -133,6 +136,7 @@ services:
ports:
- "13306:3306"
volumes:
+ - mysql-data:/var/lib/mysql
- ./init/mysql:/docker-entrypoint-initdb.d/
- ./healthcheck:/tmp/healthcheck
command:
@@ -201,3 +205,9 @@ services:
- ./init/grafana/grafana.ini:/etc/grafana/grafana.ini
-
./init/grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
- ./init/grafana/dashboards:/etc/grafana/provisioning/dashboards
+
+volumes:
+ mysql-data:
+ postgres-data:
+ hive-hdfs:
+ hive-metastore-db:
diff --git a/init/mysql/init.sql b/init/mysql/init.sql
index d0c09c9..09efa52 100644
--- a/init/mysql/init.sql
+++ b/init/mysql/init.sql
@@ -27,3 +27,23 @@ CREATE TABLE IF NOT EXISTS `demo_llamaindex`.`city_stats` (
INSERT INTO `demo_llamaindex`.`city_stats` (city_name, population, country)
VALUES ("Toronto", 2930000, "Canada");
INSERT INTO `demo_llamaindex`.`city_stats` (city_name, population, country)
VALUES ("Tokyo", 13960000, "Japan");
INSERT INTO `demo_llamaindex`.`city_stats` (city_name, population, country)
VALUES ("Berlin", 3645000, "Germany");
+
+CREATE DATABASE IF NOT EXISTS `db`;
+
+CREATE TABLE IF NOT EXISTS `db`.`iceberg_tables` (
+ `catalog_name` VARCHAR(255) NOT NULL,
+ `table_namespace` VARCHAR(255) NOT NULL,
+ `table_name` VARCHAR(255) NOT NULL,
+ `metadata_location` VARCHAR(1000),
+ `previous_metadata_location` VARCHAR(1000),
+ `iceberg_type` VARCHAR(5),
+ PRIMARY KEY (`catalog_name`, `table_namespace`, `table_name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
+
+CREATE TABLE IF NOT EXISTS `db`.`iceberg_namespace_properties` (
+ `catalog_name` VARCHAR(255) NOT NULL,
+ `namespace` VARCHAR(255) NOT NULL,
+ `property_key` VARCHAR(255) NOT NULL,
+ `property_value` VARCHAR(1000),
+ PRIMARY KEY (`catalog_name`, `namespace`, `property_key`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
diff --git a/init/spark/iceberg-seed.sh b/init/spark/iceberg-seed.sh
new file mode 100755
index 0000000..3fa6c5e
--- /dev/null
+++ b/init/spark/iceberg-seed.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Seed the Iceberg REST catalog with demo data (analytics.orders).
+# Idempotent: skips when the analytics namespace already exists.
+
+set -u
+
+SPARK_HOME=/opt/spark
+SEED_SQL=/tmp/spark/iceberg-seed.sql
+# The metalake and catalogs are created unauthenticated at startup, so that
+# identity owns them. The seed uses the same identity, which works both with
+# and without --enable-auth.
+AUTH_CONFS=""
+
+echo "[seed] Waiting for the Iceberg REST service on gravitino:9001..."
+for i in $(seq 1 60); do
+ if (echo > /dev/tcp/gravitino/9001) 2>/dev/null; then
+ break
+ fi
+ sleep 5
+done
+if ! (echo > /dev/tcp/gravitino/9001) 2>/dev/null; then
+ echo "[seed] Iceberg REST service not reachable after timeout; skipping
seed."
+ exit 0
+fi
+
+cd "${SPARK_HOME}" || exit 0
+
+if bin/spark-sql ${AUTH_CONFS} -e "SHOW NAMESPACES IN catalog_rest"
2>/dev/null | grep -q "^analytics$"; then
+ echo "[seed] analytics namespace already exists; skipping seed."
+ exit 0
+fi
+
+echo "[seed] Seeding Iceberg demo data..."
+bin/spark-sql ${AUTH_CONFS} -f "${SEED_SQL}" 2>&1 | sed 's/^/[seed] /'
+echo "[seed] Done."
diff --git a/init/spark/iceberg-seed.sql b/init/spark/iceberg-seed.sql
new file mode 100755
index 0000000..ddd85f1
--- /dev/null
+++ b/init/spark/iceberg-seed.sql
@@ -0,0 +1,35 @@
+-- Seed data for the Iceberg REST catalog (catalog_rest in Spark,
catalog_iceberg in Trino).
+USE catalog_rest;
+
+-- Two inserts on purpose: the table gets two snapshots, so snapshot history
and
+-- time travel queries are demonstrable out of the box.
+-- customer_id values overlap with the Hive sales.customers seed data, so
+-- cross-catalog joins between Hive and Iceberg work without setup.
+
+CREATE DATABASE IF NOT EXISTS analytics;
+
+CREATE TABLE IF NOT EXISTS analytics.orders (
+ order_id BIGINT,
+ customer_id INT,
+ order_date DATE,
+ region STRING,
+ amount DECIMAL(10,2),
+ status STRING
+) USING iceberg
+PARTITIONED BY (region);
+
+INSERT INTO analytics.orders VALUES
+ (1001, 11, DATE '2026-05-02', 'west', 245.50, 'shipped'),
+ (1002, 12, DATE '2026-05-03', 'east', 89.99, 'shipped'),
+ (1003, 11, DATE '2026-05-10', 'west', 512.00, 'returned'),
+ (1004, 14, DATE '2026-05-14', 'south', 133.25, 'shipped'),
+ (1005, 15, DATE '2026-05-21', 'east', 760.10, 'pending'),
+ (1006, 12, DATE '2026-06-01', 'east', 45.00, 'shipped'),
+ (1007, 16, DATE '2026-06-04', 'west', 310.75, 'shipped'),
+ (1008, 11, DATE '2026-06-09', 'west', 22.10, 'cancelled'),
+ (1009, 17, DATE '2026-06-15', 'south', 199.99, 'shipped'),
+ (1010, 15, DATE '2026-06-22', 'east', 405.60, 'shipped');
+
+INSERT INTO analytics.orders VALUES
+ (1011, 18, DATE '2026-07-01', 'west', 650.00, 'pending'),
+ (1012, 14, DATE '2026-07-03', 'south', 77.45, 'shipped');
diff --git a/init/spark/init.sh b/init/spark/init.sh
index 1a36307..b3de064 100644
--- a/init/spark/init.sh
+++ b/init/spark/init.sh
@@ -25,4 +25,5 @@ cp /tmp/spark/packages/${SPARK_CONNECTOR_JAR}
/opt/spark/jars/${SPARK_CONNECTOR_
cp /tmp/spark/packages/mysql-connector-java-8.0.27.jar
/opt/spark/jars/mysql-connector-java-8.0.27.jar
cp /tmp/spark/packages/kyuubi-spark-authz-shaded_2.12-1.9.2.jar
/opt/spark/jars/kyuubi-spark-authz-shaded_2.12-1.9.2.jar
sh /tmp/common/init_metalake_catalog.sh
+/bin/bash /tmp/spark/iceberg-seed.sh > /tmp/iceberg-seed.log 2>&1 &
tail -f /dev/null
diff --git a/playground.sh b/playground.sh
index 1785a79..6e75b8a 100755
--- a/playground.sh
+++ b/playground.sh
@@ -26,9 +26,9 @@ playground_dir="$(
playgroundRuntimeName="gravitino-playground"
requiredDiskSpaceGB=25
-requiredRamGB=6
+requiredRamGB=8
requiredCpuCores=2
-requiredPorts=(6080 8090 9001 3307 19000 19083 60070 13306 15342 18080 18888
19090 13000)
+requiredPorts=(6080 8090 9001 3307 19000 19083 60070 13306 15432 14040 18080
18888 19090 13000)
dockerComposeCommand=""
testDocker() {
@@ -43,11 +43,11 @@ testDocker() {
exit 1
fi
- for containerId in $(docker ps -a | grep hello-world | awk '{print $1}'); do
- docker rm $containerId
+ for containerId in $(docker ps -aq --filter ancestor=hello-world); do
+ docker rm "${containerId}" >/dev/null 2>&1
done
- for imageTag in $(docker images | grep hello-world | awk '{print $2}'); do
- docker rmi hello-world:$imageTag
+ for imageId in $(docker images -q hello-world); do
+ docker rmi -f "${imageId}" >/dev/null 2>&1
done
}
