mengxianwen1 opened a new issue, #11027:
URL: https://github.com/apache/gravitino/issues/11027
### Version
main branch
### Describe what's wrong
When creating a Lance table through Spark SQL (CTAS: _CREATE TABLE ... AS
SELECT_) via lance-rest-server, the Gravitino Web UI displays empty columns for
the table, even though the actual Lance dataset on S3 contains data with the
correct
schema.
In contrast, directly calling the lance-rest-server`
/v1/table/{id}/create` API (with arrowStreamBody) results in the columns being
correctly displayed in Gravitino Web UI.
The root cause is that Spark CTAS uses StagingTableCatalog, which follows
a staged creation flow:
1. `stageCreate()` → calls `declareTable()` → mapped to lance-rest-server
`/v1/table/{id}/create-empty` → only registers table metadata (name + location)
in Gravitino with empty columns and _lance.create-empty=true_ property
2. Spark writes data to S3 → `StagedCommit` directly writes to S3 via Lance
Java SDK (CommitBuilder), embedding the actual schema into the Lance dataset
3. `StagedCommit.commit()` → finishes writing to S3, but never calls back to
lance-rest-server to update the Gravitino metadata with actual columns
The staged path decouples metadata registration from data writing, and
`StagedCommit.commit()` only writes to S3 without updating Gravitino's stored
schema.
### Error message and/or stacktrace
No error or exception is thrown. The table is created successfully and data
can be read via Spark. The issue is that the Gravitino metadata store has empty
columns, so the Web UI and any Gravitino client API (loadTable) show no column
information.
### How to reproduce
1. Set up Gravitino with lance-rest-server and a Lance catalog with S3
storage options
2. Connect Spark to lance-rest-server using the Lance Spark connector
3. Execute a CTAS statement: _CREATE TABLE test_table USING lance AS SELECT
1 AS id, 'hello' AS name_
4. Check the table in Gravitino Web UI → columns are empty
5. Query the table via Spark → data and columns are correctly returned
6. Compare: directly calling POST `/v1/table/{id}/create` with Arrow IPC
stream body → columns are displayed correctly in Web UI
### Additional context
**Proposed fix:** Override `loadTable()` in LanceTableOperations to
implement a self-healing mechanism: when _lance.create-empty=true_ is detected,
read the actual schema from the Lance dataset on S3, update the Gravitino
entity store with the
correct columns, and clear the _lance.create-empty_ flag. This is a
one-time self-heal — subsequent `loadTable()` calls return the correct metadata
via the normal path.
**Key files involved:**
-
`catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java`
-
`catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableDelegator.java`
-
`lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/gravitino/GravitinoLanceTableOperations.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
