FANNG1 opened a new issue, #13805:
URL: https://github.com/apache/gravitino/issues/13805
### Describe the subtask
#12541 made Lance tables served by the Gravitino Lance REST server usable
from Spark, with no hand-written per-catalog configuration. Flink should reach
the same catalogs: a Flink SQL session configured against Gravitino can `USE
CATALOG <lance_catalog>` and read and write Lance tables through the Gravitino
Lance REST server, without defining each catalog or table by hand.
Why it is needed:
- `flink-connector/flink-common` maps only the `hive`, `lakehouse-iceberg`,
`lakehouse-paimon` and `jdbc-*` providers to Flink catalogs, so a Gravitino
catalog served by the Lance REST server cannot be used from Flink SQL at all.
- Flink users must define Lance tables by path outside Gravitino, which
loses Gravitino as the single metadata and access layer — the problem the
parent issue describes for Spark.
- Engine parity: after #12541, Spark reaches Lance REST catalogs
automatically while Flink cannot reach them.
The design is left open. `design-docs/spark-rest-catalog-registration.md`
lists Flink as a non-goal ("Flink and Trino may reuse the listing endpoint
later"), so this subtask is that follow-up. One fact worth knowing up front:
`org.lance` publishes no Flink artifact to Maven Central (only `lance-spark-*`
and `lance-trino`), and the upstream
[`lance-format/lance-flink`](https://github.com/lance-format/lance-flink)
connector — which already has a `lance-namespace` catalog with `impl=rest` and
modules for Flink 1.18/1.19/1.20 — is still `1.0.0-SNAPSHOT` and unreleased.
Related: Spark subtask #12525, Spark implementation #12541.
### Parent issue
https://github.com/apache/gravitino/issues/11064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
