wayne-x-wang opened a new pull request, #7793:
URL: https://github.com/apache/kyuubi/pull/7793
### Why are the changes needed?
Paimon persistent views can be queried through Kyuubi, but JDBC GetTables
omits them because the non-session catalog path only calls
TableCatalog.listTables. A VIEW-only request also returns ordinary tables on
this path.
Discover Paimon's optional SupportView interface through the catalog's class
hierarchy and use its listViews/loadView methods to return persistent views as
VIEW rows with their comments. Apply the existing name-pattern and namespace
handling, and honor table-type filters when ignoreTableProperties=false. The
adapter does not introduce a Paimon runtime dependency and preserves the
original causes of connector errors.
Keep the existing ignoreTableProperties=true fast path: ignore table-type
criteria, return TABLE rows with empty comments, and skip Paimon view
enumeration. Persistent Paimon view listing is therefore supported when this
configuration is false, which is the default.
The table-type filtering correction applies to non-session TableCatalog
implementations generally; Paimon-specific view discovery only applies to
catalogs exposing SupportView.
### How was this patch tested?
Added PaimonMetadataSuite with a local SQLite-backed Paimon JDBC catalog.
Its 13 tests cover table/view type filters, null and lowercase types, JDBC name
patterns, quoted names, namespaces, comments and absent comments, the
identities-only fast path, optional-interface absence, connector failures, and
filtering before loading view metadata.
Added 7 JDBC tests to the default-source SparkCatalogDatabaseOperationSuite
using a non-Paimon TableCatalog fixture. They check returned
catalog/schema/name/type/comment, matching and unsupported types, null and
empty types, metadata-loading errors, and the identities-only fast path. A
catalog that throws from loadTable verifies that the fast path skips metadata
loading while still ignoring type criteria and applying name patterns. The
suite now has 10 tests.
The Paimon suite is wired into a dedicated step in the existing Spark 3.5 /
Java 17 CI job. That step explicitly disables jdbc-shaded because shaded JDBC
classes are produced during package, whereas this step uses a test reactor.
The engine's regular Scalastyle configuration explicitly includes both
src/test/scala and src/test/scala-paimon. The existing Style CI therefore
checks the Paimon test source even without activating the connector test
profile.
With Java 17 and Scala 2.12, run the ordinary V2 tests without the Paimon
profile:
```bash
./build/mvn -B -ntp test \
-pl externals/kyuubi-spark-sql-engine -am \
-P'spark-3.5,scala-2.12,spark-provided,flink-provided,hive-provided,!jdbc-shaded'
\
-Dtest=none \
-DwildcardSuites=org.apache.kyuubi.engine.spark.operation.SparkCatalogDatabaseOperationSuite
```
Run the Paimon tests:
```bash
./build/mvn -B -ntp test \
-pl externals/kyuubi-spark-sql-engine -am \
-P'spark-3.5,scala-2.12,spark-provided,flink-provided,hive-provided,paimon-view-tests,!jdbc-shaded'
\
-Dtest=none \
-DwildcardSuites=org.apache.kyuubi.engine.spark.operation.PaimonMetadataSuite
```
Run Scalastyle and Spotless without the Paimon test profile:
```bash
./build/mvn -B -ntp scalastyle:check spotless:check \
-pl externals/kyuubi-spark-sql-engine -am \
-Pspark-3.5,scala-2.12,spark-provided,flink-provided,hive-provided
```
Current local validation on Spark 3.5.8, Java 17, Scala 2.12:
- Ordinary V2 catalog suite without the Paimon profile: 10 passed.
- Paimon metadata suite with Paimon 2.0.0: 13 passed.
- Final isolation review: both suites ran in the same test JVM, Paimon first
and ordinary V2 second; all 23 tests passed.
- Negative checks on Spark 3.5.6: bypassing table-type filtering fails the
VIEW-only and unsupported-type tests; forcing metadata loading in the fast path
fails its dedicated test. The production source was restored byte-for-byte
before the final passing runs.
- Spotless, Scalastyle, and actionlint checks passed. A local matrix check
confirms that the new Paimon step selects the intended Spark 3.5 / Java 17 cell.
- Scalastyle coverage was verified without the Paimon test profile:
temporary println violations in both test directories produced the two expected
errors. After restoring both files byte-for-byte, Scalastyle checked 109 engine
files with zero errors, and Spotless passed.
Earlier validation of the same production patch on Spark 3.5.6:
- Restoring the original catalog-listing logic caused 7 Paimon tests to
fail, with 6 passing.
- The distribution build completed successfully and compiled the Scala 2.12
and 2.13 engine artifacts. This was a fast-profile build; test execution is
accounted for separately above.
- End-to-end validation used Java 8, Spark 3.5.6, Scala 2.12, Paimon 2.0.0,
and a SQLite-backed JDBC catalog. The existing JDBC regression passed all 5
cases, and the original JDBC probe returned the persistent view as VIEW with
its comment using the existing Kyuubi 1.10.2 client.
Runtime validation does not establish support across other Spark/Paimon
combinations or Hive/REST metastores.
### Was this patch assisted by generative AI tooling?
Assisted-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
