mchades commented on issue #10561:
URL: https://github.com/apache/gravitino/issues/10561#issuecomment-4508728295
Thanks for reporting. The root cause is a hard limitation of the Hive
Metastore filter API, but there is a simple user-side workaround.
**Root cause.** `IMetaStoreClient#listTableNamesByFilter` is backed by HMS's
`ExpressionTree` parser:
1. Only `owner` / `lastAccessTime` / `parameters` are filterable (no SerDe /
InputFormat).
2. For `parameters`, JDOQL `this.parameters.get("<key>")` requires an
**exact, dot-free** key (only `discover__partitions` is special-cased).
3. The filter-string grammar itself does not accept identifiers containing
`.`.
So `spark.sql.sources.provider` is unreachable from a server-side filter. We
confirmed Trino has the same constraint
([`HiveUtil#isHudiTable`](https://github.com/trinodb/trino/blob/master/plugin/trino-hive/src/main/java/io/trino/plugin/hive/util/HiveUtil.java)
inspects InputFormat only after loading the full `Table` and does not filter
at list time).
**What we changed** (PR coming):
- Tightened the best-effort filter (`table_type=ICEBERG|PAIMON`,
`provider=hudi`) and fixed a `startsWith` over-removal bug for `_ro`/`_rt`
siblings.
- Documented the limitation and workaround on the `list-all-tables` property
and in the Hive catalog docs.
**Workaround for Spark-managed Hudi tables.** Add a dot-free `provider=hudi`
property:
```sql
ALTER TABLE <db>.<table> SET TBLPROPERTIES ('provider'='hudi');
```
or at write time:
```scala
df.write.format("hudi")
.option("hoodie.datasource.hive_sync.table_properties", "provider=hudi")
.saveAsTable("<db>.<table>")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
