Gabriel39 opened a new pull request, #70010:
URL: https://github.com/apache/doris/pull/70010
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Queries such as:
```sql
SELECT COUNT(*) FROM example_catalog.information_schema.schemata;
```
only need database and catalog names. However, `SchemaSchemataScanner` uses
the default `getDbNames` RPC, whose FE handler calls `catalog.getAllDbs()` to
also return database IDs. For external catalogs, this resolves every database
object even though SCHEMATA never reads those IDs. A large namespace list can
therefore churn a much smaller database-object cache and allocate substantial
temporary metadata while waiting inside a synchronous BE-to-FE RPC. A warm
database-name cache does not remove this work.
This PR adds the optional `include_db_ids` request field, defaulting to
`true`, and lets SCHEMATA set it to `false`. The FE then calls
`catalog.getDbNames()` and omits the unused database IDs. Both paths share the
existing privilege checks, pattern matching, and schema-name formatting.
Callers that need IDs keep the object-enumeration path without additional
intermediate lists.
The new field uses ID 8; field 7 remains reserved. Definitions were checked
on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the optional field
and keep serving the existing response; older BE callers retain the default
ID-inclusive behavior on a new FE.
### Validation
- Added a 100,000-namespace test that rejects any database-object
enumeration and verifies the result-array lengths and omitted IDs.
- Covered privilege/pattern filtering, qualified schema names,
empty/unavailable catalogs, existing ID-returning callers, and binary
deserialization of requests without the new field.
- Confirmed the new behavior tests fail on the original handler before
implementing the change. All 7 targeted FE tests pass, including the existing
`FrontendServiceImplTest#testGetDBNames`.
```bash
bash run-fe-ut.sh --run
'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'
```
- FE Checkstyle: passed with zero violations.
- Changed BE scanner: clang-format 16 and C++ syntax-only compilation with
regenerated Thrift headers passed.
An in-process FE-handler benchmark uses the actual `ExternalCatalog` /
`TestExternalCatalog` name and object-cache implementations, a synthetic
metadata provider, and stubbed catalog routing and privilege checks. It
compares the original handler source at master commit
`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler, using the
same dependencies and fixture. Each trial starts with a warm name cache and
cold object cache, whose capacity is 1,000. There are 100,000 synthetic
namespaces plus the two system databases. Settings: JDK 17, 2 GiB heap, 4
active processors; discard one warm-up pair and report medians from three
measured trials.
| Metric | Original master | Patched SCHEMATA path |
| --- | ---: | ---: |
| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |
| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |
| Database objects constructed | 100,002 | 0 |
| Name-provider invocations | 1 | 1 |
| Returned rows | 100,002 | 100,002 |
In this fixture, handler time falls by 55.4% (2.24x faster) and cumulative
allocation falls by 65.7%. Allocation is total bytes allocated during the call,
not peak live heap. This benchmark excludes network transport, Thrift
serialization, and SQL execution; it does not measure a deployed
PostgreSQL/Iceberg environment. Full BE linking and a deployed end-to-end SQL
test were not run locally.
### Release note
Improve `information_schema.schemata` enumeration for large catalogs by
avoiding unnecessary database-object loading.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (FE-handler benchmark described above)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. SQL result semantics and existing ID-returning requests are
preserved.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
