Gabriel39 opened a new pull request, #66247:
URL: https://github.com/apache/doris/pull/66247
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The Paimon JNI read path had several correctness and operational guardrail
gaps:
1. `read.batch-size` accepted zero, negative, or excessively large values.
In particular, zero can make the vectorized reader return without advancing
input.
2. `scan.manifest.parallelism=0` can block on a zero-permit semaphore, while
a value above the CPU-sized default causes Paimon to replace a JVM-static
executor and leaves the new pool visible to later queries.
3. JNI projected fields used comma-delimited framing. A legal quoted column
such as `` `region,code` `` was decoded as two fields, so it no longer matched
the single `columns_types` entry.
4. DEBUG logging printed the complete scanner parameter map, which may
contain storage or JDBC credentials.
5. An unchecked exception from connector property validation did not roll
back the tentative catalog property mutation.
6. Catalog-level `paimon.table-option.*` accepted options which can select a
different table context or affect writes after Doris has already bound the
original schema.
7. Every scanner statistics collection enumerated all JVM threads and
allocated a `ThreadInfo[]`, although the Paimon async-reader pool is JVM-global.
8. Safe reader tuning could not be supplied for an individual
`table@options(...)` relation.
### How does this PR solve it?
- Introduces one shared reader-option validator and allows only:
- `read.batch-size`: `1..65536`
- `file-reader-async-threshold`: `1 MB..1 GB`
- Applies explicit precedence without mutating the physical Paimon table:
`relation @options` > Doris catalog option > physical table option >
Paimon default.
- Bounds relation-scoped manifest parallelism to
`1..Runtime.availableProcessors()`, preserving the JVM-global executor capacity
invariant.
- Encodes each projected field independently with Base64 and keeps the
legacy parameter for rolling upgrades. For example, `region,code` is
transported as one encoded token instead of being split at the comma.
- Replaces raw parameter logging with a fixed summary containing only batch
size and projected-field count.
- Rolls catalog properties back for both checked and unchecked validation
failures before publishing the final mutation.
- Rejects context-changing or write-oriented catalog options such as
`branch`, `path`, snapshot/tag selectors, and write settings. This is important
because changing `branch` after Doris binds a schema could otherwise make scan
planning use another branch with a different schema.
- Shares one async-reader thread-count sample across scanners for one second.
Example query-level tuning:
```sql
SELECT id
FROM paimon_table@options(
'read.batch-size' = '4096',
'file-reader-async-threshold' = '32 MB');
```
Two aliases of the same table may use different values in the same statement
because each relation receives its own `Table.copy(...)`:
```sql
SELECT small.id
FROM paimon_table@options('read.batch-size' = '1') small
JOIN paimon_table@options('read.batch-size' = '8192') large
ON small.id = large.id;
```
### Thread statistics microbenchmark
Dataset and method:
- One JVM process per measurement.
- 16, 128, or 512 idle worker threads whose names match the Paimon
async-reader prefix. Including JVM service threads, the observed live-thread
counts were 22, 134, and 518.
- 200 warm-up calls followed by 2,000 measured statistics calls.
- Old path: every call executes `getAllThreadIds()` and `getThreadInfo(ids,
0)`, then scans the returned `ThreadInfo[]`.
- New path: the exact one-second shared-cache algorithm used by the scanner;
the cache is primed before measurement.
- Elapsed wall time and bytes allocated by the calling thread were measured
with `ThreadMXBean`. A checksum blackhole consumes every returned count.
| Named workers | Live JVM threads | Old time / 2,000 calls | Cached time /
2,000 calls | Old allocated bytes | Cached allocated bytes |
|---:|---:|---:|---:|---:|---:|
| 16 | 22 | 41.507 ms | 0.429 ms | 12,600,576 | 0 |
| 128 | 134 | 139.544 ms | 0.417 ms | 71,430,512 | 0 |
| 512 | 518 | 581.066 ms | 0.398 ms | 274,048,216 | 0 |
This microbenchmark isolates statistics collection rather than scan I/O. It
shows that the old cost grows with JVM thread count, while cache hits avoid
both the full thread walk and its per-call allocation. The tradeoff is that
this profile gauge can be up to one second stale; it does not affect query
execution or scheduling.
### Tests
Passed locally:
- `PaimonJniScannerTest`: 13 tests, 0 failures, Maven build cache disabled.
- FE checkstyle: 0 violations.
- Doris C++ format check.
- Groovy compilation of the new P0 regression suite.
- Paimon 1.3.1 option probe covering both bounds and unsafe-option rejection.
- Thread-statistics microbenchmark described above.
Added coverage:
- BE unit test for delimiter-safe JNI field encoding.
- Scanner unit tests for quoted identifiers, real DEBUG log
capture/redaction, and shared cache TTL.
- FE unit tests for catalog rollback, catalog/relation option bounds,
unsafe-option rejection, manifest parallelism, and relation-local copies.
- P0 external-Paimon regression for quoted comma/hash/space/Unicode
identifiers, projection and predicate reads, relation precedence, independent
aliases, invalid boundaries, unsafe catalog options, and failed-ALTER rollback.
Not run locally:
- Full FE/BE compilation, FE/BE unit-test binaries, and the external-Paimon
P0 environment. The checkout does not contain the required installed
third-party toolchain (`protoc` included). CI buildall and regression checks
are requested on this PR.
### Release note
Paimon catalogs and relation `@options` now support bounded
`read.batch-size` and `file-reader-async-threshold` reader tuning. Unsafe
catalog table options and unsafe manifest parallelism are rejected. JNI
projection supports quoted identifiers containing delimiter characters, scanner
DEBUG logs no longer include raw parameters, failed catalog validation is
atomic, and scanner statistics reuse a short-lived JVM-wide thread-count sample.
### Check List (For Author)
- Test
- Unit Test
- Regression test case added
- Behavior changed:
- Paimon dynamic reader settings now use an explicit safe allowlist and
documented bounds.
- Does this need documentation?
- Yes. Supported options, precedence, bounds, examples, and the
statistics-cache tradeoff are documented above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
