wangyong9999 opened a new issue, #305:
URL: https://github.com/apache/paimon-cpp/issues/305
### Problem
PK point reads incur two avoidable planning costs:
1. `PrimaryKeyIndexBatchScan::CreatePlan()` loads the snapshot after
creating its data plan, even when the underlying scan already holds that same
snapshot. Remote storage therefore sees another open/read and JSON parse.
2. `OptionsUtils::GetOptionalValueFromMap()` handles an absent setting by
constructing a formatted `NotExist` status and discarding it.
`CoreOptions::FromMap()` requests many defaults, so concurrent reader
construction repeatedly pays for error formatting and iostream/locale work on a
normal path.
This is a performance improvement, not a reported result-correctness bug.
The two costs are independent; neither requires a process-wide cache.
### Reproduction
Baseline: apache/paimon-cpp `ded7cfb84005a6315f4778253411dc78f7fa4cb3`.
Proposed implementation: #303, `0345e430afea1829b856e0b2ac2ea4b90cc071e7`.
- Snapshot: run
`FileFormat/PrimaryKeySortedIndexInteTest.HistoricalSnapshotsMixedFilesAndDisabledIndex/*`
with the tracking filesystem assertions in #303. For snapshots 2, 3 and 5,
both Parquet and ORC open the selected snapshot twice before the change, once
afterward, with unchanged result checks. The negative control uses the same
test with the unmodified implementation.
- Configuration: concurrently call `CoreOptions::FromMap({{"file.format",
"parquet"}})` in a loop (2,000 calls per worker; 1/8/32/64 workers released
from a common start barrier). Compile the caller and `core_options.cpp` with
C++17, `-O2 -Wall`, using the original or modified OptionsUtils header and
otherwise identical dependencies. Sum completed calls and divide by elapsed
wall time. Two repetitions were made, reversing the baseline/fixed execution
order.
### Measured effect
Configuration-only microbenchmark on Linux x86_64 (aggregate calls/s,
observed range of two runs):
| Workers | Before | After |
|---|---:|---:|
| 1 | 14,783–14,943 | 164,082–165,259 |
| 8 | 11,643–12,118 | 1,184,340–1,255,940 |
| 32 | 11,466–11,484 | 1,161,670–1,289,440 |
| 64 | 11,962–12,042 | 1,073,330–1,189,250 |
These short microbenchmark runs identify the parsing cost; their speedup is
not query throughput or a capacity claim.
In a separate application-level PK read test (3 servers, 96 logical CPUs and
25 Gbit/s networking each, SSD-backed HDFS, 12M rows, 8 fixed buckets, uniform
successful-key lookup returning a 1 KiB payload), changing only the
option-parsing implementation at C128 increased successful throughput from
2,585 to 2,804 QPS (+8.5%); AVG fell from 49.5 to 45.7 ms and P99 from 107.7 to
88.8 ms. Each version used two 60-second runs, the same snapshot/file layout
and placement, and the same admission limit. Results include admission errors
(434 before, 411 after); this does not establish error-free serving capacity.
Per-run variation exists; raw before/after details are in #303.
### Expected outcome
- Reuse the data plan's snapshot for index planning when IDs match; retain
the existing load path otherwise. Do not change snapshot discovery or freshness.
- Return an empty optional without constructing an error when an optional
key is absent. Preserve required-key errors, invalid-value errors and
explicitly supplied values.
- Keep the regression tests and document the measured effects and
limitations.
Implementation: #303. Close this issue when that PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
