contrueCT opened a new issue, #3190:
URL: https://github.com/apache/hugegraph/issues/3190
## Background
PR #3140 introduced Server-side input-ID ordering for HStore query results.
Issue #3180 later exposed an interaction between that ordering path and
flattened index queries: InputOrderIterator can probe the next query while
detecting the current query boundary, and QueryList can update the shared
origin ConditionQuery resultsFilter during that probe.
PR #3182, reviewed at head acbee16753e4bf0aba35e114e23470aab0417ccd, is
intended as a bounded correctness hotfix. It applies index-candidate filtering
before input-order restoration and protects cache paths that could otherwise
bypass that filtering. This fixes the reported failure without redesigning the
iterator boundary itself.
A follow-up refactor is needed so correctness no longer depends on the
buffering and side effects of ordinary Iterator.hasNext() calls.
## Current problems
### 1. Boundary detection can activate the next query
InputOrderIterator.fetchBatch() currently uses the following sequence:
~~~java
long queryVersion = this.queryResults.queryVersion();
List<Query> queries = this.queryResults.currentQueries();
do {
results.add(this.origin.next());
} while (this.origin.hasNext() &&
queryVersion == this.queryResults.queryVersion());
~~~
The origin.hasNext() call is not a read-only boundary check. It can advance
FlatMapperIterator or QueryTrackingIterator into the next flattened query.
QueryList.IndexQuery may then update the shared origin query's resultsFilter
before InputOrderIterator observes that queryVersion changed.
PR #3182 moves filtering into a safer position, but the underlying contract
remains implicit: the current element must be filtered and buffered before
another hasNext() call can cross the query boundary.
### 2. Input-order restoration is installed from the first active segment
QueryResults.keepInputOrderIfNeeded() decides whether to create
InputOrderIterator from currentQueries after the first successful
origin.hasNext() call.
This can miss a later segment whose ordering requirement differs from the
first segment:
~~~text
segment A: mustSortByInput = false, backend = [1]
segment B: mustSortByInput = true, input = [3, 2], backend = [2, 3]
expected: [1, 3, 2]
possible current result: [1, 2, 3]
~~~
Input-order requirements must therefore be evaluated per query batch, not
once for the flattened result stream.
### 3. Query context is represented by mutable stream-wide state
queryVersion, currentQueries, resultsFilter, input IDs, and mustSortByInput
collectively describe the active query segment, but they are stored or observed
through different objects. Mapper, filter, flat-map, paging, and cache paths do
not share one explicit boundary contract.
This makes it difficult to prove that each result is parsed, TTL-checked,
condition-filtered, and ordered with the context that produced it.
## Goal
Introduce an explicit query-batch or query-segment boundary for query result
iteration.
The implementation does not have to use a class named QueryBatch, but it
must establish these invariants:
- A batch carries the query metadata required to process its results,
including its query or query snapshot, input IDs, input-order requirement, and
filter context.
- Each result is processed with the context of the batch that produced it.
- Parse, TTL filtering, condition filtering, and input-order restoration
complete at a batch-aware boundary before that context is discarded.
- Checking whether the current batch has more results does not activate the
next query or fetch the next backend page.
- Mixed batches can independently choose whether input-order restoration is
required.
- Ordinary Iterator.hasNext() buffering behavior is not part of the
correctness contract.
## Suggested direction
Prefer an explicit batch abstraction exposed by QueryResults and produced by
QueryList/PageEntryIterator. InputOrderIterator should consume one batch
directly instead of inferring boundaries by comparing queryVersion after
calling origin.hasNext().
A smaller boundary-aware iterator API, such as hasNextInCurrentQuery() plus
an explicit operation that advances to the next query, may also be considered.
However, the boundary must propagate through QueryTrackingIterator,
MapperIterator, FilterIterator, FlatMapperIterator, page iteration, and any
other wrapper that can advance the origin. Adding such a method only to
InputOrderIterator would leave the same implicit coupling elsewhere.
## Scope
The expected implementation area includes:
- QueryResults and InputOrderIterator
- QueryList and QueryTrackingIterator
- PageEntryIterator and BatchIdHolder paths
- Mapper, filter, and flat-map wrappers that can cross a query boundary
- GraphTransaction processing order
- Cache behavior when a query still requires condition filtering
After the explicit boundary is in place, reassess whether
queryVersion/currentQueries and the post-filter cache guards introduced by
#3182 can be removed or simplified. Removal is not a requirement unless
equivalent correctness and cache behavior are demonstrated.
## Non-goals
- Do not redesign the HStore partition merge, ORDER_BY_KEY protocol, or
physical-key cursor introduced by #3140.
- Do not change the public Gremlin, REST, or graph-query result semantics.
- Do not introduce a request-level fallback between old and new query
execution paths.
- Do not combine unrelated cache-policy or index-planner changes into this
refactor.
## Acceptance criteria
Add focused RED tests before replacing the current boundary behavior.
- [ ] Two flattened queries with different filters retain the correct
results for both batches; activating query B cannot cause an element from query
A to be evaluated with filter B.
- [ ] Mixed ordering requirements work in both directions, including a first
batch with mustSortByInput=false followed by a batch with mustSortByInput=true.
- [ ] Empty intermediate batches do not leak filter or ordering state into
adjacent batches.
- [ ] Mapper results that are null or expand one backend entry into multiple
elements preserve the originating batch context.
- [ ] Multiple BatchIdHolder batches are processed independently.
- [ ] The final element of a batch still receives TTL and condition
filtering before the next batch becomes active.
- [ ] Exhausting or inspecting the current batch does not fetch or prepare
the next backend page; tests assert backend/page fetch count, not only consumed
element count.
- [ ] Existing #3140 range-index ordering, limit, offset, paging, and cursor
behavior remains unchanged.
- [ ] The vertex and edge regressions from #3180/#3182 pass on the
applicable memory, RocksDB, and HStore test profiles.
- [ ] Warm-cache PRIMARY_KEY queries, PRIMARY_KEY queries with residual
conditions, and vertex/edge INDEX + LABEL queries remain correct.
- [ ] Close and exception propagation still release the active and remaining
iterator/page resources exactly once.
## References
- #3180
- #3140
- #3182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]