contrueCT opened a new issue, #3190:
URL: https://github.com/apache/hugegraph/issues/3190

   ## Background
   
   PR #3140 introduced Server-side input-ID ordering for HStore query results. 
Issue #3180 later exposed an interaction between that ordering path and 
flattened index queries: InputOrderIterator can probe the next query while 
detecting the current query boundary, and QueryList can update the shared 
origin ConditionQuery resultsFilter during that probe.
   
   PR #3182, reviewed at head acbee16753e4bf0aba35e114e23470aab0417ccd, is 
intended as a bounded correctness hotfix. It applies index-candidate filtering 
before input-order restoration and protects cache paths that could otherwise 
bypass that filtering. This fixes the reported failure without redesigning the 
iterator boundary itself.
   
   A follow-up refactor is needed so correctness no longer depends on the 
buffering and side effects of ordinary Iterator.hasNext() calls.
   
   ## Current problems
   
   ### 1. Boundary detection can activate the next query
   
   InputOrderIterator.fetchBatch() currently uses the following sequence:
   
   ~~~java
   long queryVersion = this.queryResults.queryVersion();
   List<Query> queries = this.queryResults.currentQueries();
   
   do {
       results.add(this.origin.next());
   } while (this.origin.hasNext() &&
            queryVersion == this.queryResults.queryVersion());
   ~~~
   
   The origin.hasNext() call is not a read-only boundary check. It can advance 
FlatMapperIterator or QueryTrackingIterator into the next flattened query. 
QueryList.IndexQuery may then update the shared origin query's resultsFilter 
before InputOrderIterator observes that queryVersion changed.
   
   PR #3182 moves filtering into a safer position, but the underlying contract 
remains implicit: the current element must be filtered and buffered before 
another hasNext() call can cross the query boundary.
   
   ### 2. Input-order restoration is installed from the first active segment
   
   QueryResults.keepInputOrderIfNeeded() decides whether to create 
InputOrderIterator from currentQueries after the first successful 
origin.hasNext() call.
   
   This can miss a later segment whose ordering requirement differs from the 
first segment:
   
   ~~~text
   segment A: mustSortByInput = false, backend = [1]
   segment B: mustSortByInput = true,  input = [3, 2], backend = [2, 3]
   
   expected: [1, 3, 2]
   possible current result: [1, 2, 3]
   ~~~
   
   Input-order requirements must therefore be evaluated per query batch, not 
once for the flattened result stream.
   
   ### 3. Query context is represented by mutable stream-wide state
   
   queryVersion, currentQueries, resultsFilter, input IDs, and mustSortByInput 
collectively describe the active query segment, but they are stored or observed 
through different objects. Mapper, filter, flat-map, paging, and cache paths do 
not share one explicit boundary contract.
   
   This makes it difficult to prove that each result is parsed, TTL-checked, 
condition-filtered, and ordered with the context that produced it.
   
   ## Goal
   
   Introduce an explicit query-batch or query-segment boundary for query result 
iteration.
   
   The implementation does not have to use a class named QueryBatch, but it 
must establish these invariants:
   
   - A batch carries the query metadata required to process its results, 
including its query or query snapshot, input IDs, input-order requirement, and 
filter context.
   - Each result is processed with the context of the batch that produced it.
   - Parse, TTL filtering, condition filtering, and input-order restoration 
complete at a batch-aware boundary before that context is discarded.
   - Checking whether the current batch has more results does not activate the 
next query or fetch the next backend page.
   - Mixed batches can independently choose whether input-order restoration is 
required.
   - Ordinary Iterator.hasNext() buffering behavior is not part of the 
correctness contract.
   
   ## Suggested direction
   
   Prefer an explicit batch abstraction exposed by QueryResults and produced by 
QueryList/PageEntryIterator. InputOrderIterator should consume one batch 
directly instead of inferring boundaries by comparing queryVersion after 
calling origin.hasNext().
   
   A smaller boundary-aware iterator API, such as hasNextInCurrentQuery() plus 
an explicit operation that advances to the next query, may also be considered. 
However, the boundary must propagate through QueryTrackingIterator, 
MapperIterator, FilterIterator, FlatMapperIterator, page iteration, and any 
other wrapper that can advance the origin. Adding such a method only to 
InputOrderIterator would leave the same implicit coupling elsewhere.
   
   ## Scope
   
   The expected implementation area includes:
   
   - QueryResults and InputOrderIterator
   - QueryList and QueryTrackingIterator
   - PageEntryIterator and BatchIdHolder paths
   - Mapper, filter, and flat-map wrappers that can cross a query boundary
   - GraphTransaction processing order
   - Cache behavior when a query still requires condition filtering
   
   After the explicit boundary is in place, reassess whether 
queryVersion/currentQueries and the post-filter cache guards introduced by 
#3182 can be removed or simplified. Removal is not a requirement unless 
equivalent correctness and cache behavior are demonstrated.
   
   ## Non-goals
   
   - Do not redesign the HStore partition merge, ORDER_BY_KEY protocol, or 
physical-key cursor introduced by #3140.
   - Do not change the public Gremlin, REST, or graph-query result semantics.
   - Do not introduce a request-level fallback between old and new query 
execution paths.
   - Do not combine unrelated cache-policy or index-planner changes into this 
refactor.
   
   ## Acceptance criteria
   
   Add focused RED tests before replacing the current boundary behavior.
   
   - [ ] Two flattened queries with different filters retain the correct 
results for both batches; activating query B cannot cause an element from query 
A to be evaluated with filter B.
   - [ ] Mixed ordering requirements work in both directions, including a first 
batch with mustSortByInput=false followed by a batch with mustSortByInput=true.
   - [ ] Empty intermediate batches do not leak filter or ordering state into 
adjacent batches.
   - [ ] Mapper results that are null or expand one backend entry into multiple 
elements preserve the originating batch context.
   - [ ] Multiple BatchIdHolder batches are processed independently.
   - [ ] The final element of a batch still receives TTL and condition 
filtering before the next batch becomes active.
   - [ ] Exhausting or inspecting the current batch does not fetch or prepare 
the next backend page; tests assert backend/page fetch count, not only consumed 
element count.
   - [ ] Existing #3140 range-index ordering, limit, offset, paging, and cursor 
behavior remains unchanged.
   - [ ] The vertex and edge regressions from #3180/#3182 pass on the 
applicable memory, RocksDB, and HStore test profiles.
   - [ ] Warm-cache PRIMARY_KEY queries, PRIMARY_KEY queries with residual 
conditions, and vertex/edge INDEX + LABEL queries remain correct.
   - [ ] Close and exception propagation still release the active and remaining 
iterator/page resources exactly once.
   
   ## References
   
   - #3180
   - #3140
   - #3182
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to