contrueCT opened a new issue, #3148:
URL: https://github.com/apache/hugegraph/issues/3148

   ### Feature Description
   
   Large asynchronous Gremlin results are persisted as a compressed task-result 
BLOB. The existing task-details endpoint loads the result into the task object 
and returns it through `HugeTask.asMap()`. Even after metadata-only access was 
added in #3060, a client that actually needs the result still has to request 
the complete decompressed value. This can create substantial heap pressure and 
long response times for large persisted results, as seen in #3057 and #3059.
   
   Add a dedicated, read-only endpoint for consuming a persisted task result 
without materializing a complete decompressed JSON string or Java object tree:
   
   ```http
   GET /graphspaces/{graphspace}/graphs/{graph}/tasks/{id}/result
   ```
   
   The existing `GET .../tasks/{id}` API and its `with_result` behavior must 
remain compatible.
   
   ### Current and proposed read paths
   
   ```mermaid
   flowchart LR
       Details["GET .../tasks/{id}\nwith_result=true"] --> Task["Load task and 
full result"]
       Task --> Map["HugeTask.asMap()"]
       Map --> Existing["Task-details response"]
   
       Result["GET .../tasks/{id}/result"] --> Snapshot["Detached compressed 
snapshot"]
       Snapshot --> Stream["LZ4 stream + JSON token parser"]
       Stream --> NewResponse["Raw JSON or logical page"]
   ```
   
   For local schedulers, the snapshot comes from the task vertex result 
property. For distributed schedulers, it comes from the separate 
`HugeTaskResult` / `~taskresult` vertex introduced by the current 
result-storage model. The snapshot must be detached before the HTTP callback 
runs so no graph transaction, vertex iterator, or scheduler thread context is 
retained by a slow client.
   
   ### API behavior
   
   | Request | Response |
   | --- | --- |
   | `GET .../tasks/{id}/result` | Streams the original persisted JSON value. |
   | `GET .../tasks/{id}/result?limit=N` | Returns the first logical page. |
   | `GET .../tasks/{id}/result?page=<token>` | Returns the next logical page 
using an opaque continuation token. |
   
   A paged response uses HugeGraph's existing `limit` / `page` convention:
   
   ```json
   {
     "root_type": "array",
     "items": [1, 2],
     "page": "<opaque token or null>"
   }
   ```
   
   Top-level arrays are paged by element and top-level objects by member. 
Object members are represented as `key` / `value` items so duplicate JSON keys 
are not collapsed. Scalars can be streamed in full but are not pageable. 
`limit` and `page` are mutually exclusive, and the terminal page returns 
`"page": null`.
   
   The continuation token must be signed and bind the graphspace, graph, task 
id, result fingerprint, root type, limit, next offset, and expiry. Predictable 
validation failures must be detected before HTTP 200 is committed. The endpoint 
should reuse HugeGraph's standard gzip compression and `exception` / `message` 
/ `cause` error envelope.
   
   ### Resource and consistency requirements
   
   - Only a successfully persisted task result is readable through the new 
endpoint.
   - A changed result must invalidate an older page token instead of continuing 
over a different snapshot.
   - Logical pagination over one BLOB may rescan from the beginning, so page 
offset, decompressed scan bytes, scan time, active streams, stream duration, 
and token size/lifetime need explicit limits.
   - Client disconnects and slow-reader timeouts must close the 
parser/decompression stream, restore connection timeout state, and release the 
active-stream permit.
   - Multi-node deployments need a shared page-token secret, with 
current/previous key support for bounded rotation.
   - The scheduler SPI extension must remain compatible with custom schedulers 
that do not implement result streaming.
   
   ### Scope and non-goals
   
   This feature changes only the read path. It does not change task-result 
serialization, storage schema, write limits, or the existing task-details 
response. The current backend API still returns the compressed BLOB eagerly, so 
the feature removes the complete decompressed result and object-tree 
materialization but does not eliminate the in-memory compressed `byte[]`.
   
   Physical chunk storage and random-access page lookup remain follow-up work 
under #3071. Streaming Gremlin execution and write-time chunk publication also 
require a separate design because they change transaction, failure, 
cancellation, and size-limit semantics.
   
   ### Acceptance criteria
   
   - [ ] The existing task-details API remains backward compatible.
   - [ ] The new endpoint reads both local task-vertex results and distributed 
`HugeTaskResult` results.
   - [ ] Full-result retrieval does not create a complete decompressed result 
`String` or Java object tree.
   - [ ] Array/object pagination follows the `limit` / `page` contract and 
rejects scalar pagination.
   - [ ] Page tokens are signed, route/result-bound, expiring, 
tamper-resistant, and rotation-aware.
   - [ ] Predictable errors are returned before response commit; post-commit 
failures terminate the stream and remain observable.
   - [ ] Gzip transport, resource budgets, client disconnects, slow readers, 
metrics, and permit cleanup are covered by tests.
   - [ ] Memory profiling documents the remaining compressed-BLOB allocation 
and verifies that the full decompressed result is absent.
   
   ### Related work
   
   - #3057 and #3059 report task APIs becoming unavailable or timing out after 
very large task results.
   - #3060 added metadata-only task access and separated metadata/result reads; 
this feature builds on that boundary.
   - #3071 tracks physical chunked storage and larger-result follow-up work; 
this issue deliberately does not close or replace it.
   - Implementation: #3144.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to