imbajin commented on issue #3057: URL: https://github.com/apache/hugegraph/issues/3057#issuecomment-4658783145
Thanks for the clarification. I think this case should not be solved by increasing `task.result_size_limit`. HugeGraph already has pagination/cursor-like APIs for some data access paths, for example: - [Vertex API pagination](https://hugegraph.apache.org/docs/clients/restful-api/vertex/#method--url-6) - Traverser shard scan APIs: - `GET /graphspaces/{graphspace}/graphs/{graph}/traversers/vertices/shards?split_size=...` - `GET /graphspaces/{graphspace}/graphs/{graph}/traversers/vertices/scan?start=...&end=...&page=...&page_limit=...` - similar APIs also exist for edges For a large full-data export, the recommended direction should be based on this paged/sharded scan model: ```text get shards -> scan each shard page by page -> write results into chunk files -> task stores only metadata/progress/manifest ``` The task result itself should remain small. It should store metadata such as `status`, `progress`, `error`, and optionally an output manifest or file URI, not a 1GB/2GB+ JSON payload. However, this does not mean every API can support the same `page` semantics. The existing `page` token works well for storage-level scans or some simple list queries, but arbitrary Gremlin/traversal results may involve traversal state, aggregation, ordering, deduplication, or path expansion. Those results cannot always be resumed by a backend scan page token. So I think we should distinguish the cases: | Case | Suggested handling | |---|---| | Full vertex/edge export | Use shard + scan + `page`/`page_limit` | | Simple list/query APIs | Support `page`/`limit` where the result order and cursor are well-defined | | Arbitrary Gremlin large result | Do not return it through `/tasks/{id}` as one huge result | | Reusable 2GB+ query output | Needs a streaming/chunked export job or an API-specific cursor/result-set model | In short, HugeGraph does have paged scan APIs, and they are the right foundation for bulk export. But `/tasks/{id}` should not be treated as a bulk result download API. Large results should be paged, chunked, or streamed, while the task only tracks the export metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
