bosprimigenious opened a new issue, #905: URL: https://github.com/apache/hugegraph-toolchain/issues/905
## Search before asking - [x] Searched existing features, issues and PRs. #748 is the parent roadmap and #772 is related work with its own contributor. No dedicated raw-line child or Rust implementation was found in the checked public results; this is a scoped child proposal, not a claim that no related feature exists. ## Feature Description AI-generated proposal prepared with OpenAI Codex. No human review, maintainer approval, project assignment or OSPP selection is claimed. Parent: #748. Related coordination: [comment](https://github.com/apache/hugegraph-toolchain/issues/748#issuecomment-6764354939). This child records a **design proposal**, not an implementation or a completed Loader. The parent requires each child to define its plan before implementation; opening this proposal does not presume approval. The decisions below remain proposed/pending. ## 1. Scope, proposed owner/mentor and non-goals Proposed owner: **bosprimigenious**, volunteering only for this limited slice; no assignment is requested or asserted here. Mentor/reviewer: **unconfirmed**, awaiting community guidance. Propose an offline, single-local-file, uncompressed raw-line input primitive for a future Rust Loader. It would expose original line content and physical line positions, independently of parsing or graph transport. Non-goals: CSV/JSON parsing, skip/header policies, property mapping, HTTP, authentication, graph writes, concurrency, checkpoint/resume and replacing Java Loader defaults. This bottom-layer helper is not delivery of a complete Rust Loader. Explicitly exclude SebastianGruza's #772 scope: REST fixtures/recorder, Java/Go runners and parity matrix, Loader benchmark datasets/scripts/results, and the conditional Rust transport + `/versions` demonstration. No competing fixture format or benchmark project is proposed. ## 2. Compatibility and proposed API/error contract Reference: Toolchain `47c5fc007b0e0418dec99061e9bf0787fdab191b`. [FileLineFetcher.fetch()](https://github.com/apache/hugegraph-toolchain/blob/47c5fc007b0e0418dec99061e9bf0787fdab191b/hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/file/FileLineFetcher.java#L159) reads a physical line, increments offset, then applies skip/header and parsing. Only the first two operations are proposed. All API names and these semantics need review: - An owned, read-only File; a positive caller-specified `max_record_bytes`; synchronous pull returning an owned record, error or EOF. Reject zero before content reads. Workspace/layout, edition and MSRV remain undecided. - Cap counts raw UTF-8 **payload bytes**, including spaces and any BOM, excluding LF/CR/CRLF terminators. Empty records remain valid. With cap=1, `a\r\n` fits; `é` exceeds it. A preserved BOM counts three bytes. - Preserve LF, CRLF and bare-CR framing, original content and 1-based physical positions. Empty input yields no records; a single terminator yields one empty record; `a\n\n` yields `(1,"a"),(2,"")`; a final terminator creates no additional record. EOF does not increment position. Positions are not byte offsets or committed progress. - Preserve BOM at the beginning or inside content; no trimming, normalization or comment filtering. - A completed, valid record is returned before later read failures. In particular, a CR completes the record; optional following LF can be consumed on the next pull. An unfinished/invalid/oversized record is never returned as success. - Surface one error, then terminate: subsequent pulls return EOF, without implicit reopen/retry. Proposed precedence is byte framing/cap first, I/O failure before completion second, and strict UTF-8 validation of complete, within-cap payload last. Thus an oversized malformed record reports TooLong; a malformed unfinished record that encounters an I/O failure before completion reports I/O. - Errors identify the attempted physical line. InvalidUtf8 reports a zero-based payload-byte index at the invalid sequence start, including preceding BOM bytes, and distinguishes an incomplete final sequence. TooLong identifies the first excess byte. I/O context counts bytes successfully read into the attempted record, not a precise OS failure offset. Counters fail on overflow. - Release owned file resources on EOF, error or early drop. Returned strings outlive the reader. An internal generic Read test seam would not imply closing a caller's borrowed handle. Blocking reads have no cancellation/timeout guarantee. Compatibility is explicitly limited to **well-formed UTF-8, within-cap input**. Java supports configurable charset, replacement decoding and no record-length cap. Strict rejection and a mandatory cap are proposed experimental boundaries, not transparent default compatibility. These decisions must be settled before implementation. ## 3. Differential, fault and lifecycle validation Compare raw content and physical positions with the pinned Java reference/explicit raw-read adapter for the agreed subset. Do not substitute manually authored expected files for actual Java execution. Cover empty/final unterminated records, consecutive terminators, mixed LF/CRLF/CR, BOM, multibyte UTF-8, exact/over-cap records and line-counter overflow. Vary underlying chunks and buffers, including one-byte chunks, to split CRLF and UTF-8 sequences at every boundary. Inject errors before the first record, after a complete CR-terminated record and mid-record. Assert emitted-prefix order, error location/precedence, terminal behavior and owned-resource release. No concurrent/shared-cursor guarantee is proposed; synchronous pull provides backpressure without a queue. Upgrade/resume formats remain untouched. No Rust implementation currently exists, so these are acceptance requirements, not passing tests. ## 4. Reproducible performance/resource protocol Propose reader-only bytes/s, first-record latency and peak RSS on agreed existing local inputs, recording input SHA, size, configuration, hardware, toolchains, commands and raw outputs; separate cold/warm-cache observations. Coordinate inputs/results with #772, without creating its dataset generator or publishing another Java Loader baseline. Initial correctness work makes no speedup claim. Proposed internal memory bound is O(B+L), for fixed input buffer B and record cap L. Enumerate accumulation, decoding copies and prefetch; declare measurable allocation/read-ahead bounds rather than claiming exact B+L RSS. Counting-Read checks cannot replace OS memory measurements. Caller-retained output collections are outside the internal-buffer bound. ## 5. Security, dependencies and platforms Proposed CPU targets: Linux x86_64 and macOS arm64, pending confirmation; Windows/other targets remain undecided. Existing local Rust 1.98.1 is not a proposed MSRV. Prefer standard-library-only implementation initially; dependency, workspace and lockfile choices require review. Require rustfmt, Clippy with warnings denied, tests, applicable dependency audit, complete Apache headers and license/NOTICE checks. No unsafe code or publishing credentials are needed. Changes introducing dependencies require license/provenance review and packaging validation. Follow ASF generative-tooling guidance and disclose generated contributions without inventing human approval. ## 6. Observability, packaging, coexistence and rollback Proposed diagnostics: file identifier, attempted physical line, bounded position/cap metadata and error category, without dumping record contents. No background telemetry or service dependency. Keep the experimental component opt-in, read-only and separate from existing Java commands. No configuration, protocol, storage or progress-format migration is introduced. Rollback means stopping use of the component; no graph data was written. Do not package/publish crates or release binaries until layout, platform and distribution conventions are agreed. Any future graph-write/resume integration needs its own compatibility and recovery plan. ## 7. Acceptance, documentation and current evidence Before coding, resolve scope/mentor, workspace/MSRV/platforms, ownership, cap/encoding/error semantics and acceptance boundaries in this child. No fake assignment or fixed deadline is proposed. An implementation PR should include differential results, fault/resource controls, lint/security checks, supported-platform results and documentation: API usage, exact supported subset, errors, cap, resource behavior, reproducible verification/measurement commands and opt-in rollback. Unsupported scenarios remain explicit. CI/review failures remain visible; build success alone is not full acceptance. Current preparation evidence, as of 2026-11-10: - The pinned SDK build from `apache/hugegraph@86c05ef9c5d417bd37eab21d4ecd5af82bab17ef` completed successfully. Its manifest covers **14 required SDK modules**; **35 artifact files were rehashed with zero mismatches, including 15 JARs**. SDK tests were skipped by the official installer: this is build/provenance evidence. - An independently rerun JDK 17 oracle checked **72 framing/decoding/fault observations**, including bare CR, empty records, BOM and partial-read failures. These are JDK contract checks, **not Rust product tests or HugeGraph end-to-end validation**. Rust's standard `BufRead.lines()` probe differs on bare CR. - Toolchain Client/Loader reference installation is still running locally; **Loader tests have not run**. No Rust raw-line implementation, benchmark, full parity or production-readiness result is claimed. This issue requests concrete design feedback and a mentor/reviewer for a bounded contribution. It does not claim ownership of the whole roadmap or an OSPP place. ## Contribution intent and conduct - [x] Willing to submit a scoped PR after the design decisions are settled. - [x] Contributions in this issue will follow the [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
