yzeng1618 opened a new pull request, #11740:
URL: https://github.com/apache/seatunnel/pull/11740
## Purpose
#10990 introduced the Knowledge Sync logical metadata registry and allowed
`MetadataTransform` to project declared metadata from `SeaTunnelRow.options`.
The Markdown source still emitted only its physical RAG columns, so a real
Markdown pipeline could not use the common metadata contract without
synthetic
catalog tables and row options.
This PR adds a connector-local bridge from Markdown RAG output to Knowledge
Sync logical metadata while preserving all existing physical fields and
routing
formulas.
## Changes
- Declare `SourceUri`, `DocumentId`, `DocumentHash`, and `ChunkHash` in the
Markdown source metadata schema when `markdown_rag_metadata_enabled=true`.
- Populate the four logical keys in every emitted Markdown row's independent
`SeaTunnelRow.options` map without clearing unrelated options.
- Build a credential-free logical source identity:
- preserve the existing local-path normalization for local paths and valid
`file:` URIs;
- preserve the scheme, host, explicit port, and path of hierarchical remote
URIs;
- remove user info, the complete query component, and the fragment;
- fail unsafe URI sanitization without echoing the original sensitive
value.
- Build logical `DocumentId` as
`doc_ + lower_hex(SHA-256(UTF-8(logical SourceUri)))`.
- Calculate `DocumentHash` from the exact source bytes during the existing
stream read, before UTF-8 decoding and Markdown parsing, without a second
source read.
- Reuse the existing immediate-row text hash as logical `ChunkHash`.
- Merge registry-defined metadata columns deterministically while preserving
existing metadata, reusing compatible definitions, and rejecting
incompatible same-name definitions.
- Cover bounded existing-file discovery, empty-path fallback, continuous
discovery, and multi-table catalog construction.
- Add real Markdown catalog/row coverage to `MetadataTransform`, including
successful alias projection and existing physical-name collision rejection.
- Extend the LocalFile Markdown E2E assertion configuration with logical
metadata projection.
- Update the English and Chinese File Source and Metadata Transform
documentation.
## Compatibility
- No configuration option, dependency, physical column, or public API is
added
or renamed.
- The behavior remains gated by the existing disabled-by-default
`markdown_rag_metadata_enabled` option.
- Existing physical `source_uri`, `document_id`, `chunk_id`, `chunk_index`,
and
`content_hash` fields retain their names, order, values, formulas, and
source
routing behavior.
- Non-Markdown formats and disabled Markdown RAG configurations retain their
current metadata-schema behavior.
- Signed or credential-bearing URLs can intentionally produce different
physical and logical document IDs: the physical ID remains the
compatibility
value, while the logical ID excludes credentials and transient URI
components.
## Semantics and limitations
- The generic bridge removes the complete remote URI query. Resources whose
identity exists only in query parameters need a stable, non-sensitive path
or a future source-specific identity mechanism.
- Logical `ChunkHash` identifies the immediate Markdown output row only. A
downstream transform that changes or splits text must recompute final
`ChunkHash`, `ChunkId`, and `ChunkIndex` before a lifecycle sink consumes
the
rows.
- This bridge does not expose logical `ChunkId` or `ChunkIndex` from the
legacy
physical formulas.
- Incremental comparison, final-writer affinity, stale-chunk deletion,
tombstones, and lifecycle sink behavior are outside this PR.
## Verification
Passed after rebasing onto the latest `apache/seatunnel:dev`:
- Connector-file Markdown reader and catalog tests: 22 tests, 0 failures.
- `MetadataTransformTest`: 5 tests, 0 failures.
- Spotless checks executed by both focused Maven test commands.
- `git diff upstream/dev --check`.
The full root `./mvnw -q -DskipTests verify` reactor was attempted locally
but
did not complete within the available 30-minute execution window. One earlier
interrupted online run also encountered a transient local Maven resolver
`Malformed \\uxxxx encoding` error. The full root test suite and LocalFile
E2E
suite were therefore not run locally and should be covered by CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
