adityamparikh opened a new issue, #208:
URL: https://github.com/apache/solr-mcp/issues/208
## Summary
Add an `index-url` MCP tool that streams a JSON, CSV, XML or Markdown
document set from an http(s) URL into a Solr collection, so the payload never
passes through the model's context. It is registered in **both** the STDIO and
HTTP transports. This issue records the design and the decisions behind it
before any code is written; the full implementation spec (tool contract, fetch
algorithm, error messages, test list, threat-model edits) will accompany the PR.
## Problem
All four indexing tools (`index-json-documents`, `index-csv-documents`,
`index-xml-documents`, `index-markdown-documents`) take the payload as a
tool-call argument, so the model has to emit every byte.
Measured when #197 was closed: indexing the 61-record `shows.json` sample
through a live server took over two minutes. Solr and the server accounted for
under one second. The rest was the model emitting ~9,500 tokens of escaped
JSON. Anything larger than one response's output budget has to be split across
calls, and each call commits, so a large file lands as a partial, multi-commit
ingest with no transactional boundary.
A URL argument is ~20 tokens regardless of payload size.
## Why a URL and not a file upload
MCP has no client-to-server file transfer. SEP-2356 (declarative file
inputs) was closed in favour of SEP-2631 ("File Objects and Transfer"), which
is still an open draft. A file attached in Claude Desktop is extracted into the
model's context; the model gets content and a filename, never a path or handle
it can pass to a server. A URL is the only non-inline source both transports
can reach identically today.
#194 added a STDIO-only `index-file` tool and was closed because it "would
give the two transports different tool surfaces for the same job". `index-url`
gives HTTP a non-inline bulk path, which is what that objection asked for.
## Proposed design
One new `@Service`, `UrlIndexingService`, with no `@Profile` gate:
```
index-url(collection: String, url: String, format: String?) -> summary string
```
It fetches the URL with the JDK `HttpClient`, wraps the body in a `Reader`,
and hands it to the same streaming spine that `index-file` uses
(`IndexingDocumentCreator.stream(Reader, format, consumer)`, batches of 1,000,
commit at the end). The parsers never learn where the bytes came from, so this
is a second adapter over already-tested code, not a new pipeline.
```
INLINE index-{json,csv,xml,markdown}-documents(collection, payload) -->
List<SolrInputDocument>
|
URL index-url(collection, url, format?) HttpClient ->
InputStreamReader ---+--> stream(Reader) --> batches --> Solr
|
FILE index-file(collection, path, format?) Files.newBufferedReader
-----------+ (STDIO only)
```
Fetch behaviour:
- **Format resolution:** explicit `format` > URL path extension (query
string stripped) > `Content-Type` > error. Extension outranks the header
because `raw.githubusercontent.com` serves `.json` as `text/plain`.
`text/plain` and `text/html` map to nothing; the error suggests `format=`.
- **Non-2xx is a hard failure before parsing.** A GitHub 404 body is the
text `404: Not Found`; JSON would reject it but CSV would index it as a
document.
- **No credentials and no caller-supplied headers, ever.** The server cannot
become a credential relay.
- **Redirects** followed manually, at most 5 hops, never `https` to `http`.
- **Timeouts:** connect 10s, response headers 60s, plus a 30s read-idle
watchdog and an optional byte cap (default unlimited). The watchdog exists
because an endpoint that never closes the stream would otherwise keep the
server committing batches indefinitely under the remote party's control, which
has no filesystem analogue.
- **Charset** from `Content-Type`, else UTF-8. No gzip. No `file:` or `ftp:`.
## Decisions and why
1. **Any http(s) URL the server can reach is allowed; no operator
allow-list.** Mirrors the posture `index-file` documents ("OS permissions and
container mounts define the boundary"). The URL analogue is "the network the
server process can reach". An allow-list would make the common case (a public
raw-GitHub or S3 URL) need operator setup. This means the tool is a deliberate,
documented SSRF surface in HTTP mode, and the threat model is updated to say so
(below).
2. **Link-local and cloud-metadata addresses (`169.254.0.0/16`, `fe80::/10`,
`fd00:ec2::254`) are refused.** Not an allow-list, needs no configuration,
removes the single worst outcome (a cloud instance's IAM credentials) without
touching any legitimate use. Loopback and RFC1918 are deliberately not refused.
*Open for maintainer input.*
3. **`index-file` stays, STDIO-only.** This knowingly keeps the bean shape
#194 was closed for. Reason: the flagship deployment is Claude Desktop + STDIO
with the file on the same machine, and without `index-file` that deployment's
most natural gesture takes the worst path while the file sits readable by the
server process. With `index-url` present, `index-file` becomes a zero-network
shortcut rather than a capability HTTP lacks. The out-of-band upload endpoint
#194 mentioned is deferred, not rejected. *Open for maintainer input; the
fallback is to ship `index-url` alone.*
4. **One tool with optional `format`, not four per-format tools.** #197 kept
per-format inline tools because each signature is optimised for its format's
token shape. `index-url` carries no payload, so there is no shape to optimise.
5. **No schema-from-URL.** Schema payloads are a few KB and the model should
read and reconcile them against `get-schema` anyway.
6. **`text/html` is an error, not converted.** "Index this web page" is a
different feature.
## What HTTP mode still cannot do
A file selected in Claude Desktop while connected to an HTTP-mode server has
no bulk path. `index-file` is not registered there, and `index-url` resolves
the URL from the **server's** network position, so
`http://localhost:8000/file.json` served on the caller's machine hits the
server's own loopback instead. Under decision 1 that does not fail cleanly:
whatever the server is running on that port answers. This is structural until
SEP-2631 lands; the tutorial will say "use the STDIO transport for local
files", and the tool description says the URL is fetched from the server's side.
## Threat model impact
`THREAT_MODEL.md` §8 property 5 currently says the AI client "cannot repoint
the server or inject a target URL", with violation at critical severity, and
§12 lists "backend-target from a tool argument" as a model-changing condition.
This tool exercises that condition on purpose. The PR narrows §8.5 to the
*Solr* backend and its credentials, adds the outbound fetch as a §9 disclaimed
property, records the decision in §12 with the date, adds the `index-url`
non-finding to §11a and the "internal services reachable from the HTTP
deployment" misuse to §11, qualifies the §13 `VALID` row, adds the two
body-limit knobs to §5a, and updates `docs/security/stdio.md` and `http.md`,
which §8.5 cites. Without the §8.5 edit the next scanner files this as a
critical finding regardless of the rest.
## Tests
- `UrlIndexingIntegrationTest`: a JDK `com.sun.net.httpserver.HttpServer` on
an ephemeral port (no new dependencies, runs under `nativeTest`) serving all
four formats, a `.csv?token=x` URL, a media-type-only URL, `text/plain` and
`text/html` bodies, a 404, a redirect, a redirect loop, a generated multi-MB
CSV to prove the body streams, and a stalling handler to prove the watchdog;
real Solr via Testcontainers.
- `McpClientIntegrationTest` (HTTP) becomes the transport-parity guard:
`index-url` present, `index-file` absent, one round trip.
`McpClientStdioIntegrationTest`: both present, one round trip.
- Unit tests for the target policy, the idle watchdog, and the error mapping.
- Zero skipped tests in `./gradlew build` and `./gradlew nativeTest
-Pnative`. A native spike (`HttpClient` over https in the native binary) runs
before any test is written.
## Delivery
Two PRs, in order: first the unrelated `required`-explicit refactor already
on the branch, then the streaming spine + `index-file` + `index-url` + all doc
and threat-model edits together (the spine is ~1,100 lines and is not
reviewable without a consumer). Independent of #196, #202, #203, #205 and #207.
## Related
- #194 (closed): STDIO-only `index-file`, the objection this design answers.
- #197 (closed): the payload-cost measurement and the per-format-tools
decision.
- #88 (closed): earlier file-upload attempt.
- SEP-2631: the MCP file-transfer draft that would close the HTTP local-file
gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
