sigram commented on code in PR #4910:
URL: https://github.com/apache/solr/pull/4910#discussion_r4035079774
##########
solr/core/src/java/org/apache/solr/handler/admin/api/UpdateAPI.java:
##########
@@ -65,4 +76,33 @@ public void updateJson(SolrQueryRequest req,
SolrQueryResponse rsp) throws Excep
public void updateJavabin(SolrQueryRequest req, SolrQueryResponse rsp)
throws Exception {
updateRequestHandler.handleRequest(req, rsp);
}
+
+ /**
+ * Whether this request should be routed to the NDJSON loader. The rewritten
path applies to every
+ * content stream of the request, so a request is only treated as NDJSON
when all of its streams
+ * are; anything else keeps the historic {@code /update/json/docs} behavior.
+ */
+ private static boolean isNdJson(SolrQueryRequest req) {
+ String assumed = req.getParams().get(UpdateParams.ASSUME_CONTENT_TYPE);
+ if (assumed != null) {
+ return isNdJsonContentType(assumed);
+ }
+ Iterable<ContentStream> streams = req.getContentStreams();
+ if (streams == null) {
+ return false;
+ }
+ boolean any = false;
+ for (ContentStream stream : streams) {
+ if (!isNdJsonContentType(stream.getContentType())) {
+ return false;
+ }
+ any = true;
Review Comment:
Why do we need `any` at all? the loop immediately returns false when any
stream is not NDJSON, so after the loop we can simply return true.
##########
solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-update-handlers.adoc:
##########
@@ -478,6 +479,44 @@ You can also specify `\_version_` with each "delete":
You can specify the version of deletes in the body of the update request as
well.
+=== Newline Delimited JSON
+
+Solr also accepts https://ndjson.org/[Newline Delimited JSON] (ND-JSON), also
known as JSON Lines or JSONL: one document per line, as a JSON Object, with no
enclosing array and no commas between lines.
+
+Send it to the `/update` handler with `Content-Type: application/x-ndjson`, or
to the dedicated `/update/ndjson` path:
+
+[source,bash]
+----
+curl -X POST -H 'Content-Type: application/x-ndjson'
'http://localhost:8983/solr/my_collection/update?commit=true' --data-binary '
+{"id":"1","title":"Doc 1"}
+{"id":"2","title":"Doc 2"}
+{"id":"3","title":"Doc 3"}
+'
+----
+
+`application/jsonl` and `application/x-jsonlines` are accepted as aliases, and
the xref:post-tool.adoc[] sends `.jsonl` and `.ndjson` files automatically.
Review Comment:
jsonlines.org site lists another content type used by some vendors:
`application/jsonlines` (without `x-`, see
https://github.com/wardi/jsonlines/issues/19).
Also, since the "spec", such as it is, is specified somewhat informally I
think we should consistently point the docs to a single reference
documentation, let's say http://ndjson.com and not a mix of this and
http://jsonlines.org
##########
solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-update-handlers.adoc:
##########
@@ -478,6 +479,44 @@ You can also specify `\_version_` with each "delete":
You can specify the version of deletes in the body of the update request as
well.
+=== Newline Delimited JSON
+
+Solr also accepts https://ndjson.org/[Newline Delimited JSON] (ND-JSON), also
known as JSON Lines or JSONL: one document per line, as a JSON Object, with no
enclosing array and no commas between lines.
+
+Send it to the `/update` handler with `Content-Type: application/x-ndjson`, or
to the dedicated `/update/ndjson` path:
+
+[source,bash]
+----
+curl -X POST -H 'Content-Type: application/x-ndjson'
'http://localhost:8983/solr/my_collection/update?commit=true' --data-binary '
+{"id":"1","title":"Doc 1"}
+{"id":"2","title":"Doc 2"}
+{"id":"3","title":"Doc 3"}
+'
+----
+
+`application/jsonl` and `application/x-jsonlines` are accepted as aliases, and
the xref:post-tool.adoc[] sends `.jsonl` and `.ndjson` files automatically.
+The format is UTF-8 only: a request declaring any other `charset` is rejected
with a `415 Unsupported Media Type`.
+
+Solr parses and indexes the request as it streams in, so peak memory is
bounded by the largest single document rather than by the size of the request,
which makes the format a good fit for large bulk loads.
+Blank lines are ignored.
+A line that is not a single JSON Object -- an enclosing array, a document
spread over several lines, two documents on one line, or anything that is not
an Object -- fails the request, reporting the line number.
+Documents from earlier lines have already been submitted when a later line
fails, so a failed load may be partially applied.
+
+NDJSON carries documents only; the `add`, `delete`, `commit`, `optimize`, and
`rollback` commands are not recognized, so use request parameters such as
`commit` and `commitWithin` instead.
Review Comment:
Maybe rephrase that the NDJSON request implies "add" only, with optional
commit / commitWithin?
##########
solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-update-handlers.adoc:
##########
@@ -478,6 +479,44 @@ You can also specify `\_version_` with each "delete":
You can specify the version of deletes in the body of the update request as
well.
+=== Newline Delimited JSON
+
+Solr also accepts https://ndjson.org/[Newline Delimited JSON] (ND-JSON), also
known as JSON Lines or JSONL: one document per line, as a JSON Object, with no
enclosing array and no commas between lines.
+
+Send it to the `/update` handler with `Content-Type: application/x-ndjson`, or
to the dedicated `/update/ndjson` path:
+
+[source,bash]
+----
+curl -X POST -H 'Content-Type: application/x-ndjson'
'http://localhost:8983/solr/my_collection/update?commit=true' --data-binary '
+{"id":"1","title":"Doc 1"}
+{"id":"2","title":"Doc 2"}
+{"id":"3","title":"Doc 3"}
+'
+----
+
+`application/jsonl` and `application/x-jsonlines` are accepted as aliases, and
the xref:post-tool.adoc[] sends `.jsonl` and `.ndjson` files automatically.
+The format is UTF-8 only: a request declaring any other `charset` is rejected
with a `415 Unsupported Media Type`.
+
+Solr parses and indexes the request as it streams in, so peak memory is
bounded by the largest single document rather than by the size of the request,
which makes the format a good fit for large bulk loads.
+Blank lines are ignored.
Review Comment:
Comments (// or # style) are not allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
