I've created a ticket for this: SOLR-12798
It seems to me that not only is the loss of multipart form posting a serious matter, but also the fact that you cannot use a multipart form post for any UpdateRequest. Both are described in the ticket. This is a major problem for the ManifoldCF release cycle happening now. If there is something I've missed and there is a good way to accomplish this in SolrJ, please let me know. I also tried to tag Paul Nobel in the ticket but Jira does not recognize that name. Karl On Mon, Sep 24, 2018 at 4:01 AM Karl Wright <[email protected]> wrote: > Ok, the commit that broke this was made on 10-31-2107 by Paul Nobel: > > >>>>>> > f6f6f113209 (Noble Paul 2017-06-16 14:05:51 +0930 327) > protected HttpRequestBase createMethod(SolrRequest request, String > collection) throws IOException, SolrServerException { > f6f6f113209 (Noble Paul 2017-06-16 14:05:51 +0930 328) if > (request instanceof V2RequestSupport) { > f6f6f113209 (Noble Paul 2017-06-16 14:05:51 +0930 329) > request = ((V2RequestSupport) request).getV2Request(); > f6f6f113209 (Noble Paul 2017-06-16 14:05:51 +0930 330) } > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 331) > SolrParams params = request.getParams(); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 332) > RequestWriter.ContentWriter contentWriter = > requestWriter.getContentWriter(request); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 333) > Collection<ContentStream> streams = contentWriter == null ? > requestWriter.getContentStreams(request) : null; > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 334) > String path = requestWriter.getPath(request); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 335) if > (path == null || !path.startsWith("/")) { > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 336) > path = DEFAULT_PATH; > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 337) } > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 338) > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 339) > ResponseParser parser = request.getResponseParser(); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 340) if > (parser == null) { > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 341) > parser = this.parser; > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 342) } > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 343) > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 344) // > The parser 'wt=' and 'version=' params are used instead of the original > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 345) // > params > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 346) > ModifiableSolrParams wparams = new ModifiableSolrParams(params); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 347) if > (parser != null) { > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 348) > wparams.set(CommonParams.WT, parser.getWriterType()); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 349) > wparams.set(CommonParams.VERSION, parser.getVersion()); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 350) } > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 351) if > (invariantParams != null) { > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 352) > wparams.add(invariantParams); > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 353) } > 68d180346ab (Alan Woodward 2015-03-09 11:54:45 +0000 354) > 68d180346ab (Alan Woodward 2015-03-09 11:54:45 +0000 355) > String basePath = baseUrl; > 68d180346ab (Alan Woodward 2015-03-09 11:54:45 +0000 356) if > (collection != null) > 68d180346ab (Alan Woodward 2015-03-09 11:54:45 +0000 357) > basePath += "/" + collection; > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 358) > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 359) if > (request instanceof V2Request) { > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 360) if > (System.getProperty("solr.v2RealPath") == null) { > c8e0e939e49 (Ishan Chattopadhyaya 2017-08-20 21:00:15 +0530 361) > basePath = baseUrl.replace("/solr", "/api"); > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 362) } > else { > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 363) > basePath = baseUrl + "/____v2"; > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 364) } > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 365) } > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 366) > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 367) if > (SolrRequest.METHOD.GET == request.getMethod()) { > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 368) if > (streams != null || contentWriter != null) { > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 369) > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "GET can't > send streams!"); > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 370) } > ce172acb8fe (markrmiller 2016-04-01 12:21:59 -0400 371) > f4fe17cce85 (Uwe Schindler 2015-11-22 09:56:22 +0000 372) > return new HttpGet(basePath + path + wparams.toQueryString()); > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 373) } > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 374) > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 375) if > (SolrRequest.METHOD.DELETE == request.getMethod()) { > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 376) > return new HttpDelete(basePath + path + wparams.toQueryString()); > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 377) } > cc8b5bab0bd (Cao Manh Dat 2017-05-11 09:06:03 +0700 378) > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 379) if > (SolrRequest.METHOD.POST == request.getMethod() || SolrRequest.METHOD.PUT > == request.getMethod()) { > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 380) > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 381) > String url = basePath + path; > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 382) > boolean hasNullStreamName = false; > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 383) if > (streams != null) { > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 384) > for (ContentStream cs : streams) { > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 385) > if (cs.getName() == null) { > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 386) > hasNullStreamName = true; > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 387) > break; > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 388) > } > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 389) } > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 390) } > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 391) > boolean isMultipart = ((this.useMultiPartPost && SolrRequest.METHOD.POST > == request.getMethod()) > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 392) > || (streams != null && streams.size() > 1)) && !hasNullStreamName; > ef809a0f10e (Alan Woodward 2014-12-31 14:05:48 +0000 393) > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 394) > LinkedList<NameValuePair> postOrPutParams = new LinkedList<>(); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 395) > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 396) > if(contentWriter != null) { > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 397) > String fullQueryUrl = url + wparams.toQueryString(); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 398) > HttpEntityEnclosingRequestBase postOrPut = SolrRequest.METHOD.POST == > request.getMethod() ? > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 399) > new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 400) > postOrPut.addHeader("Content-Type", > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 401) > contentWriter.getContentType()); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 402) > postOrPut.setEntity(new BasicHttpEntity(){ > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 403) > @Override > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 404) > public boolean isStreaming() { > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 405) > return true; > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 406) > } > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 407) > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 408) > @Override > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 409) > public void writeTo(OutputStream outstream) throws IOException { > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 410) > contentWriter.write(outstream); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 411) > } > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 412) > }); > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 413) > return postOrPut; > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 414) > 706b6c91718 (Noble Paul 2017-10-31 13:16:31 +1030 415) } > else if (streams == null || isMultipart) { > 203bbf504a3 (Alan Woodward 2015-03-26 10:42:22 +0000 416) > // send server list and request list as query string params > ... > <<<<<< > > The introduction of RequestWriter essentially prevented use of streams and > multipart EXCEPT when there was no RequestWriter in a request, which is > never. > > Karl > > > On Sat, Sep 22, 2018 at 12:19 PM Karl Wright <[email protected]> wrote: > >> I'll dig to find which commit it was that introduced the changes. >> Karl >> >> On Sat, Sep 22, 2018 at 11:28 AM Alexandre Rafalovitch < >> [email protected]> wrote: >> >>> Do you know which Jira introduced the change? >>> >>> Without further details (I am barely familiar with that code part), it >>> sounds like a regression and possibly even a blocker for the next >>> version. Unless there is a workaround that is quite non-intuitive. >>> >>> Regards, >>> Alex. >>> >>> On 21 September 2018 at 11:43, Karl Wright <[email protected]> wrote: >>> > I should also mention that Solr Cell makes use of the multipart form's >>> > separator name. This functionality seems to be unavailable through >>> the URL. >>> > We have a number of clients who rely on this. >>> > >>> > Karl >>> > >>> > >>> > On Fri, Sep 21, 2018 at 10:15 AM Karl Wright <[email protected]> >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> With the release of 7.4, several things changed inside Solrj. One >>> >> critical thing that changed was that it now seems to be impossible to >>> send >>> >> POST requests with multipart form encoding. This functionality is >>> critical >>> >> to ManifoldCF since the metadata it sends to Solr regularly exceeds >>> URL >>> >> length. Here's a snippet of (modified) code from HttpSolrClient: >>> >> >>> >> >>>>>> >>> >> if(contentWriter != null) { >>> >> String fullQueryUrl = url + toQueryString(wparams, false); >>> >> HttpEntityEnclosingRequestBase postOrPut = >>> SolrRequest.METHOD.POST >>> >> == request.getMethod() ? >>> >> new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl); >>> >> System.out.println("Content type is >>> >> "+contentWriter.getContentType()); >>> >> postOrPut.addHeader("Content-Type", >>> >> contentWriter.getContentType()); >>> >> postOrPut.setEntity(new BasicHttpEntity(){ >>> >> @Override >>> >> public boolean isStreaming() { >>> >> return true; >>> >> } >>> >> >>> >> @Override >>> >> public void writeTo(OutputStream outstream) throws >>> IOException { >>> >> contentWriter.write(outstream); >>> >> } >>> >> }); >>> >> return postOrPut; >>> >> <<<<<< >>> >> >>> >> This takes over all transmissions of POST requests, in practice, and >>> >> transmits them as application/octet-stream, with metadata on the URL. >>> >> There's no way to override this; while I can instead used the >>> multipart >>> >> stuff consistently, when I do so the request to delete a document >>> fails, >>> >> with this SolrJ stack trace: >>> >> >>> >> >>>>>> >>> >> FATAL 2018-09-21T09:54:58,362 (Document delete thread '4') - Error >>> tossed: >>> >> This Should not happen >>> >> java.lang.RuntimeException: This Should not happen >>> >> at >>> >> >>> org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStreams(BinaryRequestWriter.java:67) >>> >> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc >>> - >>> >> jpountz - 2018-06-18 16:55:14] >>> >> at >>> >> >>> org.apache.manifoldcf.agents.output.solr.ModifiedHttpSolrClient.createMethod(ModifiedHttpSolrClient.java:148) >>> >> ~[classes/:?] >>> >> at >>> >> >>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253) >>> >> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc >>> - >>> >> jpountz - 2018-06-18 16:55:14] >>> >> at >>> >> >>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) >>> >> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc >>> - >>> >> jpountz - 2018-06-18 16:55:14] >>> >> at >>> >> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) >>> >> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc >>> - >>> >> jpountz - 2018-06-18 16:55:14] >>> >> at >>> >> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211) >>> >> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc >>> - >>> >> jpountz - 2018-06-18 16:55:14] >>> >> at >>> >> >>> org.apache.manifoldcf.agents.output.solr.HttpPoster$DeleteThread.run(HttpPoster.java:1364) >>> >> ~[classes/:?] >>> >> <<<<<< >>> >> >>> >> Any ideas? >>> >> >>> >> Karl >>> >> >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>>
