[jira] [Commented] (CONNECTORS-1533) Solr Connector is unable to ingest documents

[Karl Wright (JIRA)]

    [ 
https://issues.apache.org/jira/browse/CONNECTORS-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623544#comment-16623544
 ] 

Karl Wright commented on CONNECTORS-1533:
-----------------------------------------

For your reference, here is the Lucene master version of the HttpSolrClient 
method we needed to update in ModifiedHttpSolrClient to make compatible with 
SolrJ 7.4:

{code}
  protected HttpRequestBase createMethod(SolrRequest request, String 
collection) throws IOException, SolrServerException {
    if (request instanceof V2RequestSupport) {
      request = ((V2RequestSupport) request).getV2Request();
    }
    SolrParams params = request.getParams();
    RequestWriter.ContentWriter contentWriter = 
requestWriter.getContentWriter(request);
    Collection<ContentStream> streams = contentWriter == null ? 
requestWriter.getContentStreams(request) : null;
    String path = requestWriter.getPath(request);
    if (path == null || !path.startsWith("/")) {
      path = DEFAULT_PATH;
    }
    
    ResponseParser parser = request.getResponseParser();
    if (parser == null) {
      parser = this.parser;
    }
    
    // The parser 'wt=' and 'version=' params are used instead of the original
    // params
    ModifiableSolrParams wparams = new ModifiableSolrParams(params);
    if (parser != null) {
      wparams.set(CommonParams.WT, parser.getWriterType());
      wparams.set(CommonParams.VERSION, parser.getVersion());
    }
    if (invariantParams != null) {
      wparams.add(invariantParams);
    }

    String basePath = baseUrl;
    if (collection != null)
      basePath += "/" + collection;

    if (request instanceof V2Request) {
      if (System.getProperty("solr.v2RealPath") == null) {
        basePath = baseUrl.replace("/solr", "/api");
      } else {
        basePath = baseUrl + "/____v2";
      }
    }

    if (SolrRequest.METHOD.GET == request.getMethod()) {
      if (streams != null || contentWriter != null) {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "GET can't 
send streams!");
      }

      return new HttpGet(basePath + path + wparams.toQueryString());
    }

    if (SolrRequest.METHOD.DELETE == request.getMethod()) {
      return new HttpDelete(basePath + path + wparams.toQueryString());
    }

    if (SolrRequest.METHOD.POST == request.getMethod() || 
SolrRequest.METHOD.PUT == request.getMethod()) {

      String url = basePath + path;
      boolean hasNullStreamName = false;
      if (streams != null) {
        for (ContentStream cs : streams) {
          if (cs.getName() == null) {
            hasNullStreamName = true;
            break;
          }
        }
      }
      boolean isMultipart = ((this.useMultiPartPost && SolrRequest.METHOD.POST 
== request.getMethod())
          || (streams != null && streams.size() > 1)) && !hasNullStreamName;

      LinkedList<NameValuePair> postOrPutParams = new LinkedList<>();

      if(contentWriter != null) {
        String fullQueryUrl = url + wparams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = SolrRequest.METHOD.POST == 
request.getMethod() ?
            new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl);
        postOrPut.addHeader("Content-Type",
            contentWriter.getContentType());
        postOrPut.setEntity(new BasicHttpEntity(){
          @Override
          public boolean isStreaming() {
            return true;
          }

          @Override
          public void writeTo(OutputStream outstream) throws IOException {
            contentWriter.write(outstream);
          }
        });
        return postOrPut;

      } else if (streams == null || isMultipart) {
        // send server list and request list as query string params
        ModifiableSolrParams queryParams = 
calculateQueryParams(this.queryParams, wparams);
        queryParams.add(calculateQueryParams(request.getQueryParams(), 
wparams));
        String fullQueryUrl = url + queryParams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = fillContentStream(request, 
streams, wparams, isMultipart, postOrPutParams, fullQueryUrl);
        return postOrPut;
      }
      // It is has one stream, it is the post body, put the params in the URL
      else {
        String fullQueryUrl = url + wparams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = SolrRequest.METHOD.POST == 
request.getMethod() ?
            new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl);
        fillSingleContentStream(streams, postOrPut);

        return postOrPut;
      }
    }

    throw new SolrServerException("Unsupported method: " + request.getMethod());

  }
{code}

Please compare and contrast with what is committed to the ManifoldCF code base.


> Solr Connector is unable to ingest documents
> --------------------------------------------
>
>                 Key: CONNECTORS-1533
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1533
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Lucene/SOLR connector
>    Affects Versions: ManifoldCF 2.11
>            Reporter: Julien Massiera
>            Assignee: Karl Wright
>            Priority: Major
>
> The "r69acbd9 - Fix solr connector content deletion bug" has introduced 
> another bug : 
> It is now impossible to ingest documents into Solr 7.4.0, we obtain the 
> following error : Error from server at http://localhost:8983/solr/FileShare: 
> missing content stream
> The fact is, the requestWriter.getContentWriter(request) object is equal to 
> null only on commit requests. So the new lines of code introduced by the fix, 
> which are based on the test of this object, result in a null 
> Collection<ContentStream> streams object and so the update request is failing.
> Concerned class : 
> org.apache.manifoldcf.agents.output.solr.ModifiedHttpSolrClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)