Re: SolrClient from inside processAdd function
Hi,
Thanks for all this information. I am doing this now like following:
@Override
public void inform(SolrCore core) {
HttpSolrClient.Builder builder = new HttpSolrClient.Builder();
String baseURL =
core.getCoreContainer().getZkController().getBaseUrl() + "/" +
dataInfo.dataSource;
builder.withBaseSolrUrl(baseURL);
solrClient = builder.build();
core.addCloseHook(new CloseHook() {
@Override
public void preClose(SolrCore core) {
//no cleanup needed before closing the core
}
@Override
public void postClose(SolrCore core) {
try {
solrClient.close();
} catch (IOException e) {
logger.error("Not able to close solr client after closing " +
"Solr core '{}'", core.getName(), e);
}
}
});
}
On Fri, Sep 6, 2019 at 5:40 PM Arnold Bronley
wrote:
> Hi Markus,
>
> "Depending on cloudMode we create new SolrClient instances based on these
> classes. "
>
> But I still do not see SolrClient creation anywhere in your code snippet.
> Am I missing something? I tried the solution with system properties and it
> works but I would like to avoid that.
>
> On Thu, Sep 5, 2019 at 6:20 PM Markus Jelsma
> wrote:
>
>> Hello Arnold,
>>
>> In the Factory's inform() method you receive a SolrCore reference. Using
>> this you can get the CloudDescriptor and the ZkController references. These
>> provide access to what you need to open a connection for SolrClient.
>>
>> Our plugins usually work in cloud and non-cloud environments, so we
>> initialize different things for each situation. Like this abstracted in
>> some CloudUtils thing:
>>
>> cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
>> zk = core.getCoreContainer().getZkController(); // this is the
>> ZkController ref
>> coreName = core.getCoreDescriptor().getName();
>>
>> // Are we in cloud mode?
>> if (zk != null) {
>> collectionName = core.getCoreDescriptor().getCollectionName();
>> shardId = cloudDescriptor.getShardId();
>> } else {
>> collectionName = null;
>> shardId = null;
>> }
>>
>> Depending on cloudMode we create new SolrClient instances based on these
>> classes.
>>
>> Check the apidocs and you'll quickly see what you need.
>>
>> We use these api's to get what we need. But you can also find these
>> things if you check the Java system properties, which is easier. We use the
>> api's to read the data because if api's change, we get a compile error. If
>> the system properties change, we don't. So the system properties is easier,
>> but the api's are safer. Although a unit tests should guard against that as
>> well.
>>
>> Regards,
>> Markus
>>
>> ps, on this list there is normally no need to create a new thread for an
>> existing one, even if you are eagerly waiting for a reply. It might take
>> some patience though.
>>
>> -Original message-
>> > From:Arnold Bronley
>> > Sent: Thursday 5th September 2019 18:44
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: SolrClient from inside processAdd function
>> >
>> > Hi Markus,
>> >
>> > Is there any way to get the information about the current Solr endpoint
>> > from within the custom URP?
>> >
>> > On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma <
>> markus.jel...@openindex.io>
>> > wrote:
>> >
>> > > Hello Arnold,
>> > >
>> > > Yes, we do this too for several cases.
>> > >
>> > > You can create the SolrClient in the Factory's inform() method, and
>> pass
>> > > is to the URP when it is created. You must implement SolrCoreAware and
>> > > close the client when the core closes as well. Use a CloseHook for
>> this.
>> > >
>> > > If you do not close the client, it will cause trouble if you run unit
>> > > tests, and most certainly when you regularly reload cores.
>> > >
>> > > Regards,
>> > > Markus
>> > >
>> > >
>> > >
>> > > -Original message-
>> > > > From:Arnold Bronley
>> > > > Sent: Wednesday 4th September 2019 20:10
>> > > > To: solr-user@lucene.apache.org
>> > > > Subject: Re: SolrClient from inside processAdd function
>> > > >
>> > > > I need to search some other collection inside processAdd function
>> and
>> > > > append that information to the indexing request.
>> > > >
>> > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <
>> erickerick...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > This really sounds like an XY problem. What do you need the
>> SolrClient
>> > > > > _for_? I suspect there’s an easier way to do this…..
>> > > > >
>> > > > > Best,
>> > > > > Erick
>> > > > >
>> > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <
>> arnoldbron...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Is there a way to create SolrClient from inside processAdd
>> function
>> > > for
>> > > > > > custom update processor for the same Solr on which it is
>> executing?
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: SolrClient from inside processAdd function
Hi Markus,
"Depending on cloudMode we create new SolrClient instances based on these
classes. "
But I still do not see SolrClient creation anywhere in your code snippet.
Am I missing something? I tried the solution with system properties and it
works but I would like to avoid that.
On Thu, Sep 5, 2019 at 6:20 PM Markus Jelsma
wrote:
> Hello Arnold,
>
> In the Factory's inform() method you receive a SolrCore reference. Using
> this you can get the CloudDescriptor and the ZkController references. These
> provide access to what you need to open a connection for SolrClient.
>
> Our plugins usually work in cloud and non-cloud environments, so we
> initialize different things for each situation. Like this abstracted in
> some CloudUtils thing:
>
> cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
> zk = core.getCoreContainer().getZkController(); // this is the
> ZkController ref
> coreName = core.getCoreDescriptor().getName();
>
> // Are we in cloud mode?
> if (zk != null) {
> collectionName = core.getCoreDescriptor().getCollectionName();
> shardId = cloudDescriptor.getShardId();
> } else {
> collectionName = null;
> shardId = null;
> }
>
> Depending on cloudMode we create new SolrClient instances based on these
> classes.
>
> Check the apidocs and you'll quickly see what you need.
>
> We use these api's to get what we need. But you can also find these things
> if you check the Java system properties, which is easier. We use the api's
> to read the data because if api's change, we get a compile error. If the
> system properties change, we don't. So the system properties is easier, but
> the api's are safer. Although a unit tests should guard against that as
> well.
>
> Regards,
> Markus
>
> ps, on this list there is normally no need to create a new thread for an
> existing one, even if you are eagerly waiting for a reply. It might take
> some patience though.
>
> -Original message-----
> > From:Arnold Bronley
> > Sent: Thursday 5th September 2019 18:44
> > To: solr-user@lucene.apache.org
> > Subject: Re: SolrClient from inside processAdd function
> >
> > Hi Markus,
> >
> > Is there any way to get the information about the current Solr endpoint
> > from within the custom URP?
> >
> > On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma >
> > wrote:
> >
> > > Hello Arnold,
> > >
> > > Yes, we do this too for several cases.
> > >
> > > You can create the SolrClient in the Factory's inform() method, and
> pass
> > > is to the URP when it is created. You must implement SolrCoreAware and
> > > close the client when the core closes as well. Use a CloseHook for
> this.
> > >
> > > If you do not close the client, it will cause trouble if you run unit
> > > tests, and most certainly when you regularly reload cores.
> > >
> > > Regards,
> > > Markus
> > >
> > >
> > >
> > > -Original message-
> > > > From:Arnold Bronley
> > > > Sent: Wednesday 4th September 2019 20:10
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: SolrClient from inside processAdd function
> > > >
> > > > I need to search some other collection inside processAdd function and
> > > > append that information to the indexing request.
> > > >
> > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <
> erickerick...@gmail.com>
> > > > wrote:
> > > >
> > > > > This really sounds like an XY problem. What do you need the
> SolrClient
> > > > > _for_? I suspect there’s an easier way to do this…..
> > > > >
> > > > > Best,
> > > > > Erick
> > > > >
> > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <
> arnoldbron...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Is there a way to create SolrClient from inside processAdd
> function
> > > for
> > > > > > custom update processor for the same Solr on which it is
> executing?
> > > > >
> > > > >
> > > >
> > >
> >
>
RE: SolrClient from inside processAdd function
Hello Arnold,
In the Factory's inform() method you receive a SolrCore reference. Using this
you can get the CloudDescriptor and the ZkController references. These provide
access to what you need to open a connection for SolrClient.
Our plugins usually work in cloud and non-cloud environments, so we initialize
different things for each situation. Like this abstracted in some CloudUtils
thing:
cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
zk = core.getCoreContainer().getZkController(); // this is the ZkController
ref
coreName = core.getCoreDescriptor().getName();
// Are we in cloud mode?
if (zk != null) {
collectionName = core.getCoreDescriptor().getCollectionName();
shardId = cloudDescriptor.getShardId();
} else {
collectionName = null;
shardId = null;
}
Depending on cloudMode we create new SolrClient instances based on these
classes.
Check the apidocs and you'll quickly see what you need.
We use these api's to get what we need. But you can also find these things if
you check the Java system properties, which is easier. We use the api's to read
the data because if api's change, we get a compile error. If the system
properties change, we don't. So the system properties is easier, but the api's
are safer. Although a unit tests should guard against that as well.
Regards,
Markus
ps, on this list there is normally no need to create a new thread for an
existing one, even if you are eagerly waiting for a reply. It might take some
patience though.
-Original message-
> From:Arnold Bronley
> Sent: Thursday 5th September 2019 18:44
> To: solr-user@lucene.apache.org
> Subject: Re: SolrClient from inside processAdd function
>
> Hi Markus,
>
> Is there any way to get the information about the current Solr endpoint
> from within the custom URP?
>
> On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma
> wrote:
>
> > Hello Arnold,
> >
> > Yes, we do this too for several cases.
> >
> > You can create the SolrClient in the Factory's inform() method, and pass
> > is to the URP when it is created. You must implement SolrCoreAware and
> > close the client when the core closes as well. Use a CloseHook for this.
> >
> > If you do not close the client, it will cause trouble if you run unit
> > tests, and most certainly when you regularly reload cores.
> >
> > Regards,
> > Markus
> >
> >
> >
> > -Original message-
> > > From:Arnold Bronley
> > > Sent: Wednesday 4th September 2019 20:10
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: SolrClient from inside processAdd function
> > >
> > > I need to search some other collection inside processAdd function and
> > > append that information to the indexing request.
> > >
> > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson
> > > wrote:
> > >
> > > > This really sounds like an XY problem. What do you need the SolrClient
> > > > _for_? I suspect there’s an easier way to do this…..
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > Is there a way to create SolrClient from inside processAdd function
> > for
> > > > > custom update processor for the same Solr on which it is executing?
> > > >
> > > >
> > >
> >
>
Re: SolrClient from inside processAdd function
Hi Markus, Is there any way to get the information about the current Solr endpoint from within the custom URP? On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma wrote: > Hello Arnold, > > Yes, we do this too for several cases. > > You can create the SolrClient in the Factory's inform() method, and pass > is to the URP when it is created. You must implement SolrCoreAware and > close the client when the core closes as well. Use a CloseHook for this. > > If you do not close the client, it will cause trouble if you run unit > tests, and most certainly when you regularly reload cores. > > Regards, > Markus > > > > -Original message- > > From:Arnold Bronley > > Sent: Wednesday 4th September 2019 20:10 > > To: solr-user@lucene.apache.org > > Subject: Re: SolrClient from inside processAdd function > > > > I need to search some other collection inside processAdd function and > > append that information to the indexing request. > > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson > > wrote: > > > > > This really sounds like an XY problem. What do you need the SolrClient > > > _for_? I suspect there’s an easier way to do this….. > > > > > > Best, > > > Erick > > > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley > > > wrote: > > > > > > > > Hi, > > > > > > > > Is there a way to create SolrClient from inside processAdd function > for > > > > custom update processor for the same Solr on which it is executing? > > > > > > > > >
RE: SolrClient from inside processAdd function
Hello Arnold, Yes, we do this too for several cases. You can create the SolrClient in the Factory's inform() method, and pass is to the URP when it is created. You must implement SolrCoreAware and close the client when the core closes as well. Use a CloseHook for this. If you do not close the client, it will cause trouble if you run unit tests, and most certainly when you regularly reload cores. Regards, Markus -Original message- > From:Arnold Bronley > Sent: Wednesday 4th September 2019 20:10 > To: solr-user@lucene.apache.org > Subject: Re: SolrClient from inside processAdd function > > I need to search some other collection inside processAdd function and > append that information to the indexing request. > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson > wrote: > > > This really sounds like an XY problem. What do you need the SolrClient > > _for_? I suspect there’s an easier way to do this….. > > > > Best, > > Erick > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley > > wrote: > > > > > > Hi, > > > > > > Is there a way to create SolrClient from inside processAdd function for > > > custom update processor for the same Solr on which it is executing? > > > > >
Re: SolrClient from inside processAdd function
Hi Simon, I am interested in knowing what did you end up doing in your use-case then. Can you please share it at least at high level? On Wed, Sep 4, 2019 at 2:26 PM Simon Rosenthal wrote: > Similarly, I had considered a URP which would call the Solr Tagger to add > new metadata fields for indexing to incoming documents (and recall > discussing this with David Smiley), but eventually decided against this > approach on the grounds of complexity. > > -Simon > > On Wed, Sep 4, 2019 at 2:10 PM Arnold Bronley > wrote: > > > I need to search some other collection inside processAdd function and > > append that information to the indexing request. > > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson > > wrote: > > > > > This really sounds like an XY problem. What do you need the SolrClient > > > _for_? I suspect there’s an easier way to do this….. > > > > > > Best, > > > Erick > > > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley > > > wrote: > > > > > > > > Hi, > > > > > > > > Is there a way to create SolrClient from inside processAdd function > for > > > > custom update processor for the same Solr on which it is executing? > > > > > > > > > > > -- > I am transferring my email from Yahoo to simon.rosent...@gmail.com. I > will > continue to receive Yahoo email but will reply from this account. Please > update your address lists accordingly. >
Re: SolrClient from inside processAdd function
Similarly, I had considered a URP which would call the Solr Tagger to add new metadata fields for indexing to incoming documents (and recall discussing this with David Smiley), but eventually decided against this approach on the grounds of complexity. -Simon On Wed, Sep 4, 2019 at 2:10 PM Arnold Bronley wrote: > I need to search some other collection inside processAdd function and > append that information to the indexing request. > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson > wrote: > > > This really sounds like an XY problem. What do you need the SolrClient > > _for_? I suspect there’s an easier way to do this….. > > > > Best, > > Erick > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley > > wrote: > > > > > > Hi, > > > > > > Is there a way to create SolrClient from inside processAdd function for > > > custom update processor for the same Solr on which it is executing? > > > > > -- I am transferring my email from Yahoo to simon.rosent...@gmail.com. I will continue to receive Yahoo email but will reply from this account. Please update your address lists accordingly.
Re: SolrClient from inside processAdd function
I need to search some other collection inside processAdd function and append that information to the indexing request. On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson wrote: > This really sounds like an XY problem. What do you need the SolrClient > _for_? I suspect there’s an easier way to do this….. > > Best, > Erick > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley > wrote: > > > > Hi, > > > > Is there a way to create SolrClient from inside processAdd function for > > custom update processor for the same Solr on which it is executing? > >
Re: SolrClient from inside processAdd function
This really sounds like an XY problem. What do you need the SolrClient _for_? I suspect there’s an easier way to do this….. Best, Erick > On Sep 3, 2019, at 6:17 PM, Arnold Bronley wrote: > > Hi, > > Is there a way to create SolrClient from inside processAdd function for > custom update processor for the same Solr on which it is executing?
SolrClient from inside processAdd function
Hi, Is there a way to create SolrClient from inside processAdd function for custom update processor for the same Solr on which it is executing?
