SOLR 8.6.0 date Indexing Issues.
Hello Experts, I am having issues with indexing Date field in SOLR 8.6.0. I am indexing from MongoDB. In MongoDB the Format is as follows * "R_CREATION_DATE" : "12-Jul-18", "R_MODIFY_DATE" : "30-Apr-19", * In my Managed Schema I have the following entries. . I am getting an error in the Solr log. * org.apache.solr.common.SolrException: ERROR: [doc=mt_100] Error adding field 'R_MODIFY_DATE'='15-Jul-19' msg=Couldn't parse date because: Improperly formatted datetime: 15-Jul-19* Please let me know how to handle this usecase with Date format "12-JUL-18". what changes should I do to make it work ? Thanks Fiz N.
Trailing space issue with indexed data.
Hell SOLR Experts, I am using SOLR 8.6 and indexing data from MSSQL DB. after indexing is done I am seeing “Page_number”:”1“, “Doc_name”:” office 770 toll free “ “Doc_text”:” From: Hyan, gan \nTo: Delacruz Decruz \n“ I was remove these empty spaces. How to achieve it ? Thanks Fiz Nadian.
Re: Index files on Windows fileshare
Thanks Jason. Appreciate your response. Thanks Fiz N. On Thu, Jun 25, 2020 at 5:42 AM Jason Gerlowski wrote: > Hi Fiz, > > Since you're just looking for a POC solution, I think Solr's > "bin/post" tool would probably help you achieve your first > requirement. > > But I don't think "bin/post" gives you much control over the fields > that get indexed - if you need the file path to be stored, you might > be better off writing a small crawler in Java and using SolrJ to do > the indexing. > > Good luck! > > Jason > > On Fri, Jun 19, 2020 at 9:34 AM Fiz N wrote: > > > > Hello Solr experts, > > > > I am using standalone version of SOLR 8.5 on Windows machine. > > > > 1) I want to index all types of files under different directory in the > > file share. > > > > 2) I need to index absolute path of the files and store it solr field. I > > need that info so that end user can click and open the file(Pop-up) > > > > Could you please tell me how to go about this? > > This is for POC purpose once we finalize the solution we would be further > > going ahead with stable approach. > > > > Thanks > > Fiz Nadian. >
Re: Index file on Windows fileshare..
Thanks Erick. Is there easy way of doing this? Index files from windows share folder to SOLR. This is for POC only. Thanks Nadian. On Mon, Jun 22, 2020 at 3:54 PM Erick Erickson wrote: > Consider running Tika in a client and indexing the docs to Solr. > At that point, you have total control over what’s indexed. > > Here’s a skeletal program to get you started: > https://lucidworks.com/post/indexing-with-solrj/ > > Best, > Erick > > > On Jun 22, 2020, at 1:21 PM, Fiz N wrote: > > > > Hello Solr experts, > > > > I am using standalone version of SOLR 8.5 on Windows machine. > > > > 1) I want to index all types of files under different directory in the > > file share. > > > > 2) I need to index absolute path of the files and store it solr field. I > > need that info so that end user can click and open the file(Pop-up) > > > > Could you please tell me how to go about this? > > This is for POC purpose once we finalize the solution we would be further > > going ahead with stable approach. > > > > Thanks > > Fiz Nadian. > >
Index file on Windows fileshare..
Hello Solr experts, I am using standalone version of SOLR 8.5 on Windows machine. 1) I want to index all types of files under different directory in the file share. 2) I need to index absolute path of the files and store it solr field. I need that info so that end user can click and open the file(Pop-up) Could you please tell me how to go about this? This is for POC purpose once we finalize the solution we would be further going ahead with stable approach. Thanks Fiz Nadian.
Index files on Windows fileshare
Hello Solr experts, I am using standalone version of SOLR 8.5 on Windows machine. 1) I want to index all types of files under different directory in the file share. 2) I need to index absolute path of the files and store it solr field. I need that info so that end user can click and open the file(Pop-up) Could you please tell me how to go about this? This is for POC purpose once we finalize the solution we would be further going ahead with stable approach. Thanks Fiz Nadian.
Re: Indexing PDF on SOLR 8.5
Thanks Erick... On Sun, Jun 7, 2020 at 1:50 PM Erick Erickson wrote: > https://lucidworks.com/post/indexing-with-solrj/ > > > > On Jun 7, 2020, at 3:22 PM, Fiz N wrote: > > > > Thanks Jorn and Erick. > > > > Hi Erick, looks like the skeletal SOLRJ program attachment is missing. > > > > Thanks > > Fiz > > > > On Sun, Jun 7, 2020 at 12:20 PM Erick Erickson > > wrote: > > > >> Here’s a skeletal SolrJ program using Tika as another alternative. > >> > >> Best, > >> Erick > >> > >>> On Jun 7, 2020, at 2:06 PM, Jörn Franke wrote: > >>> > >>> You have to write an external application that creates multiple > threads, > >> parses the PDFs and index them in Solr. Ideally you parse the PDFs once > and > >> store the resulting text on some file system and then index it. Reason > is > >> that if you upgrade to two major versions of Solr you might need to > reindex > >> again. Then you can save time because you don’t need to parse the PDFs > >> again. > >>> It can be also useful in case you are not sure yet about the final > >> schema and need to index several times in different schemas etc > >>> > >>> You can also use Apache manifoldCF. > >>> > >>> > >>> > >>>> Am 07.06.2020 um 19:19 schrieb Fiz N : > >>>> > >>>> Hello SOLR Experts, > >>>> > >>>> I am working on a POC to Index millions of PDF documents present in > >>>> Multiple Folder in fileshare. > >>>> > >>>> Could you please let me the best practices and step to implement it. > >>>> > >>>> Thanks > >>>> Fiz Nadiyal. > >> > >> > >
Re: Indexing PDF on SOLR 8.5
Thanks Jorn and Erick. Hi Erick, looks like the skeletal SOLRJ program attachment is missing. Thanks Fiz On Sun, Jun 7, 2020 at 12:20 PM Erick Erickson wrote: > Here’s a skeletal SolrJ program using Tika as another alternative. > > Best, > Erick > > > On Jun 7, 2020, at 2:06 PM, Jörn Franke wrote: > > > > You have to write an external application that creates multiple threads, > parses the PDFs and index them in Solr. Ideally you parse the PDFs once and > store the resulting text on some file system and then index it. Reason is > that if you upgrade to two major versions of Solr you might need to reindex > again. Then you can save time because you don’t need to parse the PDFs > again. > > It can be also useful in case you are not sure yet about the final > schema and need to index several times in different schemas etc > > > > You can also use Apache manifoldCF. > > > > > > > >> Am 07.06.2020 um 19:19 schrieb Fiz N : > >> > >> Hello SOLR Experts, > >> > >> I am working on a POC to Index millions of PDF documents present in > >> Multiple Folder in fileshare. > >> > >> Could you please let me the best practices and step to implement it. > >> > >> Thanks > >> Fiz Nadiyal. > >
Indexing PDF on SOLR 8.5
Hello SOLR Experts, I am working on a POC to Index millions of PDF documents present in Multiple Folder in fileshare. Could you please let me the best practices and step to implement it. Thanks Fiz Nadiyal.
Re: Login Mechanism in SOLR
Yes Thanks and it was helpful. Thanks Fiz.. On Mon, May 18, 2020 at 8:14 AM Jan Høydahl wrote: > Hi, > > Have you read what the reference guide has to say on the topic? > > > https://lucene.apache.org/solr/guide/8_5/authentication-and-authorization-plugins.html > > Jan > > > 18. mai 2020 kl. 17:00 skrev Fiz N : > > > > Hi SOLR Experts, > > > > Could you please tell us options for login mechanism in SOLR? Is custom > or > > application login possible and also are SSO and LDAP based > > authentication/authorization possible? What is the best approach I can > > proceed with? > > > > Thanks & Regards > > Fiz Ahmed. > >
Login Mechanism in SOLR
Hi SOLR Experts, Could you please tell us options for login mechanism in SOLR? Is custom or application login possible and also are SSO and LDAP based authentication/authorization possible? What is the best approach I can proceed with? Thanks & Regards Fiz Ahmed.
2 Async exceptions during distributed update issue...
Hello Solr Experts,
Just wanted to follow up in case my question.Would appreciate help on this.
SOLR Version : 6.6.2
OS – Linux 3.1.2
JDK – 1.8
Shard – 16 – All are active.
Xms – 16 gb
Xmx – 16 gb
Host has 64 cores.
the update processor chain.attaching the complete updateRequestProcessorChain
in a file.
Attaching physical and CPU memory screenshot.
There are multiple threads sending products to solr. tested with batch size
per thread 50 & 100, it worked fine without error and if the batch size is
1000 the following error occurs.
*I am getting the following error when the batch size is 1000. Please
advise.*
2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard15_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
2 Async exceptions during distributed update:
10.YYY.40.62:8983 failed to respond
10.YYY.40.62:8983 failed to respond
2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard7_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: 10.YYY.40.81:8983 failed to
respond
2019-11-14T19:36:11,599 - ERROR
[updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
r:core_node26
n:10.YYY.40.68:8983_solr c:ducts
s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
,
collection=c:ducts, core=x:ducts_shard3_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3}
- error
org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
~[httpclient-4.4.1.jar:4.4.1]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
~[httpclient-4.4.1.jar:4.4.1]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
~[httpcore-4.4.1.jar:4.4.1]
2019-11-14T19:36:14,567 - ERROR
[updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
r:core_node25
n:10.YYY.40.68:8983_solr c:ducts
s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
,
collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
- error
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
~[?:1.8.0_232]
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
~[?:1.8.0_232]
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
~[?:1.8.0_232]
2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] -
{collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
-
null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Broken pipe (Write failed)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
Thanks
[^\w-\.]
_
-MM-dd'T'HH:mm:ss.SSSZ
-MM-dd'T'HH:mm:ss,SSSZ
-MM-dd'T'HH:mm:ss.SSS
-MM-dd'T'HH:mm:ss,SSS
-MM-dd'T'HH:mm:ssZ
-MM-dd'T'HH:mm:ss
-MM-dd'T'HH:mmZ
-MM-dd'T'HH:mm
-MM-dd HH:mm:ss.SSSZ
-MM-dd HH:mm:ss,SSSZ
-MM-dd HH:mm:ss.SSS
-MM-dd HH:mm:ss,SSS
-MM-dd HH:mm:ssZ
-MM-dd HH:mm:ss
-MM-dd HH:mmZ
-MM-dd HH:mm
-MM-dd
strings
java.lang.Boolean
booleans
java.util.Date
tdates
java.lang.Long
java.lang.Integer
tlongs
java.lang.Number
tdoubles
Fwd: 2 Async exceptions during distributed update
Thanks Jorn on getting back to me. Please find attached memory/cpu
utilization and complete update process chain.
Please let me know your thoughts.
On Sat, Nov 16, 2019 at 9:09 AM Jörn Franke wrote:
> Can you please provide the whole update chain ? Below is the
> directoryFactory and not the update processor chain.
>
> > Am 15.11.2019 um 17:34 schrieb Fiz N :
> >
> >
> > Thanks for your response.
> >
> > name="DirectoryFactory"class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
> >
> > attaching the screenshot of physical memory and cpu.
> > Please let me know your thoughts on the below issue.
> >
> >
> >> On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke
> wrote:
> >> Do you use a updateprocess factory? How does it look like?
> >>
> >> What is the physical memory size and CPU?
> >> What do you mean by “there are 64 cores sending concurrently?” An
> application has 64 threats that send concurrently those updates?
> >>
> >> > Am 15.11.2019 um 02:14 schrieb Fiz N :
> >> >
> >> > Hi Solr Experts,
> >> >
> >> > SOLR Version : 6.6.2
> >> >
> >> > OS – Linux 3.1.2
> >> >
> >> > JDK – 1.8
> >> >
> >> >
> >> >
> >> > Shard – 16 – All are active.
> >> >
> >> > Xms – 16 gb
> >> >
> >> > Xmx – 16 gb
> >> >
> >> >
> >> >
> >> > Schema fields count – 91
> >> >
> >> > Dynamic fields – 83.
> >> >
> >> >
> >> >
> >> > There are multiple threads sending products to solr. tested with
> batch size
> >> > per thread 50 & 100, it worked fine without error and if the batch
> size is
> >> > 1000 the following error occurs.
> >> >
> >> > there are 64 cores that are sending batches concurrently.
> >> >
> >> >
> >> >
> >> > *I am getting the following error when the batch size is 1000. Please
> >> > advise.*
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard15_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
> shard=s:shard15}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > 2 Async exceptions during distributed update:
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard7_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
> shard=s:shard7}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
> >> > respond
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 2019-11-14T19:36:11,599 - ERROR
> >> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
> >> > http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>
> r:core_node26
> >> > n:10.YYY.40.68:8983_solr c:ducts
> >> >
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> >> > - {ConcurrentUpdateSolrClient.url=
> >> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>,
> >> > collection=c:ducts, core=x:ducts_shard3_replica2,
> >> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
> shard=s:shard3}
> >> > - error
> >> >
> >> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
> respond
> >> >
> >> >at
> >> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> >> > ~[httpclient-4.4.1.jar:4.4.1]
> >> >
> >> >at
> >> >
> org.apache.http.impl.conn.Def
Re: 2 Async exceptions during distributed update
Thanks Jorn on getting back to me. Please find attached memory/cpu
utilization and complete update process chain.
Please let me know your thoughts.
On Sat, Nov 16, 2019 at 9:09 AM Jörn Franke wrote:
> Can you please provide the whole update chain ? Below is the
> directoryFactory and not the update processor chain.
>
> > Am 15.11.2019 um 17:34 schrieb Fiz N :
> >
> >
> > Thanks for your response.
> >
> > name="DirectoryFactory"class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
> >
> > attaching the screenshot of physical memory and cpu.
> > Please let me know your thoughts on the below issue.
> >
> >
> >> On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke
> wrote:
> >> Do you use a updateprocess factory? How does it look like?
> >>
> >> What is the physical memory size and CPU?
> >> What do you mean by “there are 64 cores sending concurrently?” An
> application has 64 threats that send concurrently those updates?
> >>
> >> > Am 15.11.2019 um 02:14 schrieb Fiz N :
> >> >
> >> > Hi Solr Experts,
> >> >
> >> > SOLR Version : 6.6.2
> >> >
> >> > OS – Linux 3.1.2
> >> >
> >> > JDK – 1.8
> >> >
> >> >
> >> >
> >> > Shard – 16 – All are active.
> >> >
> >> > Xms – 16 gb
> >> >
> >> > Xmx – 16 gb
> >> >
> >> >
> >> >
> >> > Schema fields count – 91
> >> >
> >> > Dynamic fields – 83.
> >> >
> >> >
> >> >
> >> > There are multiple threads sending products to solr. tested with
> batch size
> >> > per thread 50 & 100, it worked fine without error and if the batch
> size is
> >> > 1000 the following error occurs.
> >> >
> >> > there are 64 cores that are sending batches concurrently.
> >> >
> >> >
> >> >
> >> > *I am getting the following error when the batch size is 1000. Please
> >> > advise.*
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard15_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
> shard=s:shard15}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > 2 Async exceptions during distributed update:
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard7_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
> shard=s:shard7}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
> >> > respond
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 2019-11-14T19:36:11,599 - ERROR
> >> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
> >> > http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>
> r:core_node26
> >> > n:10.YYY.40.68:8983_solr c:ducts
> >> >
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> >> > - {ConcurrentUpdateSolrClient.url=
> >> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>,
> >> > collection=c:ducts, core=x:ducts_shard3_replica2,
> >> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
> shard=s:shard3}
> >> > - error
> >> >
> >> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
> respond
> >> >
> >> >at
> >> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> >> > ~[httpclient-4.4.1.jar:4.4.1]
> >> >
> >> >at
> >> >
> org.apache.http.impl.conn.Def
Re: 2 Async exceptions during distributed update
Hi Solr Experts,
Do you have any thoughts on the below issue ?
On Fri, Nov 15, 2019 at 11:33 AM Fiz N wrote:
> Thanks for your response.
>
> "${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>
>
>
> attaching the screenshot of physical memory and cpu.
>
> Please let me know your thoughts on the below issue.
>
>
>
> On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke wrote:
>
>> Do you use a updateprocess factory? How does it look like?
>>
>> What is the physical memory size and CPU?
>> What do you mean by “there are 64 cores sending concurrently?” An
>> application has 64 threats that send concurrently those updates?
>>
>> > Am 15.11.2019 um 02:14 schrieb Fiz N :
>> >
>> > Hi Solr Experts,
>> >
>> > SOLR Version : 6.6.2
>> >
>> > OS – Linux 3.1.2
>> >
>> > JDK – 1.8
>> >
>> >
>> >
>> > Shard – 16 – All are active.
>> >
>> > Xms – 16 gb
>> >
>> > Xmx – 16 gb
>> >
>> >
>> >
>> > Schema fields count – 91
>> >
>> > Dynamic fields – 83.
>> >
>> >
>> >
>> > There are multiple threads sending products to solr. tested with batch
>> size
>> > per thread 50 & 100, it worked fine without error and if the batch size
>> is
>> > 1000 the following error occurs.
>> >
>> > there are 64 cores that are sending batches concurrently.
>> >
>> >
>> >
>> > *I am getting the following error when the batch size is 1000. Please
>> > advise.*
>> >
>> >
>> >
>> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
>> > {collection=c:ducts, core=x:ducts_shard15_replica1,
>> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
>> shard=s:shard15}
>> > -
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> > 2 Async exceptions during distributed update:
>> >
>> > 10.YYY.40.62:8983 failed to respond
>> >
>> > 10.YYY.40.62:8983 failed to respond
>> >
>> >
>> >
>> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
>> > {collection=c:ducts, core=x:ducts_shard7_replica1,
>> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
>> shard=s:shard7}
>> > -
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
>> > respond
>> >
>> >
>> >
>> >
>> >
>> > 2019-11-14T19:36:11,599 - ERROR
>> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
>> > http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
>> > <http://10.148.40.68:8983/solr/products_shard11_replica2> r:core_node26
>> > n:10.YYY.40.68:8983_solr c:ducts
>> >
>> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
>> ]
>> > - {ConcurrentUpdateSolrClient.url=
>> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
>> > <http://10.148.40.68:8983/solr/products_shard11_replica2>,
>> > collection=c:ducts, core=x:ducts_shard3_replica2,
>> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
>> shard=s:shard3}
>> > - error
>> >
>> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
>> respond
>> >
>> >at
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
>> > ~[httpclient-4.4.1.jar:4.4.1]
>> >
>> >at
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>> > ~[httpclient-4.4.1.jar:4.4.1]
>> >
>> >at
>> > org.apache.http.impl.io
>> .AbstractMessageParser.parse(AbstractMessageParser.java:261)
>> > ~[httpcore-4.4.1.jar:4.4.1]
>> >
>> >
>> >
>> >
>> >
>> > 2019-11-14T19:36:14,567 - ERROR
>> > [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
>> > http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
>> > <http://10.148.40.62:8983/solr/products_shard2_replica1> r:core_node25
>> > n:10.YYY.40.68:8983_solr c:ducts
>> >
>> s:shard11:StreamingSolrClients$ErrorReportingCo
Re: 2 Async exceptions during distributed update
Thanks for your response.
attaching the screenshot of physical memory and cpu.
Please let me know your thoughts on the below issue.
On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke wrote:
> Do you use a updateprocess factory? How does it look like?
>
> What is the physical memory size and CPU?
> What do you mean by “there are 64 cores sending concurrently?” An
> application has 64 threats that send concurrently those updates?
>
> > Am 15.11.2019 um 02:14 schrieb Fiz N :
> >
> > Hi Solr Experts,
> >
> > SOLR Version : 6.6.2
> >
> > OS – Linux 3.1.2
> >
> > JDK – 1.8
> >
> >
> >
> > Shard – 16 – All are active.
> >
> > Xms – 16 gb
> >
> > Xmx – 16 gb
> >
> >
> >
> > Schema fields count – 91
> >
> > Dynamic fields – 83.
> >
> >
> >
> > There are multiple threads sending products to solr. tested with batch
> size
> > per thread 50 & 100, it worked fine without error and if the batch size
> is
> > 1000 the following error occurs.
> >
> > there are 64 cores that are sending batches concurrently.
> >
> >
> >
> > *I am getting the following error when the batch size is 1000. Please
> > advise.*
> >
> >
> >
> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
> > {collection=c:ducts, core=x:ducts_shard15_replica1,
> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
> shard=s:shard15}
> > -
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > 2 Async exceptions during distributed update:
> >
> > 10.YYY.40.62:8983 failed to respond
> >
> > 10.YYY.40.62:8983 failed to respond
> >
> >
> >
> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
> > {collection=c:ducts, core=x:ducts_shard7_replica1,
> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
> shard=s:shard7}
> > -
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
> > respond
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:11,599 - ERROR
> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
> > http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
> > <http://10.148.40.68:8983/solr/products_shard11_replica2> r:core_node26
> > n:10.YYY.40.68:8983_solr c:ducts
> >
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> > - {ConcurrentUpdateSolrClient.url=
> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
> > <http://10.148.40.68:8983/solr/products_shard11_replica2>,
> > collection=c:ducts, core=x:ducts_shard3_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
> shard=s:shard3}
> > - error
> >
> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
> respond
> >
> >at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> > ~[httpclient-4.4.1.jar:4.4.1]
> >
> >at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> > ~[httpclient-4.4.1.jar:4.4.1]
> >
> >at
> > org.apache.http.impl.io
> .AbstractMessageParser.parse(AbstractMessageParser.java:261)
> > ~[httpcore-4.4.1.jar:4.4.1]
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:14,567 - ERROR
> > [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
> > http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
> > <http://10.148.40.62:8983/solr/products_shard2_replica1> r:core_node25
> > n:10.YYY.40.68:8983_solr c:ducts
> >
> s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> > - {ConcurrentUpdateSolrClient.url=
> > http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
> > <http://10.148.40.62:8983/solr/products_shard2_replica1>,
> > collection=c:ducts, core=x:ducts_shard11_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> > - error
> >
> > java.net.SocketException: Broken pipe (Write failed)
> >
> >at java.net.SocketOutputStream.socketWrite0(Native Method)
> > ~[?:1.8.0_232]
> >
> >at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > ~[?:1.8.0_232]
> >
> >at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > ~[?:1.8.0_232]
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] -
> > {collection=c:ducts, core=x:ducts_shard11_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> > -
> >
> null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > Async exception during distributed update: Broken pipe (Write failed)
> >
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
> >
> >at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
> >
> >
> >
> >
> > Thanks
> >
> > Fiz..
>
2 Async exceptions during distributed update
Hi Solr Experts,
SOLR Version : 6.6.2
OS – Linux 3.1.2
JDK – 1.8
Shard – 16 – All are active.
Xms – 16 gb
Xmx – 16 gb
Schema fields count – 91
Dynamic fields – 83.
There are multiple threads sending products to solr. tested with batch size
per thread 50 & 100, it worked fine without error and if the batch size is
1000 the following error occurs.
there are 64 cores that are sending batches concurrently.
*I am getting the following error when the batch size is 1000. Please
advise.*
2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard15_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
2 Async exceptions during distributed update:
10.YYY.40.62:8983 failed to respond
10.YYY.40.62:8983 failed to respond
2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard7_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: 10.YYY.40.81:8983 failed to
respond
2019-11-14T19:36:11,599 - ERROR
[updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
r:core_node26
n:10.YYY.40.68:8983_solr c:ducts
s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
,
collection=c:ducts, core=x:ducts_shard3_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3}
- error
org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
~[httpclient-4.4.1.jar:4.4.1]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
~[httpclient-4.4.1.jar:4.4.1]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
~[httpcore-4.4.1.jar:4.4.1]
2019-11-14T19:36:14,567 - ERROR
[updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
r:core_node25
n:10.YYY.40.68:8983_solr c:ducts
s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
,
collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
- error
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
~[?:1.8.0_232]
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
~[?:1.8.0_232]
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
~[?:1.8.0_232]
2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] -
{collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
-
null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Broken pipe (Write failed)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
Thanks
Fiz..
