RE: Solr Size Limitation upto 32 kb limitation
Hi Team, We didn't get any suggested solutions. Could you help us by providing better approach or a solution to fix the issue? We'll be awaiting your reply. [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Friday, February 1, 2019 10:26 AM To: dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida ; Srinivasa Reddy Karri ; Ravi Vangala ; Suresh Malladi ; Vijay Nandula ; Michelle Ngo Subject: Re: Solr Size Limitation upto 32 kb limitation Hi Team, Thanks for your suggestions that you've posted, but none of them have fixed our issue. Could you please provide us your valuable suggestions to address this issue. We'll be awaiting your reply. Thanks, Kranthi kumar.K From: Michelle Ngo Sent: Thursday, January 24, 2019 12:00:06 PM To: Kranthi Kumar K; dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida; Srinivasa Reddy Karri; Ravi Vangala; Suresh Malladi; Vijay Nandula Subject: RE: Solr Size Limitation upto 32 kb limitation Thanks @Kranthi Kumar K<mailto:kranthikuma...@ccubefintech.com> for following up From: Kranthi Kumar K mailto:kranthikuma...@ccubefintech.com>> Sent: Thursday, 24 January 2019 4:51 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>>; Ravi Vangala mailto:ravi.vang...@ccubefintech.com>>; Suresh Malladi mailto:sur...@ccubefintech.com>>; Vijay Nandula mailto:vijay.nand...@ccubefintech.com>> Subject: RE: Solr Size Limitation upto 32 kb limitation Thank you Bernd Fehling for your suggested solution, I've tried the same by changing the type and added multivalued to true in Schema.xml file i.e, change from: Changed to: After changing it also still we are unable to import the files size > 32 kb. please find the solution suggested by Bernd in the below url: http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html Bernd Fehling, could you please suggest another alternative solution to resolve our issue, which would help us alot? Please let me know for any questions. [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Friday, January 18, 2019 4:22 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle....@ccube.com.au>>; Ravi Vangala mailto:ravi.vang...@ccubefintech.com>> Subject: RE: Solr Size Limitation upto 32 kb limitation Hi team, Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I've tried the suggested one's and still we are unable to import files havingsize >32 kb, it is displaying same error. Below link has the suggested solutions. Please have a look once. http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html 1. As per Erick Erickson, I've changed the string type to Text type based and still the issue occurs . I've changed from : Changed to: If we do so, it is showing error in the log, please find the error in the attachment. If I change to: It is not showing any error , but the issue still exists. 1. As per Jan Hoydahl, I have gone through the link that you have provided and checked 'requestParsers' tag in solrconfig.xml, RequestParsers tag in our application is as follows: '' Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb. 1. As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, 'If you are trying to add larger content then you have to "chop" that by yourself and add it as multivalued. Can be done within a self written loader. ' I'm a newbie to Solr and I didn't get what exactly 'self written loader' is? Could you please provide us sample code, that helps us to go further? [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer,
Re: Solr Size Limitation upto 32 kb limitation
Hi Team, Thanks for your suggestions that you've posted, but none of them have fixed our issue. Could you please provide us your valuable suggestions to address this issue. We'll be awaiting your reply. Thanks, Kranthi kumar.K From: Michelle Ngo Sent: Thursday, January 24, 2019 12:00:06 PM To: Kranthi Kumar K; dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida; Srinivasa Reddy Karri; Ravi Vangala; Suresh Malladi; Vijay Nandula Subject: RE: Solr Size Limitation upto 32 kb limitation Thanks @Kranthi Kumar K<mailto:kranthikuma...@ccubefintech.com> for following up From: Kranthi Kumar K Sent: Thursday, 24 January 2019 4:51 PM To: dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida ; Srinivasa Reddy Karri ; Michelle Ngo ; Ravi Vangala ; Suresh Malladi ; Vijay Nandula Subject: RE: Solr Size Limitation upto 32 kb limitation Thank you Bernd Fehling for your suggested solution, I've tried the same by changing the type and added multivalued to true in Schema.xml file i.e, change from: Changed to: After changing it also still we are unable to import the files size > 32 kb. please find the solution suggested by Bernd in the below url: http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html Bernd Fehling, could you please suggest another alternative solution to resolve our issue, which would help us alot? Please let me know for any questions. [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Friday, January 18, 2019 4:22 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>>; Ravi Vangala mailto:ravi.vang...@ccubefintech.com>> Subject: RE: Solr Size Limitation upto 32 kb limitation Hi team, Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I’ve tried the suggested one’s and still we are unable to import files havingsize >32 kb, it is displaying same error. Below link has the suggested solutions. Please have a look once. http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html 1. As per Erick Erickson, I’ve changed the string type to Text type based and still the issue occurs . I’ve changed from : Changed to: If we do so, it is showing error in the log, please find the error in the attachment. If I change to: It is not showing any error , but the issue still exists. 1. As per Jan Hoydahl, I have gone through the link that you have provided and checked ‘requestParsers’ tag in solrconfig.xml, RequestParsers tag in our application is as follows: ‘’ Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb. 1. As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, ‘If you are trying to add larger content then you have to "chop" that by yourself and add it as multivalued. Can be done within a self written loader. ’ I’m a newbie to Solr and I didn’t get what exactly ‘self written loader’ is? Could you please provide us sample code, that helps us to go further? [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K mailto:kranthikuma...@ccubefintech.com>> Sent: Thursday, January 17, 2019 12:43 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>> Subject: Re: Solr Size Limitation upto 32 kb limitation Hi Team, Can we have any updates on the below issue? We are awaiting your reply. Thanks, Kranthi kumar.K From: Kranthi Kumar K Sent: Friday, January 4, 2019 5:01:38 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org> Cc: Ananda Babu medida; Srinivasa Reddy Karri Subject: Solr Size Limitation upto 32 kb limitation Hi team, We are currently using Solr 4.2.1 version in our project and everything is going well. But recently,
Re: Solr Size Limitation upto 32 kb limitation
What Jan said. If you are getting this error on a _text_ based field, then your data is bad. What it's telling you is that _after_ tokenization, you have a single _term_ that's > 32K which is almost, but not quite totally, useless. ImagineASingleWordThatRunsOnForMoreThanThirtyTwoThousandCharactersHowWouldThatBeUsefulToEitherSearchOrReturnToTheUserWouldThisSingleWordImTypingBeUsefulAndItIsntEvenCloseToThirtyTwoThousandCharacters. So I'd try to find out what it is you're processing that shows you such a large term. It's pretty easy to run Tika on a file in SolrJ, see: https://lucidworks.com/2012/02/14/indexing-with-solrj/ There are also web sites that'll process the PDF file through Tika and show you how it parses Best, Erick On Thu, Jan 24, 2019 at 12:57 AM Jan Høydahl wrote: > > I cannot see why you'd want a single term of 32kb in your index anyway. Can > you give us examples of what these terms are and how you will search them? > What kind of files are you indexing, could it be like bad PDFs consisting of > a bunch of binary garbage? > Try adding a lengthFilterFactory to your fieldType(s). See > https://lucene.apache.org/solr/guide/7_6/filter-descriptions.html#length-filter > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > > 24. jan. 2019 kl. 06:51 skrev Kranthi Kumar K > : > > Thank you Bernd Fehling for your suggested solution, I've tried the same by > changing the type and added multivalued to true in Schema.xml file i.e, > change from: > > > > Changed to: > > multiValued="true" /> > > After changing it also still we are unable to import the files size > 32 kb. > please find the solution suggested by Bernd in the below url: > > http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html > > Bernd Fehling, could you please suggest another alternative solution to > resolve our issue, which would help us alot? > > Please let me know for any questions. > > > > > Thanks & Regards, > Kranthi Kumar.K, > Software Engineer, > Ccube Fintech Global Services Pvt Ltd., > Email/Skype: kranthikuma...@ccubefintech.com, > Mobile: +91-8978078449. > > > From: Kranthi Kumar K > Sent: Friday, January 18, 2019 4:22 PM > To: dev@lucene.apache.org; solr-u...@lucene.apache.org > Cc: Ananda Babu medida ; Srinivasa Reddy > Karri ; Michelle Ngo > ; Ravi Vangala > Subject: RE: Solr Size Limitation upto 32 kb limitation > > Hi team, > > Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested > solutions. I’ve tried the suggested one’s and still we are unable to import > files havingsize >32 kb, it is displaying same error. > > Below link has the suggested solutions. Please have a look once. > > http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html > > > As per Erick Erickson, I’ve changed the string type to Text type based and > still the issue occurs . > > I’ve changed from : > > > > Changed to: > > > > If we do so, it is showing error in the log, please find the error in the > attachment. > > If I change to: > > > > It is not showing any error , but the issue still exists. > > > As per Jan Hoydahl, I have gone through the link that you have provided and > checked ‘requestParsers’ tag in solrconfig.xml, > > > RequestParsers tag in our application is as follows: > > ‘ multipartUploadLimitInKB="2048000" > formdataUploadLimitInKB="2048" > addHttpRequestToContext="false"/>’ > Request parsers, which we are using and in the link you have provided are > similar. And still we are unable to import the files size >32 kb. > > > As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, > > ‘If you are trying to add larger content then you have to "chop" that > by yourself and add it as multivalued. Can be done within a self written > loader. ’ > > I’m a newbie to Solr and I didn’t get what exactly ‘self written loader’ is? > > Could you please provide us sample code, that helps us to go further? > > > > > > Thanks & Regards, > Kranthi Kumar.K, > Software Engineer, > Ccube Fintech Global Services Pvt Ltd., > Email/Skype: kranthikuma...@ccubefintech.com, > Mobile: +91-8978078449. > > > From: Kranthi Kumar K > Sent: Thursday, January 17, 2019 12:43 PM > To: dev@lucene.apache.org; solr-u...@lucene.apache.org > Cc: Ananda Babu medida ; Srinivasa Reddy > Karri ; Michelle Ngo > > Subject: Re: Solr Size Limitation upto 32 kb limitation > > > H
Re: Solr Size Limitation upto 32 kb limitation
I cannot see why you'd want a single term of 32kb in your index anyway. Can you give us examples of what these terms are and how you will search them? What kind of files are you indexing, could it be like bad PDFs consisting of a bunch of binary garbage? Try adding a lengthFilterFactory to your fieldType(s). See https://lucene.apache.org/solr/guide/7_6/filter-descriptions.html#length-filter <https://lucene.apache.org/solr/guide/7_6/filter-descriptions.html#length-filter> -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 24. jan. 2019 kl. 06:51 skrev Kranthi Kumar K > : > > Thank you Bernd Fehling for your suggested solution, I've tried the same by > changing the type and added multivalued to true in Schema.xml file i.e, > change from: > > > > Changed to: > > multiValued="true" /> > > After changing it also still we are unable to import the files size > 32 kb. > please find the solution suggested by Bernd in the below url: > > http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html > > <http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html> > > Bernd Fehling, could you please suggest another alternative solution to > resolve our issue, which would help us alot? > > Please let me know for any questions. > > > > Thanks & Regards, > Kranthi Kumar.K, > Software Engineer, > Ccube Fintech Global Services Pvt Ltd., > Email/Skype: kranthikuma...@ccubefintech.com > <mailto:kranthikuma...@ccubefintech.com>, > Mobile: +91-8978078449. > > > From: Kranthi Kumar K > Sent: Friday, January 18, 2019 4:22 PM > To: dev@lucene.apache.org <mailto:dev@lucene.apache.org>; > solr-u...@lucene.apache.org <mailto:solr-u...@lucene.apache.org> > Cc: Ananda Babu medida <mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri > <mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo > mailto:michelle@ccube.com.au>>; Ravi Vangala > mailto:ravi.vang...@ccubefintech.com>> > Subject: RE: Solr Size Limitation upto 32 kb limitation > > Hi team, > > Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested > solutions. I’ve tried the suggested one’s and still we are unable to import > files havingsize >32 kb, it is displaying same error. > > Below link has the suggested solutions. Please have a look once. > > http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html > > <http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html> > > As per Erick Erickson, I’ve changed the string type to Text type based and > still the issue occurs . > I’ve changed from : > > > > Changed to: > > > > If we do so, it is showing error in the log, please find the error in the > attachment. > > If I change to: > > > > It is not showing any error , but the issue still exists. > > As per Jan Hoydahl, I have gone through the link that you have provided and > checked ‘requestParsers’ tag in solrconfig.xml, > > RequestParsers tag in our application is as follows: > > ‘ multipartUploadLimitInKB="2048000" > formdataUploadLimitInKB="2048" > addHttpRequestToContext="false"/>’ > Request parsers, which we are using and in the link you have provided are > similar. And still we are unable to import the files size >32 kb. > > As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, > ‘If you are trying to add larger content then you have to "chop" that > by yourself and add it as multivalued. Can be done within a self written > loader. ’ > > I’m a newbie to Solr and I didn’t get what exactly ‘self written loader’ is? > > Could you please provide us sample code, that helps us to go further? > > > > > Thanks & Regards, > Kranthi Kumar.K, > Software Engineer, > Ccube Fintech Global Services Pvt Ltd., > Email/Skype: kranthikuma...@ccubefintech.com > <mailto:kranthikuma...@ccubefintech.com>, > Mobile: +91-8978078449. > > > From: Kranthi Kumar K <mailto:kranthikuma...@ccubefintech.com>> > Sent: Thursday, January 17, 2019 12:43 PM > To: dev@lucene.apache.org <mailto:dev@lucene.apache.org>; > solr-u...@lucene.apache.org <mailto:solr-u...@lucene.apache.org> > Cc: Ananda Babu medida <mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri > <mailto:srinivasare
Re: Solr Size Limitation upto 32 kb limitation
I guess selfwritten loader means that you split up the file in 32 kB chunks or smaller and then posts each of those 32 kB chunks to the multi valued field. > Am 18.01.2019 um 11:51 schrieb Kranthi Kumar K > : > > Hi team, > > Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested > solutions. I’ve tried the suggested one’s and still we are unable to import > files havingsize >32 kb, it is displaying same error. > > Below link has the suggested solutions. Please have a look once. > > http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html > > As per Erick Erickson, I’ve changed the string type to Text type based and > still the issue occurs . > I’ve changed from : > > > > Changed to: > > > > If we do so, it is showing error in the log, please find the error in the > attachment. > > If I change to: > > > > It is not showing any error , but the issue still exists. > > As per Jan Hoydahl, I have gone through the link that you have provided and > checked ‘requestParsers’ tag in solrconfig.xml, > > RequestParsers tag in our application is as follows: > > ‘ multipartUploadLimitInKB="2048000" > formdataUploadLimitInKB="2048" > addHttpRequestToContext="false"/>’ > Request parsers, which we are using and in the link you have provided are > similar. And still we are unable to import the files size >32 kb. > > As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, > ‘If you are trying to add larger content then you have to "chop" that > by yourself and add it as multivalued. Can be done within a self written > loader. ’ > > I’m a newbie to Solr and I didn’t get what exactly ‘self written loader’ is? > > Could you please provide us sample code, that helps us to go further? > > > > > Thanks & Regards, > Kranthi Kumar.K, > Software Engineer, > Ccube Fintech Global Services Pvt Ltd., > Email/Skype: kranthikuma...@ccubefintech.com, > Mobile: +91-8978078449. > > > From: Kranthi Kumar K > Sent: Thursday, January 17, 2019 12:43 PM > To: dev@lucene.apache.org; solr-u...@lucene.apache.org > Cc: Ananda Babu medida ; Srinivasa Reddy > Karri ; Michelle Ngo > > Subject: Re: Solr Size Limitation upto 32 kb limitation > > Hi Team, > > > > Can we have any updates on the below issue? We are awaiting your reply. > > > > Thanks, > > Kranthi kumar.K > > From: Kranthi Kumar K > Sent: Friday, January 4, 2019 5:01:38 PM > To: dev@lucene.apache.org > Cc: Ananda Babu medida; Srinivasa Reddy Karri > Subject: Solr Size Limitation upto 32 kb limitation > > Hi team, > > > > We are currently using Solr 4.2.1 version in our project and everything is > going well. But recently, we are facing an issue with Solr Data Import. It is > not importing the files with size greater than 32766 bytes (i.e, 32 kb) and > showing 2 exceptions: > > > > java.lang.illegalargumentexception > org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception > > > Please find the attached screenshot for reference. > > > > We have searched for solutions in many forums and didn’t find the exact > solution for this issue. Interestingly, we found in the article, by changing > the type of the ‘field’ from sting to ‘text_general’ might solve the issue. > Please have a look in the below forum: > > > > https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t > > > > > Schema.xml: > > Changed from: > > ‘ multiValued="true" />’ > > > > Changed to: > > ‘ multiValued="true" />’ > > > > We have tried it but still it is not importing the files > 32 KB or 32766 > bytes. > > > > Could you please let us know the solution to fix this issue? We’ll be > awaiting your reply. > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Solr Size Limitation upto 32 kb limitation
Thanks @Kranthi Kumar K<mailto:kranthikuma...@ccubefintech.com> for following up From: Kranthi Kumar K Sent: Thursday, 24 January 2019 4:51 PM To: dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida ; Srinivasa Reddy Karri ; Michelle Ngo ; Ravi Vangala ; Suresh Malladi ; Vijay Nandula Subject: RE: Solr Size Limitation upto 32 kb limitation Thank you Bernd Fehling for your suggested solution, I've tried the same by changing the type and added multivalued to true in Schema.xml file i.e, change from: Changed to: After changing it also still we are unable to import the files size > 32 kb. please find the solution suggested by Bernd in the below url: http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html Bernd Fehling, could you please suggest another alternative solution to resolve our issue, which would help us alot? Please let me know for any questions. [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Friday, January 18, 2019 4:22 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>>; Ravi Vangala mailto:ravi.vang...@ccubefintech.com>> Subject: RE: Solr Size Limitation upto 32 kb limitation Hi team, Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I've tried the suggested one's and still we are unable to import files havingsize >32 kb, it is displaying same error. Below link has the suggested solutions. Please have a look once. http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html 1. As per Erick Erickson, I've changed the string type to Text type based and still the issue occurs . I've changed from : Changed to: If we do so, it is showing error in the log, please find the error in the attachment. If I change to: It is not showing any error , but the issue still exists. 1. As per Jan Hoydahl, I have gone through the link that you have provided and checked 'requestParsers' tag in solrconfig.xml, RequestParsers tag in our application is as follows: '' Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb. 1. As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, 'If you are trying to add larger content then you have to "chop" that by yourself and add it as multivalued. Can be done within a self written loader. ' I'm a newbie to Solr and I didn't get what exactly 'self written loader' is? Could you please provide us sample code, that helps us to go further? [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K mailto:kranthikuma...@ccubefintech.com>> Sent: Thursday, January 17, 2019 12:43 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>> Subject: Re: Solr Size Limitation upto 32 kb limitation Hi Team, Can we have any updates on the below issue? We are awaiting your reply. Thanks, Kranthi kumar.K From: Kranthi Kumar K Sent: Friday, January 4, 2019 5:01:38 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org> Cc: Ananda Babu medida; Srinivasa Reddy Karri Subject: Solr Size Limitation upto 32 kb limitation Hi team, We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions: 1. java.lang.illegalargumentexception 2. org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception Please find the attached screenshot for reference. We have searched for solutions in many forums and didn't find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the 'field' from sting to 'text_general' might solve the issue. Please have a look in the below forum: https://stackov
RE: Solr Size Limitation upto 32 kb limitation
Thank you Bernd Fehling for your suggested solution, I've tried the same by changing the type and added multivalued to true in Schema.xml file i.e, change from: Changed to: After changing it also still we are unable to import the files size > 32 kb. please find the solution suggested by Bernd in the below url: http://lucene.472066.n3.nabble.com/Re-Solr-Size-Limitation-upto-32-kb-limitation-td4421569.html Bernd Fehling, could you please suggest another alternative solution to resolve our issue, which would help us alot? Please let me know for any questions. [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Friday, January 18, 2019 4:22 PM To: dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida ; Srinivasa Reddy Karri ; Michelle Ngo ; Ravi Vangala Subject: RE: Solr Size Limitation upto 32 kb limitation Hi team, Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I've tried the suggested one's and still we are unable to import files havingsize >32 kb, it is displaying same error. Below link has the suggested solutions. Please have a look once. http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html 1. As per Erick Erickson, I've changed the string type to Text type based and still the issue occurs . I've changed from : Changed to: If we do so, it is showing error in the log, please find the error in the attachment. If I change to: It is not showing any error , but the issue still exists. 1. As per Jan Hoydahl, I have gone through the link that you have provided and checked 'requestParsers' tag in solrconfig.xml, RequestParsers tag in our application is as follows: '' Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb. 1. As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, 'If you are trying to add larger content then you have to "chop" that by yourself and add it as multivalued. Can be done within a self written loader. ' I'm a newbie to Solr and I didn't get what exactly 'self written loader' is? Could you please provide us sample code, that helps us to go further? [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K mailto:kranthikuma...@ccubefintech.com>> Sent: Thursday, January 17, 2019 12:43 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org>; solr-u...@lucene.apache.org<mailto:solr-u...@lucene.apache.org> Cc: Ananda Babu medida mailto:anandababu.med...@ccubefintech.com>>; Srinivasa Reddy Karri mailto:srinivasareddy.ka...@ccubefintech.com>>; Michelle Ngo mailto:michelle@ccube.com.au>> Subject: Re: Solr Size Limitation upto 32 kb limitation Hi Team, Can we have any updates on the below issue? We are awaiting your reply. Thanks, Kranthi kumar.K From: Kranthi Kumar K Sent: Friday, January 4, 2019 5:01:38 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org> Cc: Ananda Babu medida; Srinivasa Reddy Karri Subject: Solr Size Limitation upto 32 kb limitation Hi team, We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions: 1. java.lang.illegalargumentexception 2. org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception Please find the attached screenshot for reference. We have searched for solutions in many forums and didn't find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the 'field' from sting to 'text_general' might solve the issue. Please have a look in the below forum: https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t Schema.xml: Changed from: '' Changed to: '' We have tried it but still it is not importing the files > 32 KB or 32766 bytes. Could you please let us know the solution to fix this issue? We'll be awaiting your reply.
RE: Solr Size Limitation upto 32 kb limitation
Hi team, Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I've tried the suggested one's and still we are unable to import files havingsize >32 kb, it is displaying same error. Below link has the suggested solutions. Please have a look once. http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html 1. As per Erick Erickson, I've changed the string type to Text type based and still the issue occurs . I've changed from : Changed to: If we do so, it is showing error in the log, please find the error in the attachment. If I change to: It is not showing any error , but the issue still exists. 1. As per Jan Hoydahl, I have gone through the link that you have provided and checked 'requestParsers' tag in solrconfig.xml, RequestParsers tag in our application is as follows: '' Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb. 1. As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as, 'If you are trying to add larger content then you have to "chop" that by yourself and add it as multivalued. Can be done within a self written loader. ' I'm a newbie to Solr and I didn't get what exactly 'self written loader' is? Could you please provide us sample code, that helps us to go further? [image001] Thanks & Regards, Kranthi Kumar.K, Software Engineer, Ccube Fintech Global Services Pvt Ltd., Email/Skype: kranthikuma...@ccubefintech.com<mailto:kranthikuma...@ccubefintech.com>, Mobile: +91-8978078449. From: Kranthi Kumar K Sent: Thursday, January 17, 2019 12:43 PM To: dev@lucene.apache.org; solr-u...@lucene.apache.org Cc: Ananda Babu medida ; Srinivasa Reddy Karri ; Michelle Ngo Subject: Re: Solr Size Limitation upto 32 kb limitation Hi Team, Can we have any updates on the below issue? We are awaiting your reply. Thanks, Kranthi kumar.K From: Kranthi Kumar K Sent: Friday, January 4, 2019 5:01:38 PM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org> Cc: Ananda Babu medida; Srinivasa Reddy Karri Subject: Solr Size Limitation upto 32 kb limitation Hi team, We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions: 1. java.lang.illegalargumentexception 2. org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception Please find the attached screenshot for reference. We have searched for solutions in many forums and didn't find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the 'field' from sting to 'text_general' might solve the issue. Please have a look in the below forum: https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t Schema.xml: Changed from: '' Changed to: '' We have tried it but still it is not importing the files > 32 KB or 32766 bytes. Could you please let us know the solution to fix this issue? We'll be awaiting your reply. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr Size Limitation upto 32 kb limitation
On Thu, 2019-01-17 at 07:12 +, Kranthi Kumar K wrote: > Can we have any updates on the below issue? We are awaiting your > reply. Erick Erickson posted a reply 5 hours after you asked your original question on the developer mailinglist. I suggest you write a follow-up to his answer if it did not solve your problem. Cc: to Kranthi as he might have mailinglist-related delivery problems. - Toke Eskildsen, royal Danish Library - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr Size Limitation upto 32 kb limitation
Hi Team, Can we have any updates on the below issue? We are awaiting your reply. Thanks, Kranthi kumar.K From: Kranthi Kumar K Sent: Friday, January 4, 2019 5:01:38 PM To: dev@lucene.apache.org Cc: Ananda Babu medida; Srinivasa Reddy Karri Subject: Solr Size Limitation upto 32 kb limitation Hi team, We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions: 1. java.lang.illegalargumentexception 2. org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception Please find the attached screenshot for reference. We have searched for solutions in many forums and didn’t find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the ‘field’ from sting to ‘text_general’ might solve the issue. Please have a look in the below forum: https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t Schema.xml: Changed from: ‘’ Changed to: ‘’ We have tried it but still it is not importing the files > 32 KB or 32766 bytes. Could you please let us know the solution to fix this issue? We’ll be awaiting your reply.
Re: Solr Size Limitation upto 32 kb limitation
First off, the field in question is "FileContent", why do you think the filed "text" is the problem? Try switching FileContent to a text-based type. If that's not the case, depending on the tokenizer and the input you _still_ may have an immense term even if you have a text-based field. For example, the data could be something like base64 encoded, which has no spaces and you are using a tokenizer that only breaks on whitespace. You simply have got to look at the input data to make sense of the problem Best, Erick On Fri, Jan 4, 2019 at 3:32 AM Kranthi Kumar K wrote: > > Hi team, > > > > We are currently using Solr 4.2.1 version in our project and everything is > going well. But recently, we are facing an issue with Solr Data Import. It is > not importing the files with size greater than 32766 bytes (i.e, 32 kb) and > showing 2 exceptions: > > > > java.lang.illegalargumentexception > org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception > > > > Please find the attached screenshot for reference. > > > > We have searched for solutions in many forums and didn’t find the exact > solution for this issue. Interestingly, we found in the article, by changing > the type of the ‘field’ from sting to ‘text_general’ might solve the issue. > Please have a look in the below forum: > > > > https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t > > > > Schema.xml: > > Changed from: > > ‘ multiValued="true" />’ > > > > Changed to: > > ‘ multiValued="true" />’ > > > > We have tried it but still it is not importing the files > 32 KB or 32766 > bytes. > > > > Could you please let us know the solution to fix this issue? We’ll be > awaiting your reply. > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
