RE: Highlighter throwing InvalidTokenOffsetsException for field with large number of synonyms
Hi Rick, Thanks for your response. The reason that we do it like this is that the localities are also part of another indexed field that contains the entire address. We actually do the search over that field, and we are only using the highlighting on the problematic field so that we can tell which parts of the address that we matched to. We never search for wildcards like "*cannum*". As an example, we might have an address that we index which is "19 some st cannum vic 3456". When we index the address, we actually index the text "19 some st lcx__balmoral__cannum__clear_lake__lower_norton vic 3456" into a Solr field that has our custom synonym filter. This then causes the synonyms for the locality "cannum" to be generated, and if we search for "19 some st balmoral" we will still get a match on the locality component of the address. Using this method, the searching for addresses is working fine. We have a requirement once we have a match to know which part of the address that we matched to, which is where the highlighting comes in. By loading just the locality part of the address into a separate field and applying the same synonym filter, through the highlighting we can see if we get a hit on the locality. We do this with the other components of the address, like the number, the street name, the street type, the post code etc. so that we can return to the caller what bits of their input matched to the address we are returning. I could load them as a multi-valued field for just the highlighting, but that means I need to extract them in a different format to what I am using for the whole address which I would like to avoid if possible. We are loading these addresses from a database table using the data import handler. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick & Alessandro, I have solved my problem by re-ordering the data in the SQL query. I don't know why it works but it does. I can consistently re-produce the problem without changing anything else except the database table. As our Solr build is scripted and we always build a new Solr server from scratch, I'm pretty confident that the defaults haven't changed between test runs as when we create the Solr index, Solr doesn't know what order the data in the database table is in. I did try removing the geo location field to see if that made a difference, and it didn't. Due to project commitments, I don't have any time to investigate this further at the moment. When/if things quiet down I may see if I can reproduce the problem with a smaller number of records loaded from a flat file to make it easier to share a project that shows the problem occurring. Thanks again for all of your assistance and suggestions. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Getting the error - The field '*********' does not support spatial filtering
Hi Aakanksha,
We use the following for geo queries which works for us:
/solr/core/select?defType=edismax=on=0=0=json=true=on=*=%7B!geofilt%7D=-6.08165,145.8612430=10=geoLocation=geodist()%20asc=10=*,score,distance:geodist()
This gives us the results closest to the provided point in order of their
distance from the point.
Our field definition is:
echo "$(date) Creating geoLocation field"
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"geoLocation",
"type":"location",
"stored":true,
"indexed":true
}
}' http://localhost:8983/solr/core/schema
We are running Solr 7.1.0.
Hope this helps.
Regards,
David
From: Aakanksha Gupta [mailto:aakankshagupta2...@gmail.com]
Sent: Monday, 19 February 2018 12:27 AM
To: solr-user@lucene.apache.org
Subject: Getting the error - The field '*' does not support spatial
filtering
Hi all,
I'm a newbie to Solr. I'm trying to use it for GeoSpatial Search and I'm facing
an issue while using it. I've tried using the new 'location' field
type() as well as the deprecated solr.LatLonType fieldtype, but I
always get the error:
org.apache.solr.common.SolrException: The field latlong does not support
spatial filtering
Here's a snippet of my field definition in schema.xml in the conf folder of my
core:
And here are the field type definitions:
Here's the Query I'm running:
http://localhost:8983/solr/geo2/select?wt=json=:={!geofilt
sfield=latlong}=-6.08165,145.8612430=100
http://localhost:8983/solr/geo2/select/?q=*:*={!geofilt}=latlong2=-6.08165,145.8612430=100=json
And here's the Java snippet I'm using to insert data:
String urlString = "http://localhost:8983/solr/geo2;;
SolrClient solr = new HttpSolrClient.Builder(urlString).build();
SolrInputDocument document = new SolrInputDocument();
document.addField("id", UUID.randomUUID().toString());
document.addField("driverid", "1");
document.addField("latlong", "-6.081689,145.391881");
document.addField("time", "7:01:17");
document.addField("timestamp", Long.valueOf("1518908477190"));
document.addField("latlong2", "-6.081689,145.391881");
document.addField("location_0_coordinate", Double.valueOf(-6.081689));
document.addField("location_1_coordinate", Double.valueOf(145.391881));
UpdateResponse response = solr.add(document);
solr.commit();
response.getQTime();
I've attached my schema.xml file herewith. Can someone let me know what I'm
doing wrong?
David Howe
Java Domain Architect
Postal Systems
Australia Post
Level 16, 111 Bourke Street Melbourne VIC 3000
T 0391067904
M 0424036591
E david.h...@auspost.com.au
[Australia Post website]
[StarTrack website] [Like us on Facebook]
[Connect with us on LinkedIn]
Australia Post is committed to providing our customers with excellent service.
If we can assist you in any way please telephone 13 13 18 or visit our website.
The information contained in this email communication may be proprietary,
confidential or legally professionally privileged. It is intended exclusively
for the individual or entity to which it is addressed. You should only read,
disclose, re-transmit, copy, distribute, act in reliance on or commercialise
the information if you are authorised to do so. Australia Post does not
represent, warrant or guarantee that the integrity of this email communication
has been maintained nor that the communication is free of errors, virus or
interference.
If you are not the addressee or intended recipient please notify us by replying
direct to the sender and then destroy any electronic or paper copy of this
message. Any views expressed in this email communication are taken to be those
of the individual sender, except where the sender specifically attributes those
views to Australia Post and is authorised to do so.
Please consider the environment before printing this email.
RE: Solr Plugins Documentation
You will need to use your favourite java tooling to take the code that you have written and package it as a jar file. In my case, we use maven so I have set my custom extensions up as a maven project, and in my POM file (which tells maven what dependencies your project has), I declare: org.apache.solr solr-core 6.4.0 provided This puts the Solr core classes on my classpath when I am compiling, but does not package them up when I am building my jar file (scope=provided). Once you have written your extension class, running mvn clean install will create your jar file for you. You then need to copy that jar file into the solr/server/solr/lib directory so that Solr can find it and put it on the classpath. You should then be able to reference your custom extension in the Solr config. See https://maven.apache.org/ for details on maven. See https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html for the basics on jar files. If you use a tool like IntelliJ or Exclipse, they have maven and jar file support built in. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick, I'm 99% sure that I haven't changed the field types between the two snapshots as all of my test runs are completely scripted and build a new Solr server from scratch (both the virtual machine and the Solr software). I can diff the scripts between two runs to make sure I haven't accidentally changed anything, and I have done this. The only difference is that I added docValues=false to all of the fields that are indexed=false and stored=true in the run that is smaller. I had tested this previously with the data in the order that makes the index larger and it only made a minor difference (see one of my previous posts). Unfortunately, I hadn't added the change to log the file sizes when I did that run, but it definitely didn't fix the problem. I need to try and get my project back on track now, so I will concentrate on the "fix" that I have and perhaps re-run some other scenarios when I have more time. Thanks again for your help. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick,
Thinking some more about the differences between the two sort orders has
suggested another possibility. We also have a geo spatial field defined in the
index:
echo "$(date) Creating geoLocation field"
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"geoLocation",
"type":"location",
"stored":true,
"indexed":true
}
}' http://localhost:8983/solr/address/schema
One of the differences between the two sort orders is that when the data is
sorted by locality and post code, it means that addresses that are close to
each other will be sorted together as both locality and postcode have
geographic meaning. So when they are indexed, they will be indexed in groups
of addresses that are quite near to each other.
When the data is sorted by DPID, the order is near random as the dpid has no
meaning at all, so the geo location sequence should be random as well.
I don't have time to test this at the moment, as I need to get my project back
on track after chasing this performance issue but it might ring a bell with
somebody.
Regards,
David
David Howe
Java Domain Architect
Postal Systems
Level 16, 111 Bourke Street Melbourne VIC 3000
T 0391067904
M 0424036591
E david.h...@auspost.com.au
W auspost.com.au
W startrack.com.au
Australia Post is committed to providing our customers with excellent service.
If we can assist you in any way please telephone 13 13 18 or visit our website.
The information contained in this email communication may be proprietary,
confidential or legally professionally privileged. It is intended exclusively
for the individual or entity to which it is addressed. You should only read,
disclose, re-transmit, copy, distribute, act in reliance on or commercialise
the information if you are authorised to do so. Australia Post does not
represent, warrant or guarantee that the integrity of this email communication
has been maintained nor that the communication is free of errors, virus or
interference.
If you are not the addressee or intended recipient please notify us by replying
direct to the sender and then destroy any electronic or paper copy of this
message. Any views expressed in this email communication are taken to be those
of the individual sender, except where the sender specifically attributes those
views to Australia Post and is authorised to do so.
Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick,
Below is the file listing for when the index is loaded with the table ordered
in a way that produces the smaller index.
I have checked the console, and we have no deleted docs and we have the same
number of docs in the index as there are rows in the staging table that we load
from. I would be surprised if this wasn't the case as we use the primary key
from the staging table as the id in Solr, so it is pretty much guaranteed to be
unique. The primary key in the staging table is a NUMBER(10, 0) column which
contains the row number in Oracle, so it starts from 1 and goes up to
14,061,990. We load the index in row number order.
When we get the larger sized index, the table is sequenced by a field named
DPID which is a NUMBER(10, 0) in Oracle. The corresponding Solr definition for
that field is:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"dpid",
"type":"pint",
"stored":true,
"indexed": true
}
}' http://localhost:8983/solr/address/schema
When we get the smaller sized index, the table is sequenced by locality
(VARCHAR2(80)) and then postcode (VARCHAR2(4)). The corresponding Solr
definition for these fields is:
echo "$(date) Creating locality field"
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"locality",
"type":"locality",
"stored":true,
"indexed":true
}
}' http://localhost:8983/solr/address/schema
echo "$(date) Creating postcode field"
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"postcode",
"type":"pint",
"stored":true,
"indexed":true
}
}' http://localhost:8983/solr/address/schema
Not sure if this helps or not.
Regards,
David
total 5300812
-rw-r--r-- 1 solr solr97 Feb 16 04:12 _14o.dii
-rw-r--r-- 1 solr solr 45400325 Feb 16 04:12 _14o.dim
-rw-r--r-- 1 solr solr 221114041 Feb 16 04:10 _14o.fdt
-rw-r--r-- 1 solr solr286434 Feb 16 04:10 _14o.fdx
-rw-r--r-- 1 solr solr 6370 Feb 16 04:12 _14o.fnm
-rw-r--r-- 1 solr solr 17379224 Feb 16 04:12 _14o.nvd
-rw-r--r-- 1 solr solr 463 Feb 16 04:12 _14o.nvm
-rw-r--r-- 1 solr solr 620 Feb 16 04:12 _14o.si
-rw-r--r-- 1 solr solr 147867580 Feb 16 04:11 _14o_Lucene50_0.doc
-rw-r--r-- 1 solr solr 111291706 Feb 16 04:11 _14o_Lucene50_0.pos
-rw-r--r-- 1 solr solr 18793856 Feb 16 04:11 _14o_Lucene50_0.tim
-rw-r--r-- 1 solr solr360329 Feb 16 04:11 _14o_Lucene50_0.tip
-rw-r--r-- 1 solr solr 91972283 Feb 16 04:12 _14o_Lucene70_0.dvd
-rw-r--r-- 1 solr solr 4173 Feb 16 04:12 _14o_Lucene70_0.dvm
-rw-r--r-- 1 solr solr 405 Feb 16 04:20 _16l.cfe
-rw-r--r-- 1 solr solr 10956277 Feb 16 04:20 _16l.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:20 _16l.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:30 _18t.cfe
-rw-r--r-- 1 solr solr 11619394 Feb 16 04:30 _18t.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:30 _18t.si
-rw-r--r-- 1 solr solr97 Feb 16 04:34 _19e.dii
-rw-r--r-- 1 solr solr 39424990 Feb 16 04:34 _19e.dim
-rw-r--r-- 1 solr solr 188005197 Feb 16 04:33 _19e.fdt
-rw-r--r-- 1 solr solr249160 Feb 16 04:33 _19e.fdx
-rw-r--r-- 1 solr solr 6370 Feb 16 04:34 _19e.fnm
-rw-r--r-- 1 solr solr 14660427 Feb 16 04:34 _19e.nvd
-rw-r--r-- 1 solr solr 463 Feb 16 04:34 _19e.nvm
-rw-r--r-- 1 solr solr 620 Feb 16 04:34 _19e.si
-rw-r--r-- 1 solr solr 131101691 Feb 16 04:33 _19e_Lucene50_0.doc
-rw-r--r-- 1 solr solr 97734855 Feb 16 04:33 _19e_Lucene50_0.pos
-rw-r--r-- 1 solr solr 16502289 Feb 16 04:33 _19e_Lucene50_0.tim
-rw-r--r-- 1 solr solr320224 Feb 16 04:33 _19e_Lucene50_0.tip
-rw-r--r-- 1 solr solr 78801516 Feb 16 04:34 _19e_Lucene70_0.dvd
-rw-r--r-- 1 solr solr 2097 Feb 16 04:34 _19e_Lucene70_0.dvm
-rw-r--r-- 1 solr solr 405 Feb 16 04:35 _19y.cfe
-rw-r--r-- 1 solr solr 78051374 Feb 16 04:35 _19y.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:35 _19y.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:37 _1ai.cfe
-rw-r--r-- 1 solr solr 53311170 Feb 16 04:37 _1ai.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:37 _1ai.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:40 _1b2.cfe
-rw-r--r-- 1 solr solr 70986259 Feb 16 04:40 _1b2.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:40 _1b2.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:41 _1bc.cfe
-rw-r--r-- 1 solr solr 10338200 Feb 16 04:41 _1bc.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:41 _1bc.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:42 _1bm.cfe
-rw-r--r-- 1 solr solr 68074070 Feb 16 04:42 _1bm.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:42 _1bm.si
-rw-r--r-- 1 solr solr 405 Feb 16 04:45 _1c5.cfe
-rw-r--r-- 1 solr solr 67766868 Feb 16 04:45 _1c5.cfs
-rw-r--r-- 1 solr solr 455 Feb 16 04:45 _1c5.si
-rw-r--r-- 1 solr solr91 Feb 16 04:45 _1c6.dii
-rw-r--r-- 1 solr solr666032 Feb 16 04:45 _1c6.dim
-rw-r--r-- 1 solr solr 2515129 Feb 16 04:45 _1c6.fdt
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Alessandro, There are 14,061,990 records in the staging table and that is how many documents that we end up with in Solr. I would be surprised if we have a problem with the id, as we use the primary key of the table as the id in Solr so it must be unique. The primary key of the staging table is a NUMBER(10, 0) in Oracle, and we set it to the row number when we are populating the table. So the id's will start at 1 and go up to 14,061,990 and we load the records in id order. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Emir, We have no copy field definitions. To keep things simple, we have a one to one mapping between the columns in our staging table and the fields in our Solr index. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick, I have the full dump of the Solr index file sizes as well if that is of any help. I have attached it below this message. We don't have any deleted docs in our index, as we always build it from a brand new virtual machine with a brand new installation of Solr. The ordering is definitely making a difference, as I can run the same indexing configuration over a table with the same data just in different orders and it produces these vastly different results. I have been chasing this for a couple of weeks trying to work out what the difference is when we just add one extra field. The difference that I have found is that the extra field causes the staging table population query to be optimised differently and to select the records in a different sequence. When I force the records back to their original sequence, the index goes back to being small again. I'm currently re-building my staging data to try and get it into the same order as before and including the extra field. I will post the file sizes again when I have that result. Regards, David total 14600404 -rw-r--r-- 1 solr solr 97 Feb 14 01:34 _7l.dii -rw-r--r-- 1 solr solr 83831801 Feb 14 01:34 _7l.dim -rw-r--r-- 1 solr solr 1431645451 Feb 14 01:33 _7l.fdt -rw-r--r-- 1 solr solr 381994 Feb 14 01:33 _7l.fdx -rw-r--r-- 1 solr solr 6370 Feb 14 01:34 _7l.fnm -rw-r--r-- 1 solr solr 29353048 Feb 14 01:34 _7l.nvd -rw-r--r-- 1 solr solr463 Feb 14 01:34 _7l.nvm -rw-r--r-- 1 solr solr606 Feb 14 01:34 _7l.si -rw-r--r-- 1 solr solr 734701117 Feb 14 01:34 _7l_Lucene50_0.doc -rw-r--r-- 1 solr solr 335043096 Feb 14 01:34 _7l_Lucene50_0.pos -rw-r--r-- 1 solr solr 34248274 Feb 14 01:34 _7l_Lucene50_0.tim -rw-r--r-- 1 solr solr 624945 Feb 14 01:34 _7l_Lucene50_0.tip -rw-r--r-- 1 solr solr 165958502 Feb 14 01:34 _7l_Lucene70_0.dvd -rw-r--r-- 1 solr solr 2581 Feb 14 01:34 _7l_Lucene70_0.dvm -rw-r--r-- 1 solr solr405 Feb 14 01:46 _9p.cfe -rw-r--r-- 1 solr solr 38776749 Feb 14 01:46 _9p.cfs -rw-r--r-- 1 solr solr452 Feb 14 01:46 _9p.si -rw-r--r-- 1 solr solr 97 Feb 14 02:07 _cm.dii -rw-r--r-- 1 solr solr 83111509 Feb 14 02:07 _cm.dim -rw-r--r-- 1 solr solr 1419981112 Feb 14 02:02 _cm.fdt -rw-r--r-- 1 solr solr 379544 Feb 14 02:02 _cm.fdx -rw-r--r-- 1 solr solr 6370 Feb 14 02:07 _cm.fnm -rw-r--r-- 1 solr solr 29049434 Feb 14 02:07 _cm.nvd -rw-r--r-- 1 solr solr463 Feb 14 02:07 _cm.nvm -rw-r--r-- 1 solr solr606 Feb 14 02:07 _cm.si -rw-r--r-- 1 solr solr 728509370 Feb 14 02:07 _cm_Lucene50_0.doc -rw-r--r-- 1 solr solr 332343997 Feb 14 02:07 _cm_Lucene50_0.pos -rw-r--r-- 1 solr solr 34361884 Feb 14 02:07 _cm_Lucene50_0.tim -rw-r--r-- 1 solr solr 658404 Feb 14 02:07 _cm_Lucene50_0.tip -rw-r--r-- 1 solr solr 164612509 Feb 14 02:07 _cm_Lucene70_0.dvd -rw-r--r-- 1 solr solr 2581 Feb 14 02:07 _cm_Lucene70_0.dvm -rw-r--r-- 1 solr solr405 Feb 14 02:09 _fb.cfe -rw-r--r-- 1 solr solr 44333425 Feb 14 02:09 _fb.cfs -rw-r--r-- 1 solr solr452 Feb 14 02:09 _fb.si -rw-r--r-- 1 solr solr 97 Feb 14 02:24 _h2.dii -rw-r--r-- 1 solr solr 77079684 Feb 14 02:24 _h2.dim -rw-r--r-- 1 solr solr 1304390074 Feb 14 02:22 _h2.fdt -rw-r--r-- 1 solr solr 347494 Feb 14 02:22 _h2.fdx -rw-r--r-- 1 solr solr 6370 Feb 14 02:24 _h2.fnm -rw-r--r-- 1 solr solr 26756876 Feb 14 02:24 _h2.nvd -rw-r--r-- 1 solr solr463 Feb 14 02:24 _h2.nvm -rw-r--r-- 1 solr solr606 Feb 14 02:24 _h2.si -rw-r--r-- 1 solr solr 669875920 Feb 14 02:24 _h2_Lucene50_0.doc -rw-r--r-- 1 solr solr 305954906 Feb 14 02:24 _h2_Lucene50_0.pos -rw-r--r-- 1 solr solr 32019733 Feb 14 02:24 _h2_Lucene50_0.tim -rw-r--r-- 1 solr solr 619562 Feb 14 02:24 _h2_Lucene50_0.tip -rw-r--r-- 1 solr solr 151772808 Feb 14 02:24 _h2_Lucene70_0.dvd -rw-r--r-- 1 solr solr 2497 Feb 14 02:24 _h2_Lucene70_0.dvm -rw-r--r-- 1 solr solr405 Feb 14 02:45 _mx.cfe -rw-r--r-- 1 solr solr 277937779 Feb 14 02:45 _mx.cfs -rw-r--r-- 1 solr solr452 Feb 14 02:45 _mx.si -rw-r--r-- 1 solr solr 97 Feb 14 02:47 _n9.dii -rw-r--r-- 1 solr solr 82335510 Feb 14 02:47 _n9.dim -rw-r--r-- 1 solr solr 1400595065 Feb 14 02:46 _n9.fdt -rw-r--r-- 1 solr solr 374259 Feb 14 02:46 _n9.fdx -rw-r--r-- 1 solr solr 6370 Feb 14 02:47 _n9.fnm -rw-r--r-- 1 solr solr 28775974 Feb 14 02:47 _n9.nvd -rw-r--r-- 1 solr solr463 Feb 14 02:47 _n9.nvm -rw-r--r-- 1 solr solr606 Feb 14 02:47 _n9.si -rw-r--r-- 1 solr solr 719183309 Feb 14 02:46 _n9_Lucene50_0.doc -rw-r--r-- 1 solr solr 328214265 Feb 14 02:46 _n9_Lucene50_0.pos -rw-r--r-- 1 solr solr 34098919 Feb 14 02:46 _n9_Lucene50_0.tim -rw-r--r-- 1 solr solr 654313 Feb 14 02:46 _n9_Lucene50_0.tip -rw-r--r-- 1 solr solr 163220960 Feb 14 02:46 _n9_Lucene70_0.dvd -rw-r--r-- 1 solr solr 2560 Feb 14 02:46 _n9_Lucene70_0.dvm -rw-r--r-- 1 solr solr405 Feb 14 02:52 _ns.cfe
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Alessandro, Some interesting testing today that seems to have gotten me closer to what the issue is. When I run the version of the index that is working correctly against my database table that has the extra field in it, the index suddenly increases in size. This is even though the data importer is running the same SELECT as before (which doesn't include the extra column) and loads the same number of rows. After scratching my head for a bit and browsing through both versions of the table I am loading from (with and without the extra field), I noticed that the natural ordering of the tables is different. These tables are "staging" tables that I populate with another set of queries and inserts to get the data into a format that is easy to ingest into Solr. When I add the extra field to these queries, it changes the Oracle query plan as the field is contained in a different table that I need to join to. As I don't specify an "ORDER BY" on the query (as I didn't think it would make a difference and would slow the query down), Oracle is free to chose how it orders the result set. Adding the extra field changes that natural ordering, which affects the order things go into my staging table. As I don't specify an "ORDER BY" when I select things out of the staging table, my data in the scenario that is working is being loaded in a different order to the scenario which doesn't work. I am currently running full loads to verify this under each scenario, as I have now forced the data in the scenario that doesn't work to be in the same order as the scenario that does. Will see how this load goes overnight. This leads to the question of what difference does it make to Solr what order I load the data in? I also noticed that the .cfs file is quite large in the second scenario, even though this is supposed to be disabled by default in Solr. I checked my Solr config and there is no override of the default. In answer to your questions: 1) same number of documents - YES ~14,000,000 documents 2) identical documents ( + 1 new field each not indexed) - YES, the second scenario has one extra field that is stored but not indexed 3) same number of deleted documents - YES, there are zero deleted documents in both scenarios 4) they both were born from scratch ( an empty index) - YES, both start from a brand new virtual server with a brand new installation of Solr I am using the default auto commit, which I think is 15000. Thanks again for your assistance. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
I have re-run both scenarios and captured the total size of each type of index file. The MB (1) column is for the baseline scenario which has the smaller index and acceptable performance. The MB(2) column is after I have added the extra field to the index. Ext MB (1) MB (2) .cfe0.000.01 .cfs335.01 3612.09 .dii0.000.00 .dim324.38 319.07 .doc1094.68 2767.53 .dvd1211.84 625.44 .dvm0.140.08 .fdt1633.21 5387.92 .fdx2.121.44 .fnm0.110.12 .loc0.000.00 .nvd127.84 110.67 .nvm0.010.01 .pos809.23 1272.70 .si 0.020.03 .tim137.94 156.82 .tip2.523.04 Total 5679.06 14256.98 David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au -Original Message- From: Howe, David [mailto:david.h...@auspost.com.au] Sent: Wednesday, 14 February 2018 12:49 PM To: solr-user@lucene.apache.org Subject: RE: Index size increases disproportionately to size of added field when indexed=false I have set docValues=false on all of the string fields in our index that have indexed=false and stored=true. This gave a small improvement in the index size from 13.3GB to 12.82GB. I have also tried running an optimize, which then reduced the index to 12.6GB. Next step is to dump the sizes of the Solr index files for the index version that is the correct size and the version that has the large size. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au -Original Message- From: Howe, David [mailto:david.h...@auspost.com.au] Sent: Wednesday, 14 February 2018 7:26 AM To: solr-user@lucene.apache.org Subject: RE: Index size increases disproportionately to size of added field when indexed=false Thanks Hoss. I will try setting docValues to false, as we only ever want to be able to retrieve the value of this field. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email. Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email. Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email
RE: Index size increases disproportionately to size of added field when indexed=false
I have set docValues=false on all of the string fields in our index that have indexed=false and stored=true. This gave a small improvement in the index size from 13.3GB to 12.82GB. I have also tried running an optimize, which then reduced the index to 12.6GB. Next step is to dump the sizes of the Solr index files for the index version that is the correct size and the version that has the large size. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au -Original Message- From: Howe, David [mailto:david.h...@auspost.com.au] Sent: Wednesday, 14 February 2018 7:26 AM To: solr-user@lucene.apache.org Subject: RE: Index size increases disproportionately to size of added field when indexed=false Thanks Hoss. I will try setting docValues to false, as we only ever want to be able to retrieve the value of this field. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email. Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Thanks Hoss. I will try setting docValues to false, as we only ever want to be able to retrieve the value of this field. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Erick, Thanks for responding. You are correct that we don't have any deleted docs. When we want to re-index (once a fortnight), we build a brand new installation of Solr from scratch and re-import the new data into an empty index. I will try setting docValues to false and see if that makes a difference. It sounds like we shouldn't have it on anyway, as we only ever want to be able to retrieve this field. In what situation would it make sense to have indexed=false and docValues=true? I will re-index and get a sizing for all of the different file extensions both with and without the problematic field. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Alessandro, The docker image is like a disk image of the entire server, so it includes the operating system, the Solr installation and the data. Because we run in the cloud and our index isn't that big, this is an easy and fast way for us to scale our Solr cluster without having to configure Solr clusters, replication etc. When we create a new server and "run" the docker image, the server comes up all ready to go, with Solr installed and the data already in the index. I will checkout the different file extensions and how much space they are using. Thanks, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
RE: Index size increases disproportionately to size of added field when indexed=false
Hi Alessanro, Thanks for responding. We rebuild the index every time starting from a fresh installation of Solr. Because we are running at AWS, we have automated our deployment so we start with the base docker image, configure Solr and then import our data every time the data changes (it only changes once a fortnight). Once the import finishes we save the docker image in the AWS docker repository. We then build our cluster using that image as the base. So we never re-index an existing index, we just build another one from scratch. We haven't configured anything special for segments and merges. When I look in the console, the index is shown as being optimized. There doesn't seem to be an option in the console anymore to optimize an index. If I have only ever inserted new documents, should I need to optimize? I will try an optimize when I am back in the office tomorrow. Regards, David David Howe Java Domain Architect Postal Systems Level 16, 111 Bourke Street Melbourne VIC 3000 T 0391067904 M 0424036591 E david.h...@auspost.com.au W auspost.com.au W startrack.com.au Australia Post is committed to providing our customers with excellent service. If we can assist you in any way please telephone 13 13 18 or visit our website. The information contained in this email communication may be proprietary, confidential or legally professionally privileged. It is intended exclusively for the individual or entity to which it is addressed. You should only read, disclose, re-transmit, copy, distribute, act in reliance on or commercialise the information if you are authorised to do so. Australia Post does not represent, warrant or guarantee that the integrity of this email communication has been maintained nor that the communication is free of errors, virus or interference. If you are not the addressee or intended recipient please notify us by replying direct to the sender and then destroy any electronic or paper copy of this message. Any views expressed in this email communication are taken to be those of the individual sender, except where the sender specifically attributes those views to Australia Post and is authorised to do so. Please consider the environment before printing this email.
Index size increases disproportionately to size of added field when indexed=false
Hi,
We are using Solr 7.1.0 to index a database of addresses. We have found that
our index size increases massively when we add one extra field to the index,
even though that field is stored and not indexed, and doesn’t contain a lot of
data. When this occurs, we also observe a significant increase in response
times and CPU usage on the Solr server.
When we run an index load without the problematic field present, the Solr index
size is 5.5GB. When we add the field into the index, the size grows to 13.3GB.
The field itself is a maximum of 46 characters in length and on average is 19
characters. We have ~14,000,000 rows in total to index of which only ~200,000
have this field present at all (i.e. not null in database). Given that we
don’t want to index the field, only store it I would have thought (perhaps
naively) that the storage increase would be approximately 200,000 * 19 = 3.8M
bytes = 3.6MB rather than the 7.5GB we are seeing.
Some further background on what we are doing:
- We are using the Solr 7.1.0 docker image for our Solr server
- We are importing the data from an Oracle table using JDBC and the standard
dataimport request handler
- As we want to push the docker image to AWS ECR which only accepts docker
layers of a maximum of 10GB, we load the index in four separate imports,
stopping Solr gracefully in between each load
- Our index contains 48 fields in total
- The problematic field is created through the API as follows:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"buildingName",
"type":"string",
"stored":true,
"indexed":false
}
}' http://localhost:8983/solr/address/schema
I have also tried using SolrText instead of string, but that doesn't make a
noticeable difference.
It also makes a difference how many records are loaded. If I only load
1,000,000 records (that have a proportionate number of building names) then the
size of the index with and without buildingName is about the same (~1GB).
Is there some sort of limit that I'm not aware of that we are hitting, either
number of fields or size of data? Is there some kind of corrupt data that I
need to look for in the buildingName field that could cause this (it's just a
varchar2(46) field in Oracle)?
Thanks for your assistance,
David
David Howe
Java Domain Architect
Postal Systems
Level 16, 111 Bourke Street Melbourne VIC 3000
T 0391067904
M 0424036591
E david.h...@auspost.com.au
W auspost.com.au
W startrack.com.au
Australia Post is committed to providing our customers with excellent service.
If we can assist you in any way please telephone 13 13 18 or visit our website.
The information contained in this email communication may be proprietary,
confidential or legally professionally privileged. It is intended exclusively
for the individual or entity to which it is addressed. You should only read,
disclose, re-transmit, copy, distribute, act in reliance on or commercialise
the information if you are authorised to do so. Australia Post does not
represent, warrant or guarantee that the integrity of this email communication
has been maintained nor that the communication is free of errors, virus or
interference.
If you are not the addressee or intended recipient please notify us by replying
direct to the sender and then destroy any electronic or paper copy of this
message. Any views expressed in this email communication are taken to be those
of the individual sender, except where the sender specifically attributes those
views to Australia Post and is authorised to do so.
Please consider the environment before printing this email.
