Re: Ignoring Duplicates in Multivalue Field
From memory, if you use UniqFieldsUpdateProcessor after
DistributedUpdateProcessor, then you will be filtering on the set [1,
2, 3, 2].
*updateRequestProcessorChain name=uniq-fields*
*processor
class=solr.DistributedUpdateProcessorFactory/ *
* processor
class=solr.UniqFieldsUpdateProcessorFactory*
*str name=fieldRegex myMultValueField
/str*
* /processor*
*processor class=solr.RunUpdateProcessorFactory /*
*/updateRequestProcessorChain*
On 4 November 2014 01:37, Jack Krupansky j...@basetechnology.com wrote:
The update processors are only processing the values in the source data,
not the data that has already been indexed and stored.
We probably need to file a Jira to add an insert field value option that
merges in the new field value, skipping it if it already exists or
appending it to the end of the existing list of field values for a
multivalued field.
You could try... a combination of both remove and add, assuming that
Solr applies them in the order specified, to remove any existing value and
then add it to the end.
See:
https://cwiki.apache.org/confluence/display/solr/
Updating+Parts+of+Documents
-- Jack Krupansky
-Original Message- From: Tomer Levi
Sent: Monday, November 3, 2014 4:19 AM
To: solr-user@lucene.apache.org ; Ahmet Arslan
Subject: RE: Ignoring Duplicates in Multivalue Field
Hi Ahmet,
When I add the RunUpdateProcessorFactory Solr didn't remove any
duplications.
Any other idea?
-Original Message-
From: Ahmet Arslan [mailto:iori...@yahoo.com.INVALID]
Sent: Monday, November 03, 2014 1:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Ignoring Duplicates in Multivalue Field
Hi Tomer,
What happens when you add processor class=solr.RunUpdateProcessorFactory
/ to your chain?
Ahmet
On Sunday, November 2, 2014 1:22 PM, Tomer Levi tomer.l...@nice.com
wrote:
Hi,
I’m trying to make my “update” request handler ignore multivalue
duplications in updates.
To make my use case clear, let’s assume my index already contains a
document like:
{
id:”100”,
“myMultValueField”: [“1”,”2”,”3”]
}
Later I would like to send an update like:
{
id:”100”,”
myMultValueField” {“add”:”2”}
}
How can I make the update request handler understand that “2” already
exist and ignore it?
I tried to add update chain below but it didn’t work for me.
updateRequestProcessorChain name=uniq-fields
processor class=solr.UniqFieldsUpdateProcessorFacto
ry
str name=fieldRegex myMultValueField
/str
/processor
/updateRequestProcessorChain
And add it to my requestHandler:
requestHandler name=/update class=solr.UpdateRequestHandler
lst name=defaults
str name=update.chainuniq-fields/str
/lst
/requestHandler
Tomer Levi
Software Engineer
Big Data Group
Product Technology Unit
(T) +972 (9) 775-2693
tomer.l...@nice.com
www.nice.com
Re: suggestion for new custom atomic update
You can get the summed value, 13, if you add a processor after
DistributedUpdateProcessorFactory in the URP chain. Then one possibility
would be to clone this value to another field, such as field_b, and run
other processors on that field.
Or for something more customized, you can use
the StatelessScriptUpdateProcessorFactory, and retrieve the value of
field_a with:
var doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument
var field_a = doc.getFieldValue(field_a);
Note if you try to get the value of field_a before
DistributedUpdateProcessorFactory, then using your example with an atomic
update, the value would be 5 (the value of the increment from the input
document).
On 27 October 2014 18:03, Elran Dvir elr...@checkpoint.com wrote:
I will explain with an example.
Let's say field_a is sent in the update with the value of 5. field_a is
already stored in the document with the value 8.
After the update field_a should have the value 13 (sum).
The value of field_b will be based on the value of 13 and not 5.
Is there a way in URP to know what is the value which is already stored in
field_a?
Thank you very much.
-Original Message-
From: Alexandre Rafalovitch [mailto:arafa...@gmail.com]
Sent: Sunday, October 26, 2014 6:07 PM
To: solr-user
Subject: Re: suggestion for new custom atomic update
I am not sure what the problem is. URP catches all operations. So, you can
modify the source document to add the calculation when the field_a is
either new or updated.
Or are you trying to calculate things across multiple documents? In that
case, neither mine nor your solution will work, I think.
Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov Solr resources and
newsletter: http://www.solr-start.com/ and @solrstart Solr popularizers
community: https://www.linkedin.com/groups?gid=6713853
On 26 October 2014 12:00, Elran Dvir elr...@checkpoint.com wrote:
Thanks for your response.
If the calculation is based on the most recent summed value of field_a
and the value of field_a in the update, how can I?
Thanks.
-Original Message-
From: Alexandre Rafalovitch [mailto:arafa...@gmail.com]
Sent: Sunday, October 26, 2014 2:11 PM
To: solr-user
Subject: RE: suggestion for new custom atomic update
Can't you do the calculation in custom UpdateRequestProcessor?
Regards,
Alex
On 26/10/2014 4:17 am, Elran Dvir elr...@checkpoint.com wrote:
Hi all,
Did anyone have a chance to review my idea?
Thanks.
-Original Message-
From: Elran Dvir
Sent: Monday, October 20, 2014 12:42 PM
To: solr-user
Subject: suggestion for new custom atomic update
Hi all,
This is my use case:
I have a stored field, field_a, which is atomic updated (let's say by
inc). field_a is stored but not indexed due to the large number of
distinct values it can have.
I need to index field_b (I need facet and stats on it) which is not
in the document but its value is based on a calculation of the recent
(e.g.
summed) value of field_a.
There is no way to do it nowadays.
So I thought of a new method: custom atomic update.
There will be a new interface in Solr:
public interface CustomAtomicUpdater {
public void update(SolrInputDocument oldDoc, String fieldName,
Object
fieldVal) ; }
There will be a new attribute for fields in schema.xml called
customAtomicUpdateClass (and all support in code, of course).
The value is a class which is an implementation of CustomAtomicUpdater.
In our example it will be defined for field_a.
In method getUpdatedDocument in DistributedUpdateProcessor.java, we
will add handling of custom case:
} else if (custom.equals(key)) {
updateField = true;
SchemaField sf = schema.getField(sif.getName());
String customAtomicUpdaterClassName =
sf.getCustomAtomicUpdaterClass();
if (customAtomicUpdaterClassName == null) {
throw new SolrException(ErrorCode.BAD_REQUEST, There
is no customAtomicUpdaterClass defined for + sif + .);
}
CustomAtomicUpdater updater = schema.getResourceLoader()
.newInstance(customAtomicUpdaterClassName,
CustomAtomicUpdater.class);
if (updater == null) {
throw new SolrException(ErrorCode.BAD_REQUEST, Was
unable to create instance of + customAtomicUpdaterClassName + .);
}
updater.update(oldDoc, sif.getName(), fieldVal);
}
In my implementation I will sum field_a (oldvalue + newvalue) and
update field_b according to my logic.
Example of use:
add
doc
field name=field_a update=custom128/field
/doc
/add
What do say about my suggestion?
Thanks.
Email secured by Check Point
Email secured by Check Point
Re: suggestion for new custom atomic update
No problem Elran. As Shalin mentioned, you will need to do it like this:
processor class=solr.DistributedUpdateProcessorFactory/
processor class=mycode.solr_plugins.FieldManipulationProcessorFactory /
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
On 28 October 2014 03:22, Shalin Shekhar Mangar shalinman...@gmail.com
wrote:
Hi Elran,
You need to explicitly specify the DistributedUpdateProcessorFactory in the
chain and then add your custom processor after it.
On Mon, Oct 27, 2014 at 9:26 PM, Elran Dvir elr...@checkpoint.com wrote:
Thank you very much for your suggestion.
I created an update processor factory with my logic.
I changed the update processor chain to be:
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
processor class=mycode.solr_plugins.FieldManipulationProcessorFactory
/
But nothing seems to happen.
When I move my class to be the first in the chain, the logic is running
(not as I want, of course. It's calculated based on the update value
rather
than the stored value) .
How can I define a custom update processor factory that will run after
DistributedUpdateProcessorFactory?
Thank you very much.
-Original Message-
From: Matthew Nigl [mailto:matthew.n...@gmail.com]
Sent: Monday, October 27, 2014 12:10 PM
To: solr-user@lucene.apache.org
Subject: Re: suggestion for new custom atomic update
You can get the summed value, 13, if you add a processor after
DistributedUpdateProcessorFactory in the URP chain. Then one possibility
would be to clone this value to another field, such as field_b, and run
other processors on that field.
Or for something more customized, you can use the
StatelessScriptUpdateProcessorFactory, and retrieve the value of field_a
with:
var doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument
var field_a = doc.getFieldValue(field_a);
Note if you try to get the value of field_a before
DistributedUpdateProcessorFactory, then using your example with an atomic
update, the value would be 5 (the value of the increment from the input
document).
On 27 October 2014 18:03, Elran Dvir elr...@checkpoint.com wrote:
I will explain with an example.
Let's say field_a is sent in the update with the value of 5. field_a
is already stored in the document with the value 8.
After the update field_a should have the value 13 (sum).
The value of field_b will be based on the value of 13 and not 5.
Is there a way in URP to know what is the value which is already
stored in field_a?
Thank you very much.
-Original Message-
From: Alexandre Rafalovitch [mailto:arafa...@gmail.com]
Sent: Sunday, October 26, 2014 6:07 PM
To: solr-user
Subject: Re: suggestion for new custom atomic update
I am not sure what the problem is. URP catches all operations. So, you
can modify the source document to add the calculation when the field_a
is either new or updated.
Or are you trying to calculate things across multiple documents? In
that case, neither mine nor your solution will work, I think.
Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov Solr resources
and
newsletter: http://www.solr-start.com/ and @solrstart Solr
popularizers
community: https://www.linkedin.com/groups?gid=6713853
On 26 October 2014 12:00, Elran Dvir elr...@checkpoint.com wrote:
Thanks for your response.
If the calculation is based on the most recent summed value of
field_a
and the value of field_a in the update, how can I?
Thanks.
-Original Message-
From: Alexandre Rafalovitch [mailto:arafa...@gmail.com]
Sent: Sunday, October 26, 2014 2:11 PM
To: solr-user
Subject: RE: suggestion for new custom atomic update
Can't you do the calculation in custom UpdateRequestProcessor?
Regards,
Alex
On 26/10/2014 4:17 am, Elran Dvir elr...@checkpoint.com wrote:
Hi all,
Did anyone have a chance to review my idea?
Thanks.
-Original Message-
From: Elran Dvir
Sent: Monday, October 20, 2014 12:42 PM
To: solr-user
Subject: suggestion for new custom atomic update
Hi all,
This is my use case:
I have a stored field, field_a, which is atomic updated (let's say
by inc). field_a is stored but not indexed due to the large
number of distinct values it can have.
I need to index field_b (I need facet and stats on it) which is not
in the document but its value is based on a calculation of the
recent
(e.g.
summed) value of field_a.
There is no way to do it nowadays.
So I thought of a new method: custom atomic update.
There will be a new interface in Solr:
public interface CustomAtomicUpdater {
public void update(SolrInputDocument
Re: Delete in Solr based on foreign key (like SQL delete from … where id in (select id from…)
Absolutely, using Mikhail's code would be the first thing I would do. You can see the details in SOLR-6234 and https://github.com/m-khl/lucene-join-solr-query-parser Otherwise, the only alternative I can think of (without reindexing) would be to run the select query as provided, returning the ID's of the offending documents (for reference, you could use grouping or the collapsing query parser if you just want to get distinct values; faceting is also an option). Then write a script to iterate through a batch of ID's at a time and send a delete to Solr, such as deletequeryid:(100 OR 101 OR ...)/query/delete. Since there are many documents to delete, you would want to hold off committing until the end. On 11 October 2014 02:34, Mikhail Khludnev mkhlud...@griddynamics.com wrote: On Fri, Oct 10, 2014 at 6:16 AM, Matthew Nigl matthew.n...@gmail.com wrote: But I get the same response as in https://issues.apache.org/jira/browse/SOLR-6357 there is a mention for cure (SOLR-6234 https://issues.apache.org/jira/browse/SOLR-6234) over there -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com
Re: How does one specify which collections to use?
Hi Aaron, You would need to set 'path' in the options array when you create a SolrClient. http://php.net/manual/en/solrclient.construct.php Regards, Matt On 15 October 2014 19:31, Aaron Lewis the.warl0ck.1...@gmail.com wrote: Hi, I'm using PHP client here: http://php.net/manual/en/class.solrquery.php I couldn't figure out how to use collection2 instead of collection1, I don't see such options in constructor either. Anyone know that? -- Best Regards, Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/ Finger Print: 9F67 391B B770 8FF6 99DC D92D 87F6 2602 1371 4D33
Re: Does the SolrQuery class support 'inc' command?
Hi Aaron, I do not believe that there is direct support for atomic updates with this API. Other libraries such as Solarium do support this. However, one option is that you could generate the atomic update XML in code and send it via a raw request: http://php.net/manual/en/solrclient.request.php Regards, Matt On 15 October 2014 22:23, Aaron Lewis the.warl0ck.1...@gmail.com wrote: Hi, I'm trying to do a partial update, according to https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents I can use 'inc' command to do that, but I couldn't find relevant PHP API here: http://php.net/manual/en/class.solrquery.php Anyone know what function should I use? I tried 'addField' but that would just override existing entry -- Best Regards, Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/ Finger Print: 9F67 391B B770 8FF6 99DC D92D 87F6 2602 1371 4D33
Re: Delete in Solr based on foreign key (like SQL delete from … where id in (select id from…)
I was going to say that the below should do what you are asking:
deletequery{!join from=docid_s
to=foreign_key_docid_s}(message_state_ts:[* TO 2014-10-05T00:00:00Z} AND
message_state_ts:{2014-10-01T00:00:00Z TO *])/query/delete
But I get the same response as in
https://issues.apache.org/jira/browse/SOLR-6357
I can't think of any other queries at the moment. You might consider using
the above query (which should work as a normal select query) to get the
IDs, then delete them in a separate query.
On 10 October 2014 07:31, Luis Festas Matos lufe...@gmail.com wrote:
Given the following Solr data:
doc
str name=index_id1008rs1cz0icl2pk/str
date name=message_state_ts2014-10-07T14:18:29.784Z/date
str name=docid_sh60fmtybz0i7sx87/str
long name=_version_1481314421768716288/long/docdoc
str name=index_idu42xyz1cz0i7sx87/str
str name=foreign_key_docid_sh60fmtybz0i7sx87/str
long name=_version_1481314421768716288/long/docdoc
str name=index_idu42xyz1cz0i7sx87/str
str name=foreign_key_docid_sh60fmtybz0i7sx87/str
long name=_version_1481314421448900608/long/doc
I would like to know how to *DELETE documents* above on the Solr console or
using a script that achieves the same result as issuing the following
statement in SQL (assuming all of these columns existed in a table called x
):
DELETE FROM x WHERE foreign_key_docid_s in (select docid_s from x
where message_state_ts '2014-10-05' and message_state_ts
'2014-10-01')
Basically, delete all derived documents whose foreign key is the same as
the primary key where the primary key is selected between 2 dates.
Question originally posted on stackoverflow.com -
http://stackoverflow.com/questions/26248372/delete-in-solr-based-on-foreign-key-like-sql-delete-from-where-id-in-selec
NullPointerException for ExternalFileField when key field has no terms
Hi, I use various ID fields as the keys for various ExternalFileField fields, and I have noticed that I will sometimes get the following error: ERROR org.apache.solr.servlet.SolrDispatchFilter û null:java.lang.NullPointerException at org.apache.solr.search.function.FileFloatSource.getFloats(FileFloatSource.java:273) at org.apache.solr.search.function.FileFloatSource.access$000(FileFloatSource.java:51) at org.apache.solr.search.function.FileFloatSource$2.createValue(FileFloatSource.java:147) at org.apache.solr.search.function.FileFloatSource$Cache.get(FileFloatSource.java:190) at org.apache.solr.search.function.FileFloatSource.getCachedFloats(FileFloatSource.java:141) at org.apache.solr.search.function.FileFloatSource.getValues(FileFloatSource.java:84) at org.apache.solr.response.transform.ValueSourceAugmenter.transform(ValueSourceAugmenter.java:95) at org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:252) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:170) at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:184) at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:300) at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:96) at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:61) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:765) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:426) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source) The source code referenced in the error is below (FileFloatSource.java:273): TermsEnum termsEnum = MultiFields.getTerms(reader, idName).iterator(null); So if there are no terms in the index for the key field, then getTerms will return null, and of course trying to call iterator on null will cause the exception. For my use-case, it makes sense that the key field may have no terms (initially) because there are various types of documents sharing the index, and they will not all exist at the onset. The default value for the EFF would suffice in those cases. Is this worthy of a JIRA? I have gone through whatever documentation I can find for ExternalFileField and I can't seem to find anything about requiring key terms first. It seems that this error is not encountered often because users generally set the unique key field as the external file key field, so it always exists. The workaround is to ensure at least
Re: NullPointerException for ExternalFileField when key field has no terms
Thanks Markus. I initially interpreted the line It's OK to have a keyField value that can't be found in the index as meaning that the key field value in the external file does not have to exist as a term in the index. On 8 October 2014 23:56, Markus Jelsma markus.jel...@openindex.io wrote: Hi - yes it is worth a ticket as the javadoc says it is ok: http://lucene.apache.org/solr/4_10_1/solr-core/org/apache/solr/schema/ExternalFileField.html -Original message- From:Matthew Nigl matthew.n...@gmail.com Sent: Wednesday 8th October 2014 14:48 To: solr-user@lucene.apache.org Subject: NullPointerException for ExternalFileField when key field has no terms Hi, I use various ID fields as the keys for various ExternalFileField fields, and I have noticed that I will sometimes get the following error: ERROR org.apache.solr.servlet.SolrDispatchFilter û null:java.lang.NullPointerException at org.apache.solr.search.function.FileFloatSource.getFloats(FileFloatSource.java:273) at org.apache.solr.search.function.FileFloatSource.access$000(FileFloatSource.java:51) at org.apache.solr.search.function.FileFloatSource$2.createValue(FileFloatSource.java:147) at org.apache.solr.search.function.FileFloatSource$Cache.get(FileFloatSource.java:190) at org.apache.solr.search.function.FileFloatSource.getCachedFloats(FileFloatSource.java:141) at org.apache.solr.search.function.FileFloatSource.getValues(FileFloatSource.java:84) at org.apache.solr.response.transform.ValueSourceAugmenter.transform(ValueSourceAugmenter.java:95) at org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:252) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:170) at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:184) at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:300) at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:96) at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:61) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:765) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:426) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source) The source code referenced
Re: Currency calculation does not seem to be using cross-rates
For anyone interested, I realised that there are differences between the various ways of using currency. Not all appear to support asymmetric rates, and documents that do not need conversion are converted anyway, sometimes introducing rounding error. For point queries. e.g. fq=price_c:100,AUD - the query value is converted to the default currency and the indexed value is converted to the default currency, then compared - from debug it is: frange(rawcurrency(price_c,target=USD)):[8700 TO 8700] For range queries. e.g. fq=price_c:[100,AUD TO 200,AUD] - the query value is used directly and the indexed value is directly converted to the target currency, then compared. - from debug it is: frange(rawcurrency(price_c,target=AUD)):[1 TO 2] For the currency function. e.g. currency(price_c,AUD) - the indexed value is converted to the default currency, then it is converted again to the target currency So if you rely on using the currency function, especially to return values to the view, then this may be inaccurate if the indexed value is in the same currency as the target currency, where target currency is not the default currency. I often see a value such as 500,AUD in the index get returned as 499.99 due to the rounding error when it is converted from AUD - USD - AUD. If I am using any of the above incorrectly then please let me know. Regards, Matt On 29 August 2014 14:54, Matthew Nigl matthew.nigl+s...@gmail.com wrote: Hi all, I have an issue where the currency function is not returning the expected value in particular scenarios. I believe my config is correct and I am able to replicate the issue over a few machines. The issue occurs when converting the value of a currency field from the source currency into a target currency, where neither are the default currency as set in the schema. For example: - the schema contains a field called price of type currency, where defaultCurrency=USD - the value indexed is 500,EUR - the query string is ?q=*:*fl=currency(price,AUD) - the currency.xml contains the following rates, which are set to exaggerate the issue: rate from=EUR to=AUD rate=0.5 / rate from=EUR to=USD rate=0.5 / rate from=USD to=AUD rate=2 / - the value returned is 500. The expected value is 250 So it appears that the value is calculated by converting from source currency - default currency, and then from default currency - target currency. However, the direct conversion from EUR to AUD is available, so I expect that it should be used to generate the value instead. This issue also occurs when the source currency and target currency are the same, as long as it is not the default currency. For example, when converting from AUD to AUD with a default currency of USD, I often notice rounding errors since (source-default rate) * (default-target rate) != 1 for many cases. I have included the relevant part of my schema below. I have tested on both 4.8.1 and 4.9.0. field name=price type=currency indexed=true stored=true multiValued=false/ fieldType name=currency class=solr.CurrencyField precisionStep=8 defaultCurrency=USD currencyConfig=currency.xml / Regards, Matt
Currency calculation does not seem to be using cross-rates
Hi all, I have an issue where the currency function is not returning the expected value in particular scenarios. I believe my config is correct and I am able to replicate the issue over a few machines. The issue occurs when converting the value of a currency field from the source currency into a target currency, where neither are the default currency as set in the schema. For example: - the schema contains a field called price of type currency, where defaultCurrency=USD - the value indexed is 500,EUR - the query string is ?q=*:*fl=currency(price,AUD) - the currency.xml contains the following rates, which are set to exaggerate the issue: rate from=EUR to=AUD rate=0.5 / rate from=EUR to=USD rate=0.5 / rate from=USD to=AUD rate=2 / - the value returned is 500. The expected value is 250 So it appears that the value is calculated by converting from source currency - default currency, and then from default currency - target currency. However, the direct conversion from EUR to AUD is available, so I expect that it should be used to generate the value instead. This issue also occurs when the source currency and target currency are the same, as long as it is not the default currency. For example, when converting from AUD to AUD with a default currency of USD, I often notice rounding errors since (source-default rate) * (default-target rate) != 1 for many cases. I have included the relevant part of my schema below. I have tested on both 4.8.1 and 4.9.0. field name=price type=currency indexed=true stored=true multiValued=false/ fieldType name=currency class=solr.CurrencyField precisionStep=8 defaultCurrency=USD currencyConfig=currency.xml / Regards, Matt
Currency calculation does not seem to be using cross-rates
Hi all, I have an issue where the currency function is not returning the expected value in particular scenarios. I believe my config is correct and I am able to replicate the issue over a few machines. The issue occurs when converting the value of a currency field from the source currency into a target currency, where neither are the default currency as set in the schema. For example: - the schema contains a field called price of type currency, where defaultCurrency=USD - the value indexed is 500,EUR - the query string is ?q=*:*fl=currency(price,AUD) - the currency.xml contains the following rates, which are set to exaggerate the issue: rate from=EUR to=AUD rate=0.5 / rate from=EUR to=USD rate=0.5 / rate from=USD to=AUD rate=2 / - the value returned is 500. The expected value is 250 So it appears that the value is calculated by converting from source currency - default currency, and then from default currency - target currency. However, the direct conversion from EUR to AUD is available, so I expect that it should be used to generate the value instead. This issue also occurs when the source currency and target currency are the same, as long as it is not the default currency. For example, when converting from AUD to AUD with a default currency of USD, I often notice rounding errors since (source-default rate) * (default-target rate) != 1 for many cases. I have included the relevant part of my schema below. I have tested on both 4.8.1 and 4.9.0. field name=price type=currency indexed=true stored=true multiValued=false/ fieldType name=currency class=solr.CurrencyField precisionStep=8 defaultCurrency=USD currencyConfig=currency.xml / Regards, Matt
