[jira] [Commented] (SOLR-6369) duplicate json key echoParams should be removed in ping-response
[ https://issues.apache.org/jira/browse/SOLR-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095910#comment-14095910 ] Marc Portier commented on SOLR-6369: Thx for the clarification. duplicate json key echoParams should be removed in ping-response -- Key: SOLR-6369 URL: https://issues.apache.org/jira/browse/SOLR-6369 Project: Solr Issue Type: Bug Affects Versions: 4.9 Environment: testing on ubuntu 14.04 version: 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier Priority: Trivial Apparently the /admin/ping handler will duplicate the echoParams key in the json-response. $ curl -s http://localhost:8983/solr/test-node-client/admin/ping?wt=json {responseHeader:{status:0,QTime:2,params:{df:text,echoParams:all,rows:10,echoParams:all,wt:json,q:solrpingquery,distrib:false}},status:OK} Having echoParams:all twice in that response doesn't really make sense? Strictly speaking this is allowed in the json spec, so it is not a violation in principle http://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object However there are some json-parse implementations out there that want to map this into hashtables and are not silently overwriting the key with the last seen value, but are throwing errors in the process. To those a cleanup of the json response would make life somewhat easier. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ a
[ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095920#comment-14095920 ] Marc Portier commented on SOLR-6364: As a reference to others experiencing this. Here are some (nodejs) libraries addressing the issue on the javascript end: * https://www.npmjs.org/package/bignumber.js introduces arbitrary big numbers to be handled without loss of precission in Javascript (resembles the Java BigDecimal somewhat) * https://www.npmjs.org/package/json-bigint use that to effectively allow an alternative json-parse that deals with big numbers _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as string! --- Key: SOLR-6364 URL: https://issues.apache.org/jira/browse/SOLR-6364 Project: Solr Issue Type: Bug Components: update Affects Versions: 4.9 Environment: ubuntu 14.04 desktop 32bit Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08) lucene-spec 4.9.0 lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23 solr-spec 4.9.0 solr-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] } see the returned __version__ is a multiple of 100, always! [2] update Using: { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374100 } Response Error: { responseHeader: {status: 409,QTime: 51 }, error: {msg: version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144, code: 409 }} notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100) [3] corrected update, using that effective number { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374144 } Response: { responseHeader: {status: 0,QTime: 597 }, adds: [tst-abcd,1476173026894545000 ] } Odd at first this behaviour is not shown with curl on the command line... [1] create $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 1, type: test, title: [title], _version_: -1 }]' response: {responseHeader:{status:0,QTime:587},adds:[tst-1234,1476163269470191616]} number is not rounded, looks good! [2] update $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 2, type: test, title: [title updated], _version_: 1476163269470191616 }]' response: {responseHeader:{status:0,QTime:512},adds:[tst-1234,1476163320472928256]} All this was pretty much a mistery to me untill I came across this: http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: The largest number JavaScript can handle without loss of precision is 9007199254740992 -- quoted from that stackoverflow page. There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there? - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size In terms of backwards compatibility I don't see an easy way out for the moment. - clients
[jira] [Comment Edited] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _versi
[ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095920#comment-14095920 ] Marc Portier edited comment on SOLR-6364 at 8/13/14 6:50 PM: - As a reference to others experiencing this. Here are some (nodejs) libraries addressing the issue on the javascript end: * https://www.npmjs.org/package/bignumber.js introduces arbitrary big numbers to be handled without loss of precission in Javascript (resembles the Java BigDecimal somewhat) * https://www.npmjs.org/package/json-bigint uses the previous to effectively allow an alternative json-parse that deals with big numbers was (Author: mportier): As a reference to others experiencing this. Here are some (nodejs) libraries addressing the issue on the javascript end: * https://www.npmjs.org/package/bignumber.js introduces arbitrary big numbers to be handled without loss of precission in Javascript (resembles the Java BigDecimal somewhat) * https://www.npmjs.org/package/json-bigint use that to effectively allow an alternative json-parse that deals with big numbers _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as string! --- Key: SOLR-6364 URL: https://issues.apache.org/jira/browse/SOLR-6364 Project: Solr Issue Type: Bug Components: update Affects Versions: 4.9 Environment: ubuntu 14.04 desktop 32bit Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08) lucene-spec 4.9.0 lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23 solr-spec 4.9.0 solr-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] } see the returned __version__ is a multiple of 100, always! [2] update Using: { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374100 } Response Error: { responseHeader: {status: 409,QTime: 51 }, error: {msg: version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144, code: 409 }} notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100) [3] corrected update, using that effective number { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374144 } Response: { responseHeader: {status: 0,QTime: 597 }, adds: [tst-abcd,1476173026894545000 ] } Odd at first this behaviour is not shown with curl on the command line... [1] create $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 1, type: test, title: [title], _version_: -1 }]' response: {responseHeader:{status:0,QTime:587},adds:[tst-1234,1476163269470191616]} number is not rounded, looks good! [2] update $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 2, type: test, title: [title updated], _version_: 1476163269470191616 }]' response: {responseHeader:{status:0,QTime:512},adds:[tst-1234,1476163320472928256]} All this was pretty much a mistery to me untill I came across this: http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: The largest number JavaScript can handle without loss of precision is 9007199254740992 -- quoted from that stackoverflow page. There are more references (below) talking about this being
[jira] [Created] (SOLR-6369) duplicate json key echoParams should be removed in ping-response
Marc Portier created SOLR-6369: -- Summary: duplicate json key echoParams should be removed in ping-response Key: SOLR-6369 URL: https://issues.apache.org/jira/browse/SOLR-6369 Project: Solr Issue Type: Bug Affects Versions: 4.9 Environment: testing on ubuntu 14.04 version: 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier Priority: Trivial Apparently the /admin/ping handler will duplicate the echoParams key in the json-response. $ curl -s http://localhost:8983/solr/test-node-client/admin/ping?wt=json {responseHeader:{status:0,QTime:2,params:{df:text,echoParams:all,rows:10,echoParams:all,wt:json,q:solrpingquery,distrib:false}},status:OK} Having echoParams:all twice in that response doesn't really make sense? Strictly speaking this is allowed in the json spec, so it is not a violation in principle http://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object However there are some json-parse implementations out there that want to map this into hashtables and are not silently overwriting the key with the last seen value, but are throwing errors in the process. To those a cleanup of the json response would make life somewhat easier. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as
Marc Portier created SOLR-6364: -- Summary: _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as string! Key: SOLR-6364 URL: https://issues.apache.org/jira/browse/SOLR-6364 Project: Solr Issue Type: Bug Components: update Affects Versions: 4.9 Environment: ubuntu 14.04 desktop 32bit Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08) lucene-spec 4.9.0 lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23 solr-spec 4.9.0 solr-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] } see the returned _version_ is a multiple of 100, always! [2] update Using: { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374100 } Response Error: { responseHeader: {status: 409,QTime: 51 }, error: {msg: version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144, code: 409 }} notice how the error-message-string correctly mentions the real actual _version_ that is effective (not rounded to 100) [3] corrected update, using that effective number { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374144 } Response: { responseHeader: {status: 0,QTime: 597 }, adds: [tst-abcd,1476173026894545000 ] } Odd at first this behaviour is not shown with curl on the command line... [1] create $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 1, type: test, title: [title], _version_: -1 }]' response: {responseHeader:{status:0,QTime:587},adds:[tst-1234,1476163269470191616]} number is not rounded, looks good! [2] update $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 2, type: test, title: [title updated], _version_: 1476163269470191616 }]' response: {responseHeader:{status:0,QTime:512},adds:[tst-1234,1476163320472928256]} All this was pretty much a mistery to me untill I came across this: http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded This looks like passing down the too big numbers in the _version_ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: The largest number JavaScript can handle without loss of precision is 9007199254740992 -- quoted from that stackoverflow page. There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there? - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size In terms of backwards compatibility I don't see an easy way out for the moment. - clients that expect _version_ to be numeric might not handle the string - in existing deployments it might be hard to reduce all the already existing _version_ to stay undere the limit... I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned _version_ info in a Javascript string. Hoping that might work as a timely (but not as elegant) workaround. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as
[ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marc Portier updated SOLR-6364: --- Description: There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] } see the returned __version__ is a multiple of 100, always! [2] update Using: { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374100 } Response Error: { responseHeader: {status: 409,QTime: 51 }, error: {msg: version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144, code: 409 }} notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100) [3] corrected update, using that effective number { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374144 } Response: { responseHeader: {status: 0,QTime: 597 }, adds: [tst-abcd,1476173026894545000 ] } Odd at first this behaviour is not shown with curl on the command line... [1] create $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 1, type: test, title: [title], _version_: -1 }]' response: {responseHeader:{status:0,QTime:587},adds:[tst-1234,1476163269470191616]} number is not rounded, looks good! [2] update $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 2, type: test, title: [title updated], _version_: 1476163269470191616 }]' response: {responseHeader:{status:0,QTime:512},adds:[tst-1234,1476163320472928256]} All this was pretty much a mistery to me untill I came across this: http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: The largest number JavaScript can handle without loss of precision is 9007199254740992 -- quoted from that stackoverflow page. There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there? - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size In terms of backwards compatibility I don't see an easy way out for the moment. - clients that expect __version__ to be numeric might not handle the string - in existing deployments it might be hard to reduce all the already existing __version__ to stay undere the limit... I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned __version__ info in a Javascript string. Hoping that might work as a timely (but not as elegant) workaround. was: There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] }
[jira] [Commented] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ a
[ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093750#comment-14093750 ] Marc Portier commented on SOLR-6364: bq. My suggestion would be a new json.long param (following the naming convention of json.nl and json.wrf params) that would control wether java long values should be returned as numerics or strings in the JSON response. Yep. This surely sounds like the elegant way out I was still trying to get at. The param would then take num (default for backwards compat) or text as values?. And indeed: it should affect other fields/values in the json as well. _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ == suggested resolution: json should communicate _version_ as string! --- Key: SOLR-6364 URL: https://issues.apache.org/jira/browse/SOLR-6364 Project: Solr Issue Type: Bug Components: update Affects Versions: 4.9 Environment: ubuntu 14.04 desktop 32bit Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08) lucene-spec 4.9.0 lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23 solr-spec 4.9.0 solr-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03 Reporter: Marc Portier There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic. Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision. Here is what one can do to see this in action - all steps below with 1/ using the solr4 admin UI on http://localhost:8983/solr/#/mycore/documents 2/ the request-handler box set to /update?commit=trueversions=true 3/ by adding the following into the documents section on the page: [1] for create Using: { id: tst-abcd, version: 1, type: test, title: [title], _version_: -1 } Response: { responseHeader: {status: 0,QTime: 1882 }, adds: [tst-abcd,1476172747866374100 ] } see the returned __version__ is a multiple of 100, always! [2] update Using: { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374100 } Response Error: { responseHeader: {status: 409,QTime: 51 }, error: {msg: version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144, code: 409 }} notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100) [3] corrected update, using that effective number { id: tst-abcd, version: 2, type: test, title: [title update], _version_: 1476172747866374144 } Response: { responseHeader: {status: 0,QTime: 597 }, adds: [tst-abcd,1476173026894545000 ] } Odd at first this behaviour is not shown with curl on the command line... [1] create $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 1, type: test, title: [title], _version_: -1 }]' response: {responseHeader:{status:0,QTime:587},adds:[tst-1234,1476163269470191616]} number is not rounded, looks good! [2] update $ curl $solrbase/update?commit=trueversions=true -H 'Content-type:application/json' -d '[{ id: tst-1234, version: 2, type: test, title: [title updated], _version_: 1476163269470191616 }]' response: {responseHeader:{status:0,QTime:512},adds:[tst-1234,1476163320472928256]} All this was pretty much a mistery to me untill I came across this: http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: The largest number JavaScript can handle without loss of precision is 9007199254740992 -- quoted from that stackoverflow page. There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there? - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size In terms of backwards compatibility I don't see an easy way out