[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660288#comment-14660288 ] Upayavira commented on SOLR-7576: - We've had JS, we've had XSLT, etc, in our config directory for a long time. What is different about this new approach that it demands a more secure approach in all situations? Please understand - I'm really keen on this feature, and am supportive of the auth/signing functionality. I teach Solr a lot, and I know that a lot of normal users will find the .system and the HTTP post part hard to understand when first getting started, hence wanting to make it really simple to do the simplest things, and possible to do the powerful things. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660681#comment-14660681 ] Upayavira commented on SOLR-7576: - For sure - is suspect we're more on the same page than you think. I would wholly support us moving towards such a situation where ZK is protected, and everything goes via authenticated UIs. My concern is merely about how we get there. e.g. allow filesystem based JS (for non-cloud) and JS in ZK as now for 5.x, but for 6.x remove that ability and require an API across the board. i.e. keep things as consistent as we can? Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660690#comment-14660690 ] Noble Paul commented on SOLR-7576: -- bq.e.g. allow filesystem based JS (for non-cloud) and JS in ZK as now for 5.x A lot of APIs we build of late do not have Standalone Solr support. For lack of time and resources we are unable to build that. If possible I would like to see it supported. bq.i.e. keep things as consistent as we can? yes, eventually consistent Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661362#comment-14661362 ] Noble Paul commented on SOLR-7576: -- bq.There are other features like sources for Suggesters that I look forward to seeing getting upgraded to use the blob store Exactly. That was the reason why I created that API. We need a place where we can store MBs of files and multiple versions of each of them. So a developer should be able to edit without fear and play with his production cluster without impacting his actual users bq.For this specific feature (request handlers in scripting language) why should the executable be put in the Blob store at all? Generally these scripts are going to be very small. I don't know if they are going to be very small and neither do you. In general we need a place which can store a lot of files. Moreover, whether the file is big or small the user experience is going to be exactly same, he will have to run a command line to edit this. Keeping multiple versions of the same script in the conf directory is not what I call elegant. We are just thinking of ZK as a kitchen sink because it used to be file system for standlone solr and size was never a problem Standalone Solr is not going to be the primary target for any new feature that we develop. In a not too distant future we will make the standalone Solr also use a ZK and other stuff to unify dev/user experience. After all a standalone solr can easily be a single collection/ single shard/single replica version of SolrCloud. bq. A REST API on top could add another convenient access mechanism to conf dir stuff Please , we are not going to add more stuff to conf dir. That is a slippery slope. conf dir should have well known standard files. The rest of the stuff should move away. Bloating up ZK just because we used to store things in conf dir is just inertia. We must get over it. Having said that there is nothing that stops us from configuring the JsRequestHandler to load scripts from a local file system directory which can be useful for certain usecases. bq.i.e. exactly how the Script URP works IIRC the script URP does not reload the script automatically by watching the ZK node. You need to go and restart core or other dirty stuff. In a large SolrCloud cluster that amounts to 100's of cores getting reloaded and caches trashed and what not (all queries getting slow/ a lot of GC etc). Do you still like that idea? Let's not assume that whatever we have today is optimized to run on cloud mode. There are a lot of things that got carried forward from the legacy things which are suboptimal to cloud. Instead of clinging on to the old way of life we must modernize our system to adopt to the new paradigm Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) {
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661318#comment-14661318 ] Erick Erickson commented on SOLR-7576: -- bq: guys , editing ZK directly is fraught with huge risk. Please don't recommend it to users. Of course it is. Of course not. That's why I said Totally unsuitable for production of course. But for prototyping/quick experiments on an experimental system it's waay cool. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661214#comment-14661214 ] David Smiley commented on SOLR-7576: I think the pain point here that Upayavira is getting at, and which I see, is the annoyance of having to update a file in ZooKeeper from one's file system. In stand-alone, there is no extra step between saving certain types of files in a conf/ dir on your file system and utilizing it from Solr (perhaps also needing a core reload or similar action). That file list includes script URPs (JS and other interpreted languages supported), XSLTs, Velocity templates, sometimes DIH config files considering the debug mode, and file based Spellcheck Suggester sources, and ExternalFileField's source file. Perhaps there are others I have overlooked. To some extent we could say absolutely everything in a conf/, but there are now RESTful APIs to modify various things in ways that don't involve the need to update the schema or solrconfig.xml directly. What I'd love to see is a file system to ZooKeeper updater that works automatically once something is saved. This might be a utility tool only used in development, or perhaps built-in to bin/solr -- either way it'd be super helpful. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661256#comment-14661256 ] David Smiley commented on SOLR-7576: Nice tip Erick! -- for reference: https://plugins.jetbrains.com/plugin/7364?pr=idea_ce I was just playing around with it now. The only annoyance I've seen is that when editing a file I need to right-click and choose to update ZK; closing the window loses any edits, it appears. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661282#comment-14661282 ] Noble Paul commented on SOLR-7576: -- [~dsmiley] bq. is the annoyance of having to update a file in ZooKeeper from one's file system I can see that pain point. But the current model is totally broken and we MUST move away from from that * we can't keep large amounts data in ZK * We should not give a new feature out which requires a developer direct access to write to ZK.We are talking about the same thing for a toy single node system to large clusters with 1000's of nodes. Any screw up in ZK will have huge impact . It is not like a solr node going down, it is going to blow up a huge cluster * Editing files and picking them up immediately may sound ok if you don't cache stuff. Which is terrible for performance. So there has to be a way to tell Solr that a particular resource is changed * In real production systems we need to rollback stuff if things are broken. This design lets you do that. You always upload a new version of the same file like a version control system. Uploading a new version will have no impact till you push it to production. Play with the new version till you are happy and then push it to production. The point is , the developer can't screw up the cluster by introducing a small syntax error I'm not against adding an easier dev sandbox for such features. This is just the first step. First of all we need to get the feature right for a deployed large cluster. That is the story that we will want to discuss first. This just one extra step for a developer in that phase. Ease of use stuff for playing with things is a secondary thing and we should be able to do it as a separate ticket. Understand that a standalone process and a huge cluster with 1000's of nodes will have some difference in the way stuff is deployed guys , editing ZK directly is fraught with huge risk. Please don't recommend it to users. I understand that we have a lot of legacy stuff without proper APIs and we need to edit things directly. We MUST have a safe way of achieving the same through APIs (eventually). So please do not introduce any new feature that needs ZK write. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661228#comment-14661228 ] Erick Erickson commented on SOLR-7576: -- [~dsmi...@mac.com] re: working with Zookeeper during development FWIW, there's a nifty Zookeeper plugin for IntelliJ that lets you edit files directly on the ZK nodes. Totally unsuitable for production of course, you want all that in some kind of version control. But it removes the PITA of dropping to a command window every time you want to change _anything_ in ZK. Not exactly sure how it'd work with this JIRA though. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661328#comment-14661328 ] David Smiley commented on SOLR-7576: bq. we can't keep large amounts of data in ZK Yes I am aware. I see now why you mention this: it's because the blob store feature is itself a Solr collection and it keeps all versions. I didn't know that until now (or lapse of memory ;-P); I had guessed (wrongly) it was just a REST API into ZooKeeper. I think I better understand where you're coming from now. There are other features like sources for Suggesters that I look forward to seeing getting upgraded to use the blob store, since it's bad to put large files in ZooKeeper in your conf dir. For this specific feature (request handlers in scripting language) why should the executable be put in the Blob store at all? Generally these scripts are going to be very small. Instead, why not simply access it via the SolrResourceLoader abstraction, thus it'll work for stand-alone file-system ZooKeeper if SolrCloud (i.e. exactly how the Script URP works)? A REST API on top could add another convenient access mechanism to conf dir stuff -- obviously with access control required by default though disable'able at the command-line for developers working on scripts locally. AFAIK ZooKeeper clients (e.g. Solr) can monitor ZK paths for updates; and likewise this is possible of the file system in modern Java. In summary, what I'm advocating is that the script be treated like the rest of the configuration for a core, even though technically it's code not configuration. I don't think that distinction is pertinent. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660560#comment-14660560 ] Noble Paul commented on SOLR-7576: -- bq.For sure, offer the more secure, enterprise friendly version, but is there a valid reason not to support the method we are already using e.g. for update processors? So you suggest that the way we are doing it today is good and we should follow the same path. Can you just list the steps involved for doing development process for the same ? if I'm doing iterative development of JS how do I do it bq.I dislike the idea that we are going to split ourselves in half - JS for an update processor can just be edited in place, or can be pushed to ZK via zkcli I would like to give a big -1 for any feature which requires direct access to ZK. In all our security feaatures we assume that ZK writes are protected. So anyone who has access to ZK can compromise security .If it is already there we should slowly remove it. This JS RequestHandler can actually make a scriptupdateprocessor redundant Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660392#comment-14660392 ] Noble Paul commented on SOLR-7576: -- bq.We've had JS, we've had XSLT, etc, in our config directory for a long time. What is different about this new approach that it demands a more secure approach in all situations We don't give an HTTP API to upload any executable (XSLT/JS). You need direct access to ZK. Solr is moving deeper and deeper into the enterprise and security cannot be an afterthought. So, if there are security hole sin our system, we should close them instead of opening new ones. bq. I know that a lot of normal users will find the .system and the HTTP post part hard to understand I miss that point. Users do not need to know anything about {{.system}} collection or anything. They should just blindly run a curl command to upload a new version of their JS. That is just one step. The problem starts when you start explaining the innards of the system. You should just say If you want to upload a new version of your JS, this is the command. {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} It does not matter to the user whether the JS is going to live in the cloud, ZK , filesystem or whatever, it just works. Solr is managing it for them BTW I'm not saying this is the best design possible. Please suggest a simpler way to submit code to SolrCloud and we can adopt that Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660489#comment-14660489 ] Upayavira commented on SOLR-7576: - I dislike the idea that we are going to split ourselves in half - JS for an update processor can just be edited in place, or can be pushed to ZK via zkcli, whereas now, request handler JS must use an entirely different mechanism. That's making Solr much harder to comprehend and understand. I understand how Solr is moving more into enterprises, but don't see how repeating a pattern we have been using for years is making things considerably worse - that's what I'd like to understand. For sure, offer the more secure, enterprise friendly version, but is there a valid reason *not* to support the method we are already using e.g. for update processors? Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660150#comment-14660150 ] Upayavira commented on SOLR-7576: - To most people who are used to using Solr, this development process will seem weird and over-complex - because the blob store functionality isn't yet much used. Can't we have a mode where you just treat your JS the same as the rest of your configs? And also a more advanced mode to handle dynamically uploaded/signed JS like the one you are proposing above? What does this development process give us that we wouldn't have with a file on the filesystem? Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660185#comment-14660185 ] Noble Paul commented on SOLR-7576: -- bq.Can't we have a mode where you just treat your JS the same as the rest of your configs? JS is not configuration. I tis executable code. bq.To most people who are used to using Solr, this development process will seem weird and over-complex - because the blob store functionality isn't yet much used. Overly complex ? If you look at the steps, most of it is one time task (that is because of the security implications of running executable code in your VM). and the only thing you need to do is a {{curl}} command to upload your JS step #4 in the description. I can't imagine how can anything be simpler than this bq.What does this development process give us that we wouldn't have with a file on the filesystem? filesystem? Which file system when you are using solrcloud ? If you wish to use it in standalone mode, then it is not supported, it can be another JIRA Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595083#comment-14595083 ] David Smiley commented on SOLR-7576: I've vaguely heard of the new blob store signing. I took a look at https://cwiki.apache.org/confluence/display/solr/Adding+Custom+Plugins+in+SolrCloud+Mode I get that it's secure but following that would be a real PITA during development. Is there a faster way to to redeploy a changed file during development? Maybe a simple script could do that? Or better yet, a mode during development that would simply pick up the change from the file system. Oh wait; in a SolrCloud/ZooKeeper world, that may not make sense. H :-/ Related to this is the possibility of introducing some sort of AbstractSolrTestCase thingamagic (or new subclass) to test that a request employing some specific JavaScript (but defined in the test or as a test resource) is working. That could lead to a faster dev cycle, and it encourages testing. How does one go about logging? (again, not necessarily for real/production but for development to inspect what the heck some object is or its members). The examples look nice; I like the use of '$'. It'd be nice to see an example of taking the response of a query to pull out it's contents. I understand that JS is a first-class-citizen but if I want to use something else, like Groovy, is that possible? What remains? Did you look at SOLR-5005? Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595091#comment-14595091 ] David Smiley commented on SOLR-7576: bq. Groovy is a security issue . JavaScript is easy to lock down Can you please elaborate how that is so on the JVM? I was unaware that some JVM languages were more secure than others, or frankly that it even matters since this is trusted code (we even have to sign it). Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595089#comment-14595089 ] Noble Paul commented on SOLR-7576: -- All you need is just a curl command to load the new script. I'll post the steps I don't need a generic framework for all platforms . I wish to do JavaScript well and worry about other languages later. Groovy is a security issue . JavaScript is easy to lock down Logging and debugging is something that needs too be added.codebase has changed a lot and the patch in 5005 is not very relevant. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595173#comment-14595173 ] Noble Paul commented on SOLR-7576: -- Groovy has all permissions a normal java program may have . So it is equivalent to native java code. nashorn JS engine has some nice sandboxing features http://mail.openjdk.java.net/pipermail/nashorn-dev/2013-September/002010.html. recently ES dropped groovy citing security concerns I wish to develop the SDK that is tailored for a given language. It has to work well for that language. If we need to support other languages , we should invest some time and thought in providing a nice set of APIs for that instead of trying to provide a sub par experience by catering to all languages. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595180#comment-14595180 ] David Smiley commented on SOLR-7576: Interesting -- thanks for sharing. I promise I won't hassle you any more on this issue about supporting other languages -- that can in another issue. :-) But I find the security issue here quite moot since the code is signed. -- i.e. I could very well *want* to do whatever I want without restrictions. Furthermore, I read the link you pointed to and much of it doesn't apply unless you both run with a security manager and remove various Java things in global scope. And I don't think we should do those things. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts 1 sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595185#comment-14595185 ] Noble Paul commented on SOLR-7576: -- I have changed the description to explain the dev process Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch, SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well the handler {{/js}} is implicitly registered To make this work * Solr should be started with {{-Denable.js.loading=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Sign the javascript and pass the signature in a param called {{_sig}} The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} Steps for developing scripts # start the cluster with the {{enable.js.loading}} . If you are starting using our script it would be {{bin/solr start -e cloud -a -Denable.js.loading=true}} . You would not need security during development , so don't add the private keys to Solr # create {{.system}} collection {{bin/solr create -c .system}} # Write your javascript code . (say {{test.js}} ) # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test.js http://localhost:8983/solr/.system/blob/test}} # run your script {{http://host:8983/solr/gettingstarted/js/test/1}} # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}} . So on and so forth sample programs 1) writes a val to output {code:javascript} //empty line $.response().add('testkey','Test Val'); {code} 2) manipulate the output to add an extra field to each doc {code} //empty line var l = []; $.query({ q: '*:*', qt: '/select', start:0, }).forEach('response', function(doc) { doc.put('script', 'Added this value'); l.push(doc); }); $.response().add('alldocs', l); {code} 3) stream through all the docs {code:Javascript} //empty line $.query({ q: '*:*', qt: '/select', start:0, distrib:'false' }).pipe('response', 'docs', function(doc) { // the pipe function is executed right before the response writer and right after the transformers if('IT'== doc.get('genre_s')) return null; doc.put('script', 'Added this value'); return doc; }); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564284#comment-14564284 ] Noble Paul commented on SOLR-7576: -- I have missed SOLR-5005 I'm mostly done with this. Planning to commit it soon Do you think anything is missing in this patch you wish to include. The objective is not exactly to make just a JS handler. The idea is to provide a comprehensive API set which the functional nature of Javascript can leverage on It also should have the security mechanisms which loading executable code to Solr must adhere to. I'll add security to this before committing Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well example of creating a RequestHandler {code:javascript} curl http://localhost:8983/solr/collection1/config -H 'Content-type:application/json' -d '{ create-requesthandler : {name: jshandler , class:solr.JSRequestHandler, defaults: { js: myreqhandlerjs, //this is the name of the blob in .system collection version:3, sig:mW1Gwtz2QazjfVdrLFHfbGwcr8xzFYgUOLu68LHqWRDvLG0uLcy1McQ+AzVmeZFBf1yLPDEHBWJb5KXr8bdbHN/PYgUB1nsr9pk4EFyD9KfJ8TqeH/ijQ9waa/vjqyiKEI9U550EtSzruLVZ32wJ7smvV0fj2YYhrUaaPzOn9g0= } } }' {code} To make this work * Solr should be started with {{-Denable.runtime.lib=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and configure the signature if security is enabled The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565815#comment-14565815 ] David Smiley commented on SOLR-7576: I particularly like that in SOLR-5005 I didn't hard-code the choice of JavaScript. You could perhaps borrow the script execution mechanism there. I thought through the multi-threading model of Java's scripting API, or lack thereof ;-P, when I wrote that. Without having to review the patch, I think it's important to show here some JavaScript samples that do little interesting things, thereby showing off whatever utilities provided by this script offer. It'd be good to get some peer review on that because it in effect creates an API that is somewhat public, and therefore not trivial to change on a whim later. Good point on security. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well example of creating a RequestHandler {code:javascript} curl http://localhost:8983/solr/collection1/config -H 'Content-type:application/json' -d '{ create-requesthandler : {name: jshandler , class:solr.JSRequestHandler, defaults: { js: myreqhandlerjs, //this is the name of the blob in .system collection version:3, sig:mW1Gwtz2QazjfVdrLFHfbGwcr8xzFYgUOLu68LHqWRDvLG0uLcy1McQ+AzVmeZFBf1yLPDEHBWJb5KXr8bdbHN/PYgUB1nsr9pk4EFyD9KfJ8TqeH/ijQ9waa/vjqyiKEI9U550EtSzruLVZ32wJ7smvV0fj2YYhrUaaPzOn9g0= } } }' {code} To make this work * Solr should be started with {{-Denable.runtime.lib=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and configure the signature if security is enabled The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
[ https://issues.apache.org/jira/browse/SOLR-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564099#comment-14564099 ] David Smiley commented on SOLR-7576: This duplicates SOLR-5005; but the approach appears a little different. Perhaps you forgot about that issue? I admit I've been too busy to finish that up; not that it doesn't work but could use more polish tests. Implement RequestHandler in Javascript -- Key: SOLR-7576 URL: https://issues.apache.org/jira/browse/SOLR-7576 Project: Solr Issue Type: New Feature Reporter: Noble Paul Attachments: SOLR-7576.patch Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126 We can extend the same functionality with JS as well example of creating a RequestHandler {code:javascript} curl http://localhost:8983/solr/collection1/config -H 'Content-type:application/json' -d '{ create-requesthandler : {name: jshandler , class:solr.JSRequestHandler, defaults: { js: myreqhandlerjs, //this is the name of the blob in .system collection version:3, sig:mW1Gwtz2QazjfVdrLFHfbGwcr8xzFYgUOLu68LHqWRDvLG0uLcy1McQ+AzVmeZFBf1yLPDEHBWJb5KXr8bdbHN/PYgUB1nsr9pk4EFyD9KfJ8TqeH/ijQ9waa/vjqyiKEI9U550EtSzruLVZ32wJ7smvV0fj2YYhrUaaPzOn9g0= } } }' {code} To make this work * Solr should be started with {{-Denable.runtime.lib=true}} * The javascript must be loaded to the {{.system}} collection using the blob store API * Configure the requesthandler with the JS blob name and version * Sign the javascript and configure the signature if security is enabled The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/jsname/version}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org