[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[ https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161976#comment-16161976 ] Jan Høydahl commented on SOLR-10665: There are other factors that point towards B as well, such as when upgrading Solr versions, there will be a need to upgrade many plugins at the same time, so that needs to be done offline anyway. We could have GUI support to notify about plugin upgrades, but the upgrade button would tell the user to stop solr, then run bin/solr plugin upgrade and then start Solr again. > POC for a PF4J based plugin system > -- > > Key: SOLR-10665 > URL: https://issues.apache.org/jira/browse/SOLR-10665 > Project: Solr > Issue Type: New Feature > Components: Plugin system >Reporter: Jan Høydahl >Assignee: Jan Høydahl > Labels: pf4j, plugins > Fix For: master (8.0) > > Attachments: SOLR-10665.patch > > > In SOLR-5103 we have been discussing improvements to Solr plugin system, with > ability to bundle a plugin as zip, and easily install from shell or Admin UI. > This task aims to create a working POC to demonstrate how PF4J (Plugin > Framework4J) can be used to bring a very simple plugin packaging and > installation system to Solr with a minimum of effort. Code speaks louder than > words :) > The POC effort is a quite large patch and will be cutting some corners to get > the feature in the hands of people who can test and evaluate. If there is > consensus to add this to Solr, there will be other sub tasks to split up the > elephant into committable chunks. > The design document is located here: https://s.apache.org/solr-plugin (Google > Doc) - comments are welcome in the document or here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[ https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161712#comment-16161712 ] Alexandre Rafalovitch commented on SOLR-10665: -- I think (B) is a great way to get somewhere. On the other hand, (A) is a great way to reinvent OSGi (which was my idea that I never followed-up on). > POC for a PF4J based plugin system > -- > > Key: SOLR-10665 > URL: https://issues.apache.org/jira/browse/SOLR-10665 > Project: Solr > Issue Type: New Feature > Components: Plugin system >Reporter: Jan Høydahl >Assignee: Jan Høydahl > Labels: pf4j, plugins > Fix For: master (8.0) > > Attachments: SOLR-10665.patch > > > In SOLR-5103 we have been discussing improvements to Solr plugin system, with > ability to bundle a plugin as zip, and easily install from shell or Admin UI. > This task aims to create a working POC to demonstrate how PF4J (Plugin > Framework4J) can be used to bring a very simple plugin packaging and > installation system to Solr with a minimum of effort. Code speaks louder than > words :) > The POC effort is a quite large patch and will be cutting some corners to get > the feature in the hands of people who can test and evaluate. If there is > consensus to add this to Solr, there will be other sub tasks to split up the > elephant into committable chunks. > The design document is located here: https://s.apache.org/solr-plugin (Google > Doc) - comments are welcome in the document or here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[ https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161092#comment-16161092 ] Jan Høydahl commented on SOLR-10665: One challenge with installing and especially upgrading plugins is that the rest of the system may assume a static classpath and never re-scan for new SPI classes, and a core using version X of a plugin class may not be aware that version Y is now on the classpath after a plugin upgrade. Even worse, both versions X and Y may exist. So my current thinking is that we either A. Find a way to reload the whole CoreContainer, all cores with config, all security plugins etc gets refreshed and reloaded after a plugin event B. Only allow install and uninstall while running. Upgrade only supported in offline mode What do you think? > POC for a PF4J based plugin system > -- > > Key: SOLR-10665 > URL: https://issues.apache.org/jira/browse/SOLR-10665 > Project: Solr > Issue Type: New Feature > Components: Plugin system >Reporter: Jan Høydahl >Assignee: Jan Høydahl > Labels: pf4j, plugins > Fix For: master (8.0) > > Attachments: SOLR-10665.patch > > > In SOLR-5103 we have been discussing improvements to Solr plugin system, with > ability to bundle a plugin as zip, and easily install from shell or Admin UI. > This task aims to create a working POC to demonstrate how PF4J (Plugin > Framework4J) can be used to bring a very simple plugin packaging and > installation system to Solr with a minimum of effort. Code speaks louder than > words :) > The POC effort is a quite large patch and will be cutting some corners to get > the feature in the hands of people who can test and evaluate. If there is > consensus to add this to Solr, there will be other sub tasks to split up the > elephant into committable chunks. > The design document is located here: https://s.apache.org/solr-plugin (Google > Doc) - comments are welcome in the document or here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[
https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097816#comment-16097816
]
Jan Høydahl commented on SOLR-10665:
[~noble.paul], [~shalinmangar]: This POC is not supposed to give all the
answers, but to point in a direction, thus the "will be cutting some corners"
phrase in issue description. To comment on your points
{quote} How does it work with SolrCloud?
How do you ensure that all nodes in a cloud setup has the same binaries?
How hard is it to enhance the blob store system for managing plugins?
{quote}
[Appendix
B|https://docs.google.com/document/d/1567P8EaLqzmKdtEMu5NVv9Tty_feihGUcnit6TM5V3A/edit#heading=h.ch0p6fdhi10g]
discusses this and lists it as a point to flesh out further. The
plugin-classloader is pluggable, so we could add versions that can load plugins
from HDFS, .system collection etc. later.
bq. 1. I think we should split up plugin system and plugin repositories
features into separate issues. The former can be designed and built without the
latter.
The elephant will be split up in many many small issues before commit. This is
a fast-track POC approach to development where I get the whole thing working
first, and then split things up into parts that can be polished and committed
independently.
bq. 2. We should make it easy to build install custom plugins. Perhaps a maven
archetype might be useful to bootstrap custom plugin projects.
For sure, see [Appendix C - Solr Plugin
SDK|https://docs.google.com/document/d/1567P8EaLqzmKdtEMu5NVv9Tty_feihGUcnit6TM5V3A/edit#heading=h.oxd7nx7mk8j2]
I already started this effort :-)
bq. 3. SolrCloud presents its own challenges which aren't addressed by the
design at all:
See answer to Noble above. There's a lot to flesh out regarding distrib
support. *3.1* Restart is only required in this early POC.
bq. 3.3 Are plugins always cluster-level or should we have them on a per
collection basis thereby exposing certain plugins only to specific collections.
I prefer to simplify this, at least in first version of the feature. I see no
clear benefit in allowing per-collection plugins, as long as we can solve user
needs with a cluster-wide classpath. If we open up for per-collection stuff we
also need to deal with two collections requesting different version of the same
plugin, which is opening up a can of worms, unless you want to refactor all of
Solr into an OSGI app? See [Appendix B - Class-loading and colliding
dependencies|https://docs.google.com/document/d/1567P8EaLqzmKdtEMu5NVv9Tty_feihGUcnit6TM5V3A/edit#heading=h.t9x3xsmmmexu]
for some class loading considerations.
bq. 3.4 I agree with Noble that we should consider enhancing blob store to deal
with this use-case.
That could come as a later option, it is not a deal-breaker if we can make sure
the nodes have the same binaries in a simpler way in v1. How would you make the
{{.system}} collection available for class loading before CoreContainer is
started and e.g. let authentication plugins be loaded from it?
bq. The API can be called just /admin/plugins instead of /admin/pluginbundles
Agree.
bq. It is not clear how we can decide if a plugin is incompatible. If we just
rely only on versions specified in the manifest, it would mean that every
plugin must have a corresponding release for each (major?) Solr release even if
it is only to bump the version number in the manifest.
The plugin author fills the plugin metadata {{Plugin-Requires}} and thus
decides for what Solr versions it is allowed to load. Typically that would be
something like {{>=6.0.0}} to set a lower bound only. That could be successful
if you release a very simple plugin on a stable API that rarely changes, but an
expression that would probably be better in most cases is e.g. {{>=6.0.0 &
<8.0.0}} to say that this will work on the 6.x and 7.x range, but since stuff
may be deprecated in 7.x the plugin author cannot say anything about 8.x. If
the plugin is actively maintained, the author should make sure to release a new
version that is compatible with 8.x before 8.x is out the door. The "this and
next major version" Requires-expression could be the default suggestion in the
SDK.
> POC for a PF4J based plugin system
> --
>
> Key: SOLR-10665
> URL: https://issues.apache.org/jira/browse/SOLR-10665
> Project: Solr
> Issue Type: New Feature
> Components: Plugin system
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Labels: pf4j, plugins
> Fix For: master (8.0)
>
> Attachments: SOLR-10665.patch
>
>
> In SOLR-5103 we have been discussing improvements to Solr plugin system, with
> ability to bundle a plugin as zip, and easily install from shell or Admin UI.
> This task aims to create a working POC to demonstrate how PF4J (Plugin
> Framework4J)
[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[ https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097076#comment-16097076 ] Shalin Shekhar Mangar commented on SOLR-10665: -- Thanks Jan. I have a few comments: # I think we should split up plugin system and plugin repositories features into separate issues. The former can be designed and built without the latter. # We should make it easy to build install custom plugins. Perhaps a maven archetype might be useful to bootstrap custom plugin projects. # SolrCloud presents its own challenges which aren't addressed by the design at all: ## We shouldn't force people to deploy plugins by ssh'ing into each box and restarting Solr ## As Noble said, every node must have the same binary. This must be enforced. ## Are plugins always cluster-level or should we have them on a per collection basis thereby exposing certain plugins only to specific collections. ## I agree with Noble that we should consider enhancing blob store to deal with this use-case. # The framework must play well with the security API. Only authorized users must be able to install/upgrade/remove plugins. There should be a separate permission for these operations. # The API can be called just /admin/plugins instead of /admin/pluginbundles > POC for a PF4J based plugin system > -- > > Key: SOLR-10665 > URL: https://issues.apache.org/jira/browse/SOLR-10665 > Project: Solr > Issue Type: New Feature > Components: Plugin system >Reporter: Jan Høydahl >Assignee: Jan Høydahl > Labels: pf4j, plugins > Fix For: master (8.0) > > Attachments: SOLR-10665.patch > > > In SOLR-5103 we have been discussing improvements to Solr plugin system, with > ability to bundle a plugin as zip, and easily install from shell or Admin UI. > This task aims to create a working POC to demonstrate how PF4J (Plugin > Framework4J) can be used to bring a very simple plugin packaging and > installation system to Solr with a minimum of effort. Code speaks louder than > words :) > The POC effort is a quite large patch and will be cutting some corners to get > the feature in the hands of people who can test and evaluate. If there is > consensus to add this to Solr, there will be other sub tasks to split up the > elephant into committable chunks. > The design document is located here: https://s.apache.org/solr-plugin (Google > Doc) - comments are welcome in the document or here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10665) POC for a PF4J based plugin system
[ https://issues.apache.org/jira/browse/SOLR-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097052#comment-16097052 ] Noble Paul commented on SOLR-10665: --- How does it work with SolrCloud? How do you ensure that all nodes in a cloud setup has the same binaries? How hard is it to enhance the blob store system for managing plugins? > POC for a PF4J based plugin system > -- > > Key: SOLR-10665 > URL: https://issues.apache.org/jira/browse/SOLR-10665 > Project: Solr > Issue Type: New Feature > Components: Plugin system >Reporter: Jan Høydahl >Assignee: Jan Høydahl > Labels: pf4j, plugins > Fix For: master (8.0) > > Attachments: SOLR-10665.patch > > > In SOLR-5103 we have been discussing improvements to Solr plugin system, with > ability to bundle a plugin as zip, and easily install from shell or Admin UI. > This task aims to create a working POC to demonstrate how PF4J (Plugin > Framework4J) can be used to bring a very simple plugin packaging and > installation system to Solr with a minimum of effort. Code speaks louder than > words :) > The POC effort is a quite large patch and will be cutting some corners to get > the feature in the hands of people who can test and evaluate. If there is > consensus to add this to Solr, there will be other sub tasks to split up the > elephant into committable chunks. > The design document is located here: https://s.apache.org/solr-plugin (Google > Doc) - comments are welcome in the document or here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
