[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967248#comment-16967248 ] David Smiley commented on SOLR-13661: - [~ichattopadhyaya] Just checking; is the description here pointing to that Google Doc still the most up-to-date one and accurate? > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945571#comment-16945571 ] Ishan Chattopadhyaya commented on SOLR-13661: - We shall shortly document the steps for upgrades across minor Solr versions and major Solr versions. I think attempting to answer them here (as replies to your specific JIRA comments) will not be effective. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945564#comment-16945564 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. [~janhoy]: But I agree that, if handled carefully and labeled "experimental", we may introduce this in a series of 8.x point releases I think now we're in agreement. bq.: [~janhoy] .. and perhaps start moving features from core into packages in master, targeting a slim solr-core in 9.0, which is a major "feature" in itself. That is something we can address in long term (preferably after we've moved over to Gradle). And surely, not in 8x. TLDR; lets put the supporting pieces for the package manager in 8x now, and open up the usecase fully for 3rd party plugin/package writers. For first party plugins/packages, we will wait for a major release (because it affects how we distribute Solr) and deal with that in a different JIRA/Epic. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945484#comment-16945484
]
Jan Høydahl commented on SOLR-13661:
{quote}You seem to have very little understanding of how classloaders work.
Putting multiple classloaders together will result in classcastexceptions which
are impossible to debug
{quote}
Who says anything about putting class loaders together? PF4J and the POC also
has isolated class loaders per plugin bundle to achieve isolation. But my POC
uses a special "MultiPluginBundleClassLoader" in SolrResourceLoader, which is
consulted after lookup in WEB-INF/lib and SOLR_HOME/lib fails, and it hunts
through all installed plugins. It also works for looking up any resource on
class path.
I'm thus *not* talking about mashing all class loaders together, but when a
configSet declares that it uses e.g. {{kuromoji}} and you in
{{managed-schema}} declare {{}} then Solr would first lookup that
plugin class in core/global class loader, and when not found, it will look in
the "kuromoji" package where it will be found. You still have the isolation,
and any class loading performed by Kuromoji itself to load its dependencies are
handled by the isolated package class loader, so it won't use an incompatible
version of some dependency. It is just the first main plugin class lookup that
needs to resolve packages. And then we won't need a
{{class="kuromoji:solr.JapaneseTokenizerFactory"}} syntax in every possible
piece of the product that load classes or resources.
Another related question that I thought about in my POC but never got to test
thoroughly was SPI plugins. We use SPI for most Schema plugins, so in fact as
of SOLR-13593 you an now say {{}} or {{}} in your schema instead of referring to class names. This
will look up the plugin through SPI. For this to work with the package system
you would either need to bolt on a special name syntax here, e.g.
":" or let SPI find the class by also looking in the declared
packages.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945481#comment-16945481
]
Jan Høydahl commented on SOLR-13661:
{quote}[~noble.paul]: Rolling restarts are NEVER REQUIRED. Solr behaves very
badly during rolling restarts, we lose shards/replicas or our overseer get
clogged with messages and needs manual intervention.
{quote}
I understand the objective of hot update not needing restart. But in my point
5) i was talking about *rolling upgrades* as officially documented and
recommended in ref-guide
([https://lucene.apache.org/solr/guide/8_1/upgrading-a-solr-cluster.html)]. I
hear you say that you personally have bad experience with rolling restarts, but
are you also saying that the new package design is incompatible with the
rolling upgrade procedure?
If we still want to support rolling upgrade, I think it is crucial that we have
a well designed and tested story for package upgrades during a rolling restart
where some nodes are 8.x and other nodes are 9.x. In that case ZK as source of
truth for which plugin version a collection has is simply not good enough. Each
node must be able to choose and upgrade a package to a package version that is
compatible with the new Solr version.
That is part of the reason I want a static/cold way of installing a package and
resolve version, before the node is even started. And on node startup, the node
needs to validate every package/plugin and refuse to start if an active
package's version constraint is not satisfied. It must then be possible to
upgrade packages on that node (which is just upgrade to 9.0) only without
forcing the same version of a package for all other nodes (which still run Solr
8.x).
Plugin/package authors are of course encouraged to make sure their plugins will
not break if two versions are in use by the same cluster at the same time. This
may be challenging for e.g. LTR that use shared data files (such as ML models).
But I guess it will have to be handled and documented in the same way we do for
Solr in general. E.g. "In order to upgrade to LTR v9.0 you first need to be
running v 8.x which supports both old and new model format" or similar. Ideally
such upgrade requirements would be possible to encode in the manifest.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945471#comment-16945471
]
Noble Paul commented on SOLR-13661:
---
bq.This is what we have ResourceLoader and class loaders for. If we cannot find
{{com.example.MyPlugin}} in main class loader, then hunt through every package
class loader until you have a match, ...
You seem to have very little understanding of how classloaders work .
Putting multiple classloaders together will result in classcastexceptions which
are impossible to debug. This also means we cannot reload a package independent
of another.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945459#comment-16945459 ] ASF subversion and git services commented on SOLR-13661: Commit 515af6d3ecc5ea1b3754776f9ce4e39857f99734 in lucene-solr's branch refs/heads/jira/SOLR-13787 from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=515af6d ] SOLR-13661: Reverting all half-baked stuff from SOLR-13707, SOLR-13659, SOLR-13565, SOLR-13650, SOLR-13710, SOLR-13721, SOLR-13637 All half baked package management and hot-classloading code reverted to allow for a fresh start. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945457#comment-16945457
]
Noble Paul commented on SOLR-13661:
---
bq.Can you elaborate on that? Is it simply so that a collection using packages
cannot also use "..
No, That's not true.
If your plugin uses {{package=}} that plugin cannot use classes
from {{}}. All other components can use {{}}
bq. But being a totally new way of distributing Solr and having huge impact on
the whole user and wider dev community,
This does not change the way Solr is distributed. Yes, this CAN BE used to
change the distribution in the future and that SHOULD BE done in a major
release. But
* You can't change the distribution AND introduce new capabilities in one go.
Your new distribution mechanism should be built on a tried and tested feature
* You can't deny features to users who wants to have an easier way to
install/update plugins.
The key is to incrementally build features , test them in the wild , fix issues
and move on. I don't think big bang releases of new features is a good idea.
That requires enormous amount of time and resources and we mostly get it wrong
and sometimes we end up building the wrong features and all the efforts go
waste.
This feature should be marked experimental so that users can try this and get
feedback and we can improve upon it and it's made ready for Solr to rely on
this for distribution
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945432#comment-16945432
]
Jan Høydahl commented on SOLR-13661:
{quote}New features can be introduced in any release, irrespective of how big a
feature is
{quote}
Of course. But being a totally new way of distributing Solr and having huge
impact on the whole user and wider dev community, I think it makes totally
sense to introduce pkg mgmt in a major release as beta, and use the following
releases to adjust and settle the APIs and solicit feedback from 3rd party
plugin devs etc. But I agree that, if handled carefully and labeled
"experimental", we may introduce this in a series of 8.x point releases, and
perhaps start moving features from core into packages in master, targeting a
slim solr-core in 9.0, which is a major "feature" in itself.
{quote}This change is not backward incompatible or it does not affect users of
current versions
{quote}
You mentioned that "" is incompatible with your class loader work. Can you
elaborate on that? Is it simply so that a collection using packages cannot also
use ""?
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945333#comment-16945333 ] ASF subversion and git services commented on SOLR-13661: Commit 515af6d3ecc5ea1b3754776f9ce4e39857f99734 in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=515af6d ] SOLR-13661: Reverting all half-baked stuff from SOLR-13707, SOLR-13659, SOLR-13565, SOLR-13650, SOLR-13710, SOLR-13721, SOLR-13637 All half baked package management and hot-classloading code reverted to allow for a fresh start. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945290#comment-16945290 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. Wrt 8.3 release and the half-merged stuff, may I suggest a revert and un-mark this as Blocker. Exactly, that's the plan. Noble is working on it. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945202#comment-16945202 ] Noble Paul commented on SOLR-13661: --- bq. In my opinion the package mgmt stuff don't belong in 8.x at all, That's not how our project works. New features can be introduced in any release, irrespective of how big a feature is. A feature is big or small has no relevance. However, backward incompatible changes are only introduced in major versions. This change is not backward incompatible or it does not affect users of current versions > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945145#comment-16945145 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. . In my opinion the package mgmt stuff don't belong in 8.x at all, it is a huge feature that should target 9.0.0(-alpha) Why do you think so? > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945137#comment-16945137
]
Jan Høydahl commented on SOLR-13661:
{quote}This shall be easier for reviewing.
[https://github.com/apache/lucene-solr/pull/915]
{quote}
Hmm, I find this monster PR915 not easier to review, so I think it is better to
do it per-feature and start with the fundamentals like FileStore, AnnotatedApi
etc and get those merged separately when they are stable.
Wrt 8.3 release and the half-merged stuff, may I suggest a revert and un-mark
this as Blocker. In my opinion the package mgmt stuff don't belong in 8.x at
all, it is a huge feature that should target 9.0.0(-alpha).
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942656#comment-16942656 ] Ishan Chattopadhyaya commented on SOLR-13661: - I have re-raised the same PR, but re-organized the commits into reverting previous committed code and all new code. This shall be easier for reviewing. https://github.com/apache/lucene-solr/pull/915 > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 1h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942569#comment-16942569 ] Noble Paul commented on SOLR-13661: --- As long as somebody says they are reviewing/ or planning to review I'll hold on > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 50m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942472#comment-16942472 ] David Wayne Smiley commented on SOLR-13661: --- Don't merge this to master. Please be patient for peer review. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 50m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942436#comment-16942436 ] Noble Paul commented on SOLR-13661: --- I'm code complete and I'll spend time to ensure that the tests are passing consistently. Please review & I plan to merge this to master shortly > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 50m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941551#comment-16941551 ] ASF subversion and git services commented on SOLR-13661: Commit 8f3b7133359d95c419bee3568863867d947f6775 in lucene-solr's branch refs/heads/jira/SOLR-13661 from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8f3b713 ] SOLR-13661:Changed the name of blob-store to 'filestore' . All APIs are updated to reflect this > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 50m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941355#comment-16941355 ] Ishan Chattopadhyaya commented on SOLR-13661: - As per a discussion in 8.3 release thread, branch cutting has been delayed for a week. Let us do the needful (either revert or review/merge this PR) by then. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 40m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941047#comment-16941047 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. We revert those changes. But they seem complicated enough that I don't want to attempt it. The commits are here: https://issues.apache.org/jira/browse/SOLR-13710 > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 40m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941012#comment-16941012 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. I want to be clear on one thing: The concern/frustration that Jan and I have on peer review is because this issue is not some ordinary JIRA issue. It's highly impactful to Solr. As-such, IMO peer review is required for at least the major ideas / high level, naming, CLI, release-plan. Getting into some small details, no, not needed. Thankfully the peer review is here now Thanks David and Jan for your help with reviews. I am glad to receive the reviews and the improvement that brings to our design and implementation. I shall update the design document with all the details that we discussed offline (on slack) that will make it easier to understand the workflows involved here. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 40m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941010#comment-16941010 ] Ishan Chattopadhyaya commented on SOLR-13661: - We are in a difficult situation here. There are some commits concerning the class loader changes and new blob store that are already in master and branch_8x. But I am supposed to cut the 8.3 branch today. Here are the options we have: # We revert those changes. But they seem complicated enough that I don't want to attempt it. # We review and merge https://github.com/apache/lucene-solr/pull/910 which will complete the blob store and class loader work, but might take 1-2 days to review? # We leave the unfinished code in the branch and disable the feature and cut the branch. [~noble.paul], [~dsmiley], [~janhoy] any thoughts? > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 40m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941007#comment-16941007 ] David Wayne Smiley commented on SOLR-13661: --- The design document is pretty fantastic in its overall scope (not too much or too little) and structure (easy to consume). Of course I have things inside to debate but it was a breath of fresh air to consume. I want to be clear on one thing: The concern/frustration that Jan and I have on peer review is because this issue is not some ordinary JIRA issue. It's highly impactful to Solr. As-such, IMO peer review is _required_ for at least the major ideas / high level, naming, CLI, release-plan. Getting into some small details, no, not needed. Thankfully the peer review is here now :-) > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 40m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940750#comment-16940750 ] ASF subversion and git services commented on SOLR-13661: Commit 7779be1017c93166cf10d8debc8765e7d121037c in lucene-solr's branch refs/heads/jira/SOLR-13661 from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7779be1 ] SOLR-13661: Changed the blob store to use sha256- as the blob id instead of just sha256 > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939153#comment-16939153 ] ASF subversion and git services commented on SOLR-13661: Commit 68232b0d746e9ade0f7019ab01db9f15e194abb8 in lucene-solr's branch refs/heads/jira/SOLR-13661 from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=68232b0 ] SOLR-13661: merge commit from SOLR-13722 branch > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 20m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938991#comment-16938991
]
Noble Paul commented on SOLR-13661:
---
bq.Simplicity should be front seat. Don't force users to have to add
{{package="my-pkg"}}
A name is something everyone needs . A package name is an extremely important
aspect. You cannot load reload plugins from a package we do not know the name
of the package. Just a class name means nothing. Isolated classloaders are
extremely important. Every sensible platform is built with isolation.
We can possibly later add a global feature called , {{config-package= mypkg}} .
This would mean that every plugin will load from {{mypkg}}. There is no reason
why it cannot be added later. But, not being able to load a plugins from
multiple packages is a strict NO. You are just ensuring that multiple packages
from multiple plugin writers cannot coexist.
Another issue id backward incompatibility. The new class loader design is
different and it can break current deploymemts
This new design totally disallows per core classloaders. This can be a problem
for users who use Solr with core level libs. So, it has to be backward
incompatble as well
bq.Robustness during upgrades is another concern. I don't see mentioned in the
design doc what happens during a Solr upgrade.
I'm not sure you have read the design properly. Robustness of design is the
paramount feature of this design. We went out of our way to ensure that
"update" is non-disruptive. Rolling restarts are NEVER REQUIRED. Solr behaves
very badly during rolling restarts, we lose shards/replicas or our overseer get
clogged with messages and needs manual intervention.
Overall feedback on your feedback [~janhoy]
I'm happy to receive and address feedback. We need to divide the following parts
# Enhancements which can be implemented with the current design . But the
current design is meeting the minimum viable product and does not obstruct or
hamper implementing an enhancement
# The current design is suboptimal or bad UX
# Comments on "inadequate eyes" etc. We are an OSS project and we don't work
in the same org. So collaboration is always "inadequate" . We should always
work towards having 100% collaboration in every single feature that we build.
The fact that we are discussing this here means that we can have collaboration
as long as people are interested in it. So, please refrain from giving this
"feedback". We should keep the comments short and sweet and make them
"actionable"
I'll give examples of #1 and #2
h3. using a Package.java to load plugins
This falls into #1
Is this a good suggestion? I would say it's questionable. But, I won't delve
into that here.
But, can we implement it if required in the future? Yes, of course. Does it
have to be discussed in this ticket? I would say it's out of the scope.
h3. Changing the blob names to hash-filename
This falls into #2
This is a usability issue. I had thought about this while building it, ab had
raised the concern in he design doc. It is probably something we should
address before committing this. If you have gone through the comments you would
have seen it.
Let's keep the feedback coming. But let's not distract from the current task at
hand. We don't have a working useful package management system today and a
minimum viable product is what we needs now. bells and whistles can be added
later.
We need the wheels, steering, transmission and seats of the car to be built
first. Yes , we can add seat belts , Air conditioning etc later.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938881#comment-16938881 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. Let's understand that this is not our hobby. It's a job. We are all able to do this because somebody is funding the development .When somebody is funding the development, they will have certain requirements and all their requirements need to be met. I'm sure every org works like that. Salesforce, Bloomberg, Apple , Lucidworks and all the big contributors are building big features to satisfy their businesses. I disagree with the spirit of this comment. I'd have done this work, irrespective of someone funding this development. For me, it is as much a hobby as much a job. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 20m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938870#comment-16938870 ] Ishan Chattopadhyaya commented on SOLR-13661: - bq. We went through deployment lifecycles of at least 3 of my clients, went through state of the art package management systems like apt, dnf etc. and thought through all potential usecases and future of Solr as a lean core with all non-essential features stripped out as packages in an Apache repository), as well as went through your talk+code+presentation+document. Our main focus is ease of use and a robust plugin lifecycle management experience. Your PoC was lacking some major pieces that we've covered meticulously here: security, efficient loading (without needing to restart nodes as in your PoC), ease of deployment, not requiring packages to depend on PF4J (your PoC forced users to add PF4J as a dependency, perhaps to facilitate the loading) etc. Also, went through some bits of the user experience of ES plugin system. [~jpountz], [~jim.ferenczi], [~mikemccand], would love your thoughts on this, based on your experience with ES. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 20m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938863#comment-16938863
]
Ishan Chattopadhyaya commented on SOLR-13661:
-
1.
{quote}too many decisions seem to be made with too few eyes
{quote}
Noble, Ishan, Jan, Andrzej, David. We've had at least these many "eyes". How
many more or who else are needed?
2.
{quote}"package" concept seems to be designed for ONE use case only, customer's
internal custom packages, with arbitrary local naming of repos and packages
{quote}
Not at all. Apache's contrib modules can easily be installed/deployed the same
way, they can be packages themselves.
{quote}before such a feature goes mainstream, the design should also include
converting some of our contrib modules to packages that we release as separate
binaries in the mirrors, and enable an "apache" Repo as default.
{quote}
Converting a contrib module needn't be a precursor to releasing this feature.
Moreover, without the Gradle work completed, I don't want to attempt to change
the build system too much (only to do it again later).
{quote}Perhaps that would mean some name spacing or name collision resolution
{quote}
Namespacing the packages is something we thought of. We already have two pieces
of information available in public APIs and all system boundaries/interfaces:
"repository" and "package-name". If we need to internally namespace the
packages, we can do so later. However, I don't think we need to do this: look
at dnf, apt etc. systems, and there's no concept of namespaces of packages.
That means, third parties should be careful enough not to name their packages
same as official packages.
3. Sure.
{quote}We need a plan for how 3rd party plugin developers can publish their
plugins on their own web site or on GitHub in a well defined way.
{quote}
I can add a document to that effect in our ref-guide. Initially, just the
repository structure documented in the design document will be supported.
Github support can be added subsequently.
4.
{quote}Hot/Cold deploy. I don't like systems where you, as part of the install
need to spin up a server.
{quote}
Spinning up a cluster prior to installing the packages is not "needed". Someone
can cold start a cluster with plugins pre-installed. Noble and I have both
documented those steps in the design doc as a reply to your comment.
6. That's a tradeoff. I initially raised the same point, but Noble suggested
that adding a new set of znode watchers per collection is an overhead. I'm +0
on your suggestion.
7. We have a disagreement here. Needing the user to edit solrconfig.xml
necessarily in order to specify which collection uses which package is bad from
simplicity standpoint. Please keep in mind that hand editing configset is an
expert feature.
8. No, we're not forcing regular users to do anything extra. Regular users
should be using config APIs to register/deregister their plugins. Expert users
are expert enough to just add an extra package name while they are hand editing
their solrconfig.xml. I think this is a very reasonable compromise.
9. Plan is to support both: (a) jar without manifest + external manifest, (b)
jar containing manifest.
The first scenario should be preferred in case of multi-jar packages.
10. Sure, I like the idea of supporting multiple jars as well as a zip
containing all the jars. It can be supported.
11. Plugin initialisation commands are not complex at all; they are typically
just regular config API commands to register the plugins. They are necessary
for pleasant adoption. A user just needs to install a package and specify which
collection he needs to deploy his package to. Simple! If we go by your idea,
then user would install the package but need to hand-edit the solrconfig.xml in
order to add the plugins from the package to his collection.
12. I agree. We thought a lot about how to support filenames, and all the
approaches seemed to have some deficiency. I've documented that as a comment in
the design document. Without an additional distributed KV datastore, this seems
hard to get right. The sha256.properties file approach will not work well,
since every node will have its own version of the file, and also maintaining
update consistency will be hard.
{quote}But I put a lot of careful thought into the POC which I feel is largely
lacking here
{quote}
I assure you that we've put a lot of thought into how we designed this. Our
main focus is ease of use and a robust plugin lifecycle management experience.
Your PoC was lacking some major pieces that we've covered meticulously here:
security, efficient loading (without needing to restart nodes as in your PoC),
ease of deployment, not requiring packages to depend on PF4J (your PoC forced
users to add PF4J as a dependency, perhaps to facilitate the loading) etc.
Just because I haven't documented each and every user story that you
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938830#comment-16938830
]
Jan Høydahl commented on SOLR-13661:
I have just looked at some of the code and will not have time for a more
thorough review until week after next.
Here is a list of my main concerns so far:
# My main concern is that too many decisions seem to be made with too few
eyes, combined with a goal of merging very soon.
# One example of "too few eyes" is that the "package" concept seems to be
designed for ONE use case only, customer's internal custom packages, with
arbitrary local naming of repos and packages. I think before such a feature
goes mainstream, the design should also include converting some of our contrib
modules to packages that we release as separate binaries in the mirrors, and
enable an "apache" Repo as default. That requires some more thought behind
stable name-spacing, so that e.g. “bin/solr install ltr” will mean the same for
all customers. Perhaps that would mean some name spacing or name collision
resolution, so if you have a custom local repo with a package also called
"ltr", then you get an error which can be resolved by qualifying the package
name like e.g. "apache:ltr" or "mylocalrepo:ltr".
# We need a plan for how 3rd party plugin developers can publish their plugins
on their own web site or on GitHub in a well defined way. The use of
pf4j-update lib takes care of much of this, and this is also something that can
be added incrementally, but the design needs to allow for this. My POC has a
RepositoryFactory class that parses the repo URL (e.g. "bin/solr plugin repo
add myrepo [https://host.com/repo/name];) and selects the
GitHubUpdateRepository if it is a GitHub URL, the ApacheMirrorsUpdateRepository
if it is an apache.org address, and the default site/FS repo else. Each of
these handle the download process and signature verification in a different way.
# Hot/Cold deploy. I don't like systems where you, as part of the install need
to spin up a server. We already have this with setting urlScheme in ZK for
HTTPS. But ideally it should be possible to do a Solr install including plugins
before you need to spin up Solr. Elasticsearch uses such a static plugin
installer (but also don't support hot install). Having a "staging" folder where
you can drop package ZIP files (or JARs) where the node can self-install
packages during first startup could be one way to handle this.
# Robustness during upgrades is another concern. I don't see mentioned in the
design doc what happens during a Solr upgrade. We should think through the
scenario for both minor and major version upgrade for Solr, and then I mean
rolling upgrade. Having ZK as only master for what version of a plugin should
be used is probably not sufficient, as during a rolling Solr upgrade, you could
have one node on 8.3 and another node on, say, 9.0. And you could have
packageA:1.0 installed but Solr9 requires v2.0 due to removal of some APIs or
what not. In the cold scenario (as in POC) you'd shut down a Solr node, upgrade
Solr, then run "bin/solr plugin upgrade outdated" before starting that node
again, and that would make sure it has the correct plugin version. Since you
cannot upgrade Solr while it is running, perhaps we need to hook in some
validation on node startup that it does not have any packages that won't work
with that Solr version, and refuse to start. And some way to have two versions
of a package installed at the same time, and then instead of using the latest,
the Solr node will select the newest version that is compatible. Then when that
node is upgraded it will select the new version of the plugin automatically
based on Version.java.
# Package system deserves its own Znode in Zookeeper instead of abusing
clusterProps
# I don't like the concept of an admin needing to "deploy" a package to a
collection using a command. Rather, the collections should require a set of
packages (optionally with min version) and fail to start if it is not available
in the system. If the package is available in the system, the collection should
gain access to the package(s) it required without running a deploy command.
# Simplicity should be front seat. Don't force users to have to add
{{package="my-pkg"}} wherever they today can say
{{class="com.example.MyPlugin"}}. This is what we have ResourceLoader and class
loaders for. If we cannot find {{com.example.MyPlugin}} in main class loader,
then hunt through every package class loader until you have a match, if no
match then throw ClassNotFound. (I never liked the {{runtimeLib=true}}
equivalent in the old blob store.)
# The package design says that a manifest is not required for a package and
that any plain jar can function as a package just by registering it manually.
That is ok as an alternative workflow, but most packages (and all
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937957#comment-16937957 ] Noble Paul commented on SOLR-13661: --- #1 & #2 can be implemented in the future and there is nothing in the current impl that prevents is from doing any of it. #3 is not something I'm sure about > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 10m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937954#comment-16937954 ] David Smiley commented on SOLR-13661: - There are a few particular things I'm not sure I've seen here yet that I want to ensure _could be supported_ (i.e. without major redesign/rework). (1) Declare plugin/package dependencies for the whole configSet, such as via additional options to solrconfig.xml "" tags. While this would not work with hot deployment of plugins since a core reload is needed, this would at least support the entire range of plugin interfaces we have that work from within a SolrCore, and thus support all of the schema as well _without_ making users add new tags to all these various elements. (2) Use the new blob store for _resources_ (data) as an alternative to the configSet. The configSet lives in ZooKeeper and as-such is a poor fit for, say, machine learning models, language models, or any other large immutable files needed within a running SolrCore. Such resource data might be versioned like plugins but would be considered data and not code. Plugins could declare dependencies on resources, or might be declared directly by the configSet in some manner. Detaching from the configSet is also good for re-use of large files across configs. I could imagine a SolrResourceLoader that interprets special prefixes like "blob://mymodel/foo.dat" or some-such to generically plug into our existing resource spots. (3) Pluggable blob store implementation and/or support for a shared file system across the cluster with a configurable path. > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > Time Spent: 10m > Remaining Estimate: 0h > > Here's the design doc: > https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935531#comment-16935531
]
Ishan Chattopadhyaya commented on SOLR-13661:
-
[~janhoy], [~dsmiley], [~erickerickson], please review the design.
Most of the class loader work is already done and committed as it was a
standalone piece. The package management stuff resides in two public branches
(SOLR-13722 and SOLR-13662) where Noble and I are collaborating.
{quote}So my criteria is that nothing we do should make "the silent majority"
of user's lives more difficult.
{quote}
[~erickerickson], This package management piece is going to be an opt-in and an
optional feature. It will not get in the way of any conceivable existing user
workflow (simple or complex). For those who will choose to use it, it will be a
charm and their lives will never be the same again!
{quote}I'm kind of sad to see this new JIRA not connecting to prior work at
all, it should be a best practice to research prior art or at least ask on the
list before starting a new Jira of this magnitude.
{quote}
[~janhoy] Initially, this work started with just trying to build the hot
loading capabilities. That was independent of anything that we had in Solr, so
Noble went ahead and committed that piece. As things progressed and I joined
forces, the story evolved into building a complete package management system.
In that pursuit, I did fully research your prior work. I asked you for the
code
https://issues.apache.org/jira/browse/SOLR-13661?focusedCommentId=16923943=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16923943,
went over it fully. I find your sadness completely baseless. While I couldn't
leverage your code piece directly, I borrowed the choice of your library
(PF4J-update) and rolled it into a prototype (branch jira/solr-13662).
{quote}I also don't like off-list development followed by huge code dumps in a
rapid pace.
{quote}
All development happens off-list; they happen in our Git repository and
referred to from our JIRA issues. Noble and I did exactly that: all code was
available in a public JIRA branch on our main Apache repository, right from the
day the code was ever written. The reason why you may have missed that is
likely because pushes to those branches did not emit a JIRA comment. As for
speed of development, "rapid" is cool ;)
{quote}To be frank, I have a very hard time grasping the 10.000ft vision behind
the current blob package effort, what the big picture is, how it will work our
for users (small and large), and what about 3rd party plugin developers etc
{quote}
Fair concern; hopefully the latest document will address your concerns. The big
picture remains the same as what you dreamed of; the eventual goal is for all
of the extraneous features of Solr to sit as packages that users can install
(from Apache repositories). A lite / safe Solr will be free from the optional
features. However, this work will take time (I plan to start on the splitting
up of Solr once the Gradle work is done). Meanwhile, the package manager will
be immensely useful for 3rd party packages and repositories.
{quote}I was also sad to see this new Jira get filed without any investigation
/ mention of previous efforts, and see a bunch of code fly in rapidly without
review. Slightly angry/frustrated even.
{quote}
Cute!
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935494#comment-16935494
]
Noble Paul commented on SOLR-13661:
---
{quote}lt. Sure, for those who need custom code we should provide ease-of-use,
but not, repeat NOT if it causes "regular" users grief.
{quote}
There are 4 types of users
# happy to use Solr OOTB
# Use only 3rd party plugins
# Write their own custom plugins
# Write and publish custom plugins for others to use
This ticket is not for #1 . They can peacefully happily use Solr without
thinking that this feature exists.
Is there anything in this ticket that suggests we are sacrificing the
usability for #2 in favor of #3 ? OTOH what I was trying to say is that we
shouldn't assume that an overwhelming majority of users belong to #2 based on a
sample survey and we have no other data to go by and make life easy for #2, #3
and #4
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
>
> The design doc is WIP.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935436#comment-16935436 ] Erick Erickson commented on SOLR-13661: --- [~noble.paul] Well, generally we only get called into complicated situations ;). It's always a bit weird. I'd _really_ like to see a comprehensive data set on how many people use Solr OOTB with nothing except maybe some configuration changes, IOW nothing custom at all. We're always in danger of our expectations being set by the clients we see, which are, IMO, not your "typical user". Solr "just works" and only users with complicated requirements need consultants. So my criteria is that nothing we do should make "the silent majority" of user's lives more difficult. Sure, for those who need custom code we should provide ease-of-use, but not, repeat NOT if it causes "regular" users grief. Here's where my age makes references like "the silent majority" obscure to you young whipper-snappers, especially anyone who wasn't following US politics in, ...er... about 1970. "Before you was born dude" in Paul Simon's song... > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > > The design doc is WIP. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935421#comment-16935421 ] Noble Paul commented on SOLR-13661: --- bq. The only poll I have seen in the Solr community on plugin usage is my own... This survey does not really reflect the reality. Every shop I have been in had their own custom plugins. We should not guide our decisions based on the results of this survey > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > Attachments: plugin-usage.png, repos.png > > > The design doc is WIP. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935156#comment-16935156
]
Jan Høydahl commented on SOLR-13661:
Thanks for adding some high level design spec. I really like the distributed
blob system, and how a node can bootstrap from other nodes by pulling needed
jars. And the isolated class loaders that can just swap out a plugin with a new
one. I also have some questions:
# How would e.g. SolrCell / Tika contrib be defined as a package? It has some
42 dependency jars. Guess through a blobs array?
# If you DELETE a package a package through {{/api/cluster/packages}}, how
will it know what blobs to delete and which to leave? Another package may have
the exact same dependency jar with the same hash
# Why do you need a {{package="pkg_name"}} in addition to
{{class="my.plugin.Class}}? That sounds very invasive, since we'd need to add
such package qualifiers to schema, security.json, solr.xml and virtually
everywhere? I guess it is an explicit way of selecting class loader and
obtaining isolation, but is there a smarter, less invasive way?
# You say that the version is not used by the system. So time comes to upgrade
Solr, how do you weed out the incompatible packages
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
> h2. What is a package?
> A package has 2 parts
> * The {{jar}} binaries which may contain standard plugins (or even other
> libraries). It does not expect anything special. Basically , any jar file can
> be added into a package
> * The configuration in ZK. The structure is as follows
> {code:json}
> {
> "name": "package-name",
> "version": "some-version-string",
> "blob": {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> }
> // use the following format for your multi jar package
> {
> "name": "package-name",
> "version": "some-version-string",
> "blobs": [
> {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> ]
> }
> {code}
> * {{*name*}} : this is identifier used to access the package
> * {{*version*}}: This is not consumed by the system. But, it is required so
> that the user can make out which version is being used
> * {{*blob/sha256*}}: The sha256 of the blob. You should have uploaded this
> blob into the new FS blob store
> * {{*blob/sig*}} : signature. ensure that your public keys are already there
> in {{/keys/exe}} in ZK
> * {{blob/name}} : a friendly identifier (not required)
> Voila! this is all you need to have a package in Solr
> Every package gets its own isolated unique classloader in a Solr node. The
> classloader is a child of the {{CoreContainer}} classloader. Basically, your
> classes have to be either in the root classloader or in the blobs in your
> package
> h2. What are the components of this system?
> h3. {{PackageResourceLoader}} class
> This is a standard {{SolrResourceLoader}} . It has all the blobs of a package
> in its classpath.
> h3. {{PackageManager}} class
> This is a registry of all Package classloaders . essentially a {{Map PackageResourceLoader>}}
> This class is responsible for listening to the state changes in ZK to the
> packages ADD/UPDATE/DELETE operations.
> * ADD : create a new {{PackageResourceLoader}} and add it to the registry
> * UPDATE: create a new {{PackageResourceLoader}}, update the registry and
> notify every listener
> * DELETE: remove a {{PackageResourceLoader}}
> h4. The Public APIs
> * {{GET /api/cluster/packages}} : Get a list of all registered packages
> * {{POST /api/cluster/packages}}
> ** {{add}} command: create a new package
> ** {{add}} command: update an existing package
> ** {{delete}} command: delete an existing package
> h4. configuring your plugins to use packages
> * in your {{solrconfig.xml}}
> {code:xml}
> package="pkg_name">
>
> {code}
> * using config API
> {code:xml}
> {
> "add-requesthandler": {"name":""/my_handler", "class":"my.full.ClassName",
> "package":"pkg_name"
> }
> }
> {code}
> h4. Example hot loading schema from package (TBD)
> {code:java}
>
> {code}
> h4. verifying your components
> {code:java}
> GET /solr//config//component-name?meta=true
> {code}
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934482#comment-16934482 ] Noble Paul commented on SOLR-13661: --- bh. Eeh, no - this is community over code. Yes, It's community over code. That's why we are all collaborating and discussing this. Most of the times nobody bothers to collaborate or comment. You did your part in your PoC and there was not enough feedback or collaboration. The community thing works only if people are willing and have time. All my development is done in public and I'm happy to discuss with anyone who is asking questions > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > > Solr needs a unified cohesive package management system so that users can > deploy/redeploy plugins in a safe manner. This is an umbrella issue to > eventually build that solution -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934478#comment-16934478
]
Noble Paul commented on SOLR-13661:
---
{quote}No, users won't have to do anything.
{quote}
Our users mostly write plugins and deploy their plugins. Third party plugins
are very few. So everyone who wishes to use your solution will have to write
extra code to comply with your package format
In this case, there is no change required to your existing plugin. You could
deploy everything in 3 steps
# POST your jar to /api/cluster/blob
# create a package using a command
# edit your solrconfig.xml to add package="package-name"
>From now on
* he has all his custom plugins loaded from this own jar
* he can update his plugins with just 2 commands with zero downtime
* if there is something wrong, he can rollback to the old plugin version in
exactly one command
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
> Labels: package
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934449#comment-16934449
]
Jan Høydahl commented on SOLR-13661:
{quote}The reason why we built hot deployment was that Solr was unstable when
we end up restarting a 1000 node cluster
{quote}
Hot deployment was identified as a future improvement in my POC. And don't get
me wrong, I'm very impressed by all your hot jar deployment work, it solves
many use cases. It also makes deploy a bit more messy since Solr needs to be up
and running as part of the deploy process itself. That is not very cool when
writing Dockerfiles and ansible deploy scripts. Some kind of plugins could be
fundamental to even getting the cluster started and so I'd prefer if there was
an "offline/static" way to pull and install certain packages before needing to
fire up the cluster. That was the *only* way to install packages in my POC, but
perhaps we could handle both. I like the fact that a node can come up and then
consult another node to fetch its missing plugins, after asking ZK about what
plugins are needed. That should make for a possibility of pulling a bunch of
plugin packages (zip/jar) into say /var/solr/plugins on a cold node, then
declaring those somewhere in ZK and when the cluster bootstraps it will
self-distribute those plugins across the entire cluster.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
> Labels: package
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934433#comment-16934433
]
Jan Høydahl commented on SOLR-13661:
{quote}Let's understand that this is not our hobby. It's a job
{quote}
Eeh, no - this is community over code.
To be frank, I have a very hard time grasping the 10.000ft vision behind the
current blob package effort, what the big picture is, how it will work our for
users (small and large), and what about 3rd party plugin developers etc (which
was my main focus for the Pf4J effort). This is a too big/important of an
improvement for Solr to just slide in under the radar in a series commits
without taking time to anchor the master plan and chosen design in the
community.
{quote}Jans plugin package mechanism required users to build their custom
packages. In this case you could deploy your existing jars without any
modification. You don't have to do anything special
{quote}
No, users won't have to do anything. We as committers publish the packages. 3rd
party plugin developers on GitHub publish their Solr plugins as packages. Users
just do {{bin/solr install }}. And the packages are not custom,
they follow the well-defined PF4J format. Which is pluggable and extensible.
One of the supported package formats is a JAR with a few extra lines in the
MANIFEST. Or an existing JAR with the manifest as a separate file. Or whatever
we agree to define. But there needs to be a manifest declaring the name,
version and dependencies, else it's not a package.
> A package management system for Solr
>
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
> Labels: package
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934139#comment-16934139 ] Jan Høydahl commented on SOLR-13661: If you don't feel like reading the 16 page design spec, here's a perhaps more consumable 60-slide slide deck [https://www.slideshare.net/janhoy/solrs-missing-plugin-ecosystem] and my Revolution talk [https://www.youtube.com/watch?v=hZKcULlZEQQ] > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > > Solr needs a unified cohesive package management system so that users can > deploy/redeploy plugins in a safe manner. This is an umbrella issue to > eventually build that solution -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933849#comment-16933849 ] Ishan Chattopadhyaya commented on SOLR-13661: - Thanks for your offer to help. As of right now, we're working on branches in SOLR-13722 and SOLR-13662. We quickly jotted down our next steps in this sheet: https://docs.google.com/spreadsheets/d/1qG1f-mO49lp9abh1XScufOTHpb-Sb0C6Y4WOsoJj3f4/edit?ts=5d7fb030#gid=0. We'll incorporate that into JIRA and commits into branches and eventually master. Would greatly appreciate your help in reviews! bq. I'll be landing some big commits to the branch and seeking reviews and feedback. As for timelines, we're hoping to wrap up the work left to do (in that sheet) by next week. That work includes some massive cleanups too (mainly on the code I have committed to the branch). bq. I originally did not start this as a proper package manager. the original idea was to just have a minimal improvement to hot load libraries Originally, Noble worked on the plugin loading mainly focussing on the "hot loading" part (without needing core reload). When I joined him, I convinced him that we have the opportunity to turn this into a complete user story by adding the package management part (on the lines of apt or dnf and similar to what we saw in Jan's prototype). > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > > Solr needs a unified cohesive package management system so that users can > deploy/redeploy plugins in a safe manner. This is an umbrella issue to > eventually build that solution -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933802#comment-16933802 ] Noble Paul commented on SOLR-13661: --- bq.did you forget (again) to resolve the issues you committed or are they deliberately open because they haven't landed on 8.x yet There was scope creep. I originally did not start this as a proper package manager. the original idea was to just have a minimal improvement to hot load libraries. Eventually, I decided to do a proper coherent story around this so that it is done well. As you may know , it's very difficult to get buy in from your employer to undertake a huge task of this magnitude. bq.There have been no code reviews yet on this committed code; right? I looked and see none but I wasn't exhaustive. I'm glad that there are people volunteering to review code. As long as there are reviewers willing to collaborate, It's a good thing to have. There largely has been developer apathy to new features (or even other changes). As we have got some visibility through the talk in activate conf, we have managed to spark some interest. Yes, reviews are welcome and seriously appreciated. I'll be landing some big commits to the branch and seeking reviews and feedback. cheers > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > > Solr needs a unified cohesive package management system so that users can > deploy/redeploy plugins in a safe manner. This is an umbrella issue to > eventually build that solution -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13661) A package management system for Solr
[ https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930799#comment-16930799 ] Ishan Chattopadhyaya commented on SOLR-13661: - Based on feedback from Activate conference attendees and some internal design discussions, here's what we plan to do: # Support for multiple jars in the same package # Packages should support verify operations after installation/updation # Put package metadata (like setup commands etc.) into the blob store > A package management system for Solr > > > Key: SOLR-13661 > URL: https://issues.apache.org/jira/browse/SOLR-13661 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Labels: package > > Solr needs a unified cohesive package management system so that users can > deploy/redeploy plugins in a safe manner. This is an umbrella issue to > eventually build that solution -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
