[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938863#comment-16938863
]
Ishan Chattopadhyaya edited comment on SOLR-13661 at 9/26/19 6:24 PM:
----------------------------------------------------------------------
1.
{quote}too many decisions seem to be made with too few eyesÂ
{quote}
Noble, Ishan, Jan, Andrzej, David. We've had at least these many "eyes". How
many more or who else are needed?
2.
{quote}"package" concept seems to be designed for ONE use case only, customer's
internal custom packages, with arbitrary local naming of repos and packages
{quote}
Not at all. Apache's contrib modules can easily be installed/deployed the same
way, they can be packages themselves. End of the day, a "package" is just a jar
and some metadata. A repository is just a named location containing packages.
There is no "arbitrary local naming of repos and packages", those aspects are
left to the plugin writer (which can be Apache as well in case of official
contrib modules).
{quote}before such a feature goes mainstream, the design should also include
converting some of our contrib modules to packages that we release as separate
binaries in the mirrors, and enable an "apache" Repo as default.
{quote}
Converting a contrib module needn't be a precursor to releasing this feature.
Moreover, without the Gradle work completed, I don't want to attempt to change
the build system too much (only to do it again later). The system design
supports us doing so, and it can be taken up later.
{quote}Perhaps that would mean some name spacing or name collision resolution
{quote}
Namespacing the packages is something we thought of. We already have two pieces
of information available in public APIs and all system boundaries/interfaces:
"repository" and "package-name". If we need to internally namespace the
packages, we can do so later. However, I don't think we need to do this: look
at dnf, apt etc. systems, and there's no concept of namespaces of packages.
That means, third parties should be careful enough not to name their packages
same as official packages.
3. Sure.
{quote}We need a plan for how 3rd party plugin developers can publish their
plugins on their own web site or on GitHub in a well defined way.
{quote}
I can add a document to that effect in our ref-guide. Initially, just the
repository structure documented in the design document will be supported.
Github support can be added subsequently.
4.
{quote}Hot/Cold deploy. I don't like systems where you, as part of the install
need to spin up a server.
{quote}
Spinning up a cluster prior to installing the packages is not "needed". Someone
can cold start a cluster with plugins pre-installed. Noble and I have both
documented those steps in the design doc as a reply to your comment.
6. That's a tradeoff. I initially raised the same point, but Noble suggested
that adding a new set of znode watchers per collection is an overhead. I'm +0
on your suggestion.
7. We have a disagreement here. Needing the user to edit solrconfig.xml
necessarily in order to specify which collection uses which package is bad from
simplicity standpoint. Please keep in mind that hand editing configset is an
expert feature.
8. No, we're not forcing regular users to do anything extra. Regular users
should be using config APIs to register/deregister their plugins. Expert users
are expert enough to just add an extra package name while they are hand editing
their solrconfig.xml. I think this is a very reasonable compromise.
9. Plan is to support both: (a) jar without manifest + external manifest, (b)
jar containing manifest.
The first scenario should be preferred in case of multi-jar packages.
10. Sure, I like the idea of supporting multiple jars as well as a zip
containing all the jars. It can be supported.
11. Plugin initialisation commands are not complex at all; they are typically
just regular config API commands to register the plugins. They are necessary
for pleasant adoption. A user just needs to install a package and specify which
collection he needs to deploy his package to. Simple! If we go by your idea,
then user would install the package but need to hand-edit the solrconfig.xml in
order to add the plugins from the package to his collection.
12. I agree. We thought a lot about how to support filenames, and all the
approaches seemed to have some deficiency. I've documented that as a comment in
the design document. Without an additional distributed KV datastore, this seems
hard to get right. The sha256.properties file approach will not work well,
since every node will have its own version of the file, and also maintaining
update consistency will be hard.
{quote}But I put a lot of careful thought into the POC which I feel is largely
lacking here
{quote}
I assure you that we've put a lot of thought into how we designed this. Our
main focus is ease of use and a robust plugin lifecycle management experience.
Your PoC was lacking some major pieces that we've covered meticulously here:
security, efficient loading (without needing to restart nodes as in your PoC),
ease of deployment, not requiring packages to depend on PF4J (your PoC forced
users to add PF4J as a dependency, perhaps to facilitate the loading) etc.
Just because I haven't documented each and every user story that you brought up
(helpfully so) doesn't mean "careful thought .. is largely lacking" in the
*design*. It could just be a limitation of how we've documented it; but I'm
open to address any major design concern. IMHO, the document has sufficient
details not to overwhelm a reader (as opposed to the original 16 page document
that you authored), and yet give the reader a complete idea of the system
design. I agree that the design document is weak in discussing each and every
user story, but that was not the goal of that document: we can document user
stories in the ref guide or separately (in an appendix or another document).
{quote}I'm pretty sure we as a community will arrive at a better design by
considering and discussing more use cases than the one use case/need of your
employer.
{quote}
Feel free to bring up and issues with the current design and I'll be happy to
answer them.
{quote}than the one use case/need of your employer
{quote}
I don't see why you're bothered about my employer, or you think there's just
"one use case/need". It is my responsibility to ensure all usecases are met,
from community as well as third party plugin writers' point of view.
was (Author: ichattopadhyaya):
1.
{quote}too many decisions seem to be made with too few eyesÂ
{quote}
Noble, Ishan, Jan, Andrzej, David. We've had at least these many "eyes". How
many more or who else are needed?
2.
{quote}"package" concept seems to be designed for ONE use case only, customer's
internal custom packages, with arbitrary local naming of repos and packages
{quote}
Not at all. Apache's contrib modules can easily be installed/deployed the same
way, they can be packages themselves. End of the day, a "package" is just a jar
and some metadata. A repository is just a named location containing packages.
There is no "arbitrary local naming of repos and packages", those aspects are
left to the plugin writer (which can be Apache as well in case of official
contrib modules).
{quote}before such a feature goes mainstream, the design should also include
converting some of our contrib modules to packages that we release as separate
binaries in the mirrors, and enable an "apache" Repo as default.
{quote}
Converting a contrib module needn't be a precursor to releasing this feature.
Moreover, without the Gradle work completed, I don't want to attempt to change
the build system too much (only to do it again later).
{quote}Perhaps that would mean some name spacing or name collision resolution
{quote}
Namespacing the packages is something we thought of. We already have two pieces
of information available in public APIs and all system boundaries/interfaces:
"repository" and "package-name". If we need to internally namespace the
packages, we can do so later. However, I don't think we need to do this: look
at dnf, apt etc. systems, and there's no concept of namespaces of packages.
That means, third parties should be careful enough not to name their packages
same as official packages.
3. Sure.
{quote}We need a plan for how 3rd party plugin developers can publish their
plugins on their own web site or on GitHub in a well defined way.
{quote}
I can add a document to that effect in our ref-guide. Initially, just the
repository structure documented in the design document will be supported.
Github support can be added subsequently.
4.
{quote}Hot/Cold deploy. I don't like systems where you, as part of the install
need to spin up a server.
{quote}
Spinning up a cluster prior to installing the packages is not "needed". Someone
can cold start a cluster with plugins pre-installed. Noble and I have both
documented those steps in the design doc as a reply to your comment.
6. That's a tradeoff. I initially raised the same point, but Noble suggested
that adding a new set of znode watchers per collection is an overhead. I'm +0
on your suggestion.
7. We have a disagreement here. Needing the user to edit solrconfig.xml
necessarily in order to specify which collection uses which package is bad from
simplicity standpoint. Please keep in mind that hand editing configset is an
expert feature.
8. No, we're not forcing regular users to do anything extra. Regular users
should be using config APIs to register/deregister their plugins. Expert users
are expert enough to just add an extra package name while they are hand editing
their solrconfig.xml. I think this is a very reasonable compromise.
9. Plan is to support both: (a) jar without manifest + external manifest, (b)
jar containing manifest.
The first scenario should be preferred in case of multi-jar packages.
10. Sure, I like the idea of supporting multiple jars as well as a zip
containing all the jars. It can be supported.
11. Plugin initialisation commands are not complex at all; they are typically
just regular config API commands to register the plugins. They are necessary
for pleasant adoption. A user just needs to install a package and specify which
collection he needs to deploy his package to. Simple! If we go by your idea,
then user would install the package but need to hand-edit the solrconfig.xml in
order to add the plugins from the package to his collection.
12. I agree. We thought a lot about how to support filenames, and all the
approaches seemed to have some deficiency. I've documented that as a comment in
the design document. Without an additional distributed KV datastore, this seems
hard to get right. The sha256.properties file approach will not work well,
since every node will have its own version of the file, and also maintaining
update consistency will be hard.
{quote}But I put a lot of careful thought into the POC which I feel is largely
lacking here
{quote}
I assure you that we've put a lot of thought into how we designed this. Our
main focus is ease of use and a robust plugin lifecycle management experience.
Your PoC was lacking some major pieces that we've covered meticulously here:
security, efficient loading (without needing to restart nodes as in your PoC),
ease of deployment, not requiring packages to depend on PF4J (your PoC forced
users to add PF4J as a dependency, perhaps to facilitate the loading) etc.
Just because I haven't documented each and every user story that you brought up
(helpfully so) doesn't mean "careful thought .. is largely lacking" in the
*design*. It could just be a limitation of how we've documented it; but I'm
open to address any major design concern. IMHO, the document has sufficient
details not to overwhelm a reader (as opposed to the original 16 page document
that you authored), and yet give the reader a complete idea of the system
design. I agree that the design document is weak in discussing each and every
user story, but that was not the goal of that document: we can document user
stories in the ref guide or separately (in an appendix or another document).
{quote}I'm pretty sure we as a community will arrive at a better design by
considering and discussing more use cases than the one use case/need of your
employer.
{quote}
Feel free to bring up and issues with the current design and I'll be happy to
answer them.
{quote}than the one use case/need of your employer
{quote}
I don't see why you're bothered about my employer, or you think there's just
"one use case/need". It is my responsibility to ensure all usecases are met,
from community as well as third party plugin writers' point of view.
> A package management system for Solr
> ------------------------------------
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Noble Paul
> Assignee: Ishan Chattopadhyaya
> Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Here's the design doc:
> https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
