> We need to make an API work on a per core basis and cores can come up and go > down randomly
Ah, yeah. I'll admit I was overlooking some of the wrinkles around custom request handlers etc. That is a problem for switching to some off-the-shelf framework for sure. It may still be possible, but it's definitely a big hurdle. I wonder how commonly users still take advantage of the custom requestHandler stuff these days? I haven't run across many customers who used them extensively, but maybe that's self-selecting in some way? > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app SolrJ has a handful of dependencies - is there a reason we take this approach with Jackson, but not with any of SolrJ's other deps? (apache-httpcomponents, apache-commons, zookeeper-client, etc.) Granted, I get that we want SolrJ to be as painless as possible for its users, and jar-conflicts are a part of that and really suck. But there are good tools out there for handling jar conflicts, and reimplementing chunks of library functionality just to avoid a gradle-dep strikes me as unsustainable from a maintenance/community perspective. Not to mention that forking/mimicking Jackson code deprives our users of any efficiency/security improvements that Jackson might get tomorrow or next week or next year. Unless there's some reason specific to Jackson I guess I just don't get it. But if I'm the minority opinion on that, fair enough I guess. On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> wrote: > > The @JsonProperty annotations are added because it's a dependency in SolrJ as > well > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app. > > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> wrote: >> >> Also keep in mind that the same endpoint can be accessed with a core name >> and a collection name prefixes. >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> wrote: >>> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ >>> HttpSolrCall. A lot of our initializations occur there. >>> >>> We need to make an API work on a per core basis and cores can come up and >>> go down randomly. So we need to register these endpoints on a core. >>> >>> I'm not sure if any framework can achieve the same. >>> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> wrote: >>>> >>>> IIRC last time I looked restlet had the unsavory property of existing >>>> outside of the SolrDispatchFilter, unlike everything else which made for >>>> special cases because several things that probably ought to be their own >>>> siervlet filters are glommed into SolrDispatchFilter, like security, >>>> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if >>>> one could just wrap such filters around it too... >>>> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowsk...@gmail.com> >>>> wrote: >>>>> >>>>> > These are minor improvements compared to a full rewrite of the entire >>>>> > framework >>>>> >>>>> If you think data type support is minor, fair enough. But to clarify >>>>> I'm not suggesting a rewrite - I'm suggesting using something that >>>>> already exists off the shelf. Jersey (e.g.) itself provides the >>>>> framework - there would be no "rewrite". >>>>> >>>>> re: past restlet use >>>>> >>>>> > It was not playing well with our security framework. The framework was >>>>> > not working well with Solr APIs >>>>> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in >>>>> our little spike branch, but it's definitely a concern. Do you >>>>> remember the specific concerns? Or recall where any of the discussion >>>>> around this happened? >>>>> >>>>> Without the context of that past discussion, it seems like the >>>>> "PermissionNameProvider" interface could be implemented just as well >>>>> by a class with (e.g.) Jersey annotations as one with our own custom >>>>> annotations. Certainly there'd need to be some >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but >>>>> nothing that feels insurmountable. >>>>> >>>>> Maybe I can try spiking it out soon and find the issues myself, but >>>>> it'd be much easier if someone happens to remember and can save me the >>>>> trouble :-p >>>>> >>>>> Best, >>>>> >>>>> Jason >>>>> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> wrote: >>>>> > >>>>> > The Annotations framework was written after playing with other >>>>> > frameworks. There were many shortcomings which were hard to overcome. >>>>> > >>>>> > The best example is a per collection API . How do you register an >>>>> > endpoint for a collection/core ? >>>>> > >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com> wrote: >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski >>>>> >> <gerlowsk...@gmail.com> wrote: >>>>> >>> >>>>> >>> > Is there some problem with our annotations that we hope to solve >>>>> >>> > using third party dependencies? >>>>> >>> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust >>>>> >>> solutions, whereas our annotations still need support added now and >>>>> >>> then for even primitive data types like "long" (see SOLR-15619). >>>>> >> >>>>> >> These are minor improvements compared to a full rewrite of the entire >>>>> >> framework >>>>> >> >>>>> >> >>>>> >>> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from >>>>> >>> improving Solr in some other way. >>>>> >>> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a >>>>> >>> third-party dep, sure. But, even if our annotations did everything >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it. >>>>> >>> Every LOC in our code base brings along with it some maintenance cost: >>>>> >>> it might have bugs, needs tested, takes time for new contributors to >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of >>>>> >>> that. If off-the-shelf is available for some given functionality, we >>>>> >>> should need a compelling reason to NOT use it. >>>>> >>> >>>>> >>> Lastly, I think there's an "approachability" argument for using >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey >>>>> >>> would make Solr all the easier to approach and contribute to for that >>>>> >>> pool of developers. >>>>> >>> >>>>> >>> > By the way, we have used Restlet in the past and that has been a >>>>> >>> > regrettable decision. >>>>> >> >>>>> >> >>>>> >>> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer >>>>> >>> to related discussions, or remember what made this "regrettable"? All >>>>> >>> the theoretical benefits in the world don't matter much if we've >>>>> >>> already tried something like this in the past and decided against it. >>>>> >> >>>>> >> >>>>> >> It was not playing well with our security framework. The framework was >>>>> >> not working well with Solr APIs >>>>> >> >>>>> >>> >>>>> >>> (Unrelated - Happy Thanksgiving all!) >>>>> >>> >>>>> >>> Best, >>>>> >>> >>>>> >>> Jason >>>>> >>> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com> >>>>> >>> wrote: >>>>> >>> > >>>>> >>> > Have you gone through an API written using the @EndPoint annotation? >>>>> >>> > >>>>> >>> > I strongly recommend that you do >>>>> >>> > >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh >>>>> >>> > <ep...@opensourceconnections.com> wrote: >>>>> >>> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to >>>>> >>> >> understand. Part of it is how the code is intertwined with >>>>> >>> >> support for V1, however it’s also because there aren’t really >>>>> >>> >> resources to go look at to understand how it should work! >>>>> >>> >> Maintaining the API should be very simple work, as they just exist >>>>> >>> >> as a translation. The home grown stuff may make sense if you are >>>>> >>> >> a super knowledgable Solr developer, but if you are just a new >>>>> >>> >> person, it’s a lot harder to contribute. >>>>> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of >>>>> >>> >> projects use it very successfully, and if I want to implement >>>>> >>> >> something, well, there are lots of blogs and resources out there! >>>>> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what >>>>> >>> >> lessons learned there might apply to adopting Jersey for API >>>>> >>> >> support? Looking at >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14659, it was partly >>>>> >>> >> deprecated because we were not using it to support all the API, >>>>> >>> >> only the ManagedResource ones, and >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14766 suggests that >>>>> >>> >> RESTlet maybe was no longer being updated? One reason why we >>>>> >>> >> spiked out Jersey was because of the broad support in the Java >>>>> >>> >> world! Looking at how much work we have to do in the V2 API >>>>> >>> >> world, we need a much broader pool of developers contributing to >>>>> >>> >> get there! >>>>> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our annotations >>>>> >>> >> that enable things in Solr that couldn’t be done otherwise? >>>>> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya >>>>> >>> >> <ichattopadhy...@gmail.com> wrote: >>>>> >>> >> >>>>> >>> >> Is there some problem with our annotations that we hope to solve >>>>> >>> >> using third party dependencies? >>>>> >>> >> By the way, we have used Restlet in the past and that has been a >>>>> >>> >> regrettable decision. >>>>> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski >>>>> >>> >> <gerlowsk...@gmail.com> wrote: >>>>> >>> >>> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', >>>>> >>> >>> etc.) has >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having >>>>> >>> >>> them >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use >>>>> >>> >>> our own >>>>> >>> >>> annotations here instead of something off the shelf? >>>>> >>> >>> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with >>>>> >>> >>> more >>>>> >>> >>> maintenance burden than we'd have if we used something off the >>>>> >>> >>> shelf. >>>>> >>> >>> There are plenty of well-used, active projects out there whose >>>>> >>> >>> whole >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them? >>>>> >>> >>> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this >>>>> >>> >>> route? >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context >>>>> >>> >>> on >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations >>>>> >>> >>> originally? >>>>> >>> >>> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much >>>>> >>> >>> trouble. >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still >>>>> >>> >>> a lot >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we >>>>> >>> >>> both >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of >>>>> >>> >>> all, >>>>> >>> >>> APIs using our current homegrown annotation framework the >>>>> >>> >>> switchover >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much. >>>>> >>> >>> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with >>>>> >>> >>> the >>>>> >>> >>> annotation setup we use today! >>>>> >>> >>> >>>>> >>> >>> Best, >>>>> >>> >>> >>>>> >>> >>> Jason >>>>> >>> >>> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and >>>>> >>> >>> children) >>>>> >>> >>> [2] >>>>> >>> >>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E >>>>> >>> >>> [3] >>>>> >>> >>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. >>>>> >>> >>> >>>>> >>> >>> --------------------------------------------------------------------- >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>>> >>> >>> For additional commands, e-mail: dev-h...@solr.apache.org >>>>> >>> >>> >>>>> >>> >> >>>>> >>> >> _______________________ >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | >>>>> >>> >> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed >>>>> >>> >> This e-mail and all contents, including attachments, is considered >>>>> >>> >> to be Company Confidential unless explicitly stated otherwise, >>>>> >>> >> regardless of whether attachments are marked as such. >>>>> >>> >> >>>>> >>> >>>>> >>> --------------------------------------------------------------------- >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org >>>>> >>> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> ----------------------------------------------------- >>>>> >> Noble Paul >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > ----------------------------------------------------- >>>>> > Noble Paul >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>>> For additional commands, e-mail: dev-h...@solr.apache.org >>>>> >>>> >>>> >>>> -- >>>> http://www.needhamsoftware.com (work) >>>> http://www.the111shift.com (play) >>> >>> >>> >>> -- >>> ----------------------------------------------------- >>> Noble Paul >> >> >> >> -- >> ----------------------------------------------------- >> Noble Paul --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
