Looks like Jersey can build endpoints and APIs programmatically too, see https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest3x/resource-builder.html So I guess it can be done by hooking in at the right places if we want. It would be really encouraging if we could modernize this part of Solr, we have too many custom hard-to-grasp concepts everywhere that fends off new developers.
Jan > 30. nov. 2021 kl. 16:37 skrev Ishan Chattopadhyaya > <ichattopadhy...@gmail.com>: > > > > On Tue, Nov 30, 2021 at 8:44 PM Jason Gerlowski <gerlowsk...@gmail.com > <mailto:gerlowsk...@gmail.com>> wrote: > > We need to make an API work on a per core basis and cores can come up and > > go down randomly > > Ah, yeah. I'll admit I was overlooking some of the wrinkles around > custom request handlers etc. That is a problem for switching to some > off-the-shelf framework for sure. It may still be possible, but it's > definitely a big hurdle. I wonder how commonly users still take > advantage of the custom requestHandler stuff these days? > > Here's YASA using annotations: > https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45 > > <https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45> > > I haven't > run across many customers who used them extensively, but maybe that's > self-selecting in some way? > > > If SolrJ has a Jackson dependency, it can have a conflict (different > > versions) with the Jackson used by the client app > > SolrJ has a handful of dependencies - is there a reason we take this > approach with Jackson, but not with any of SolrJ's other deps? > (apache-httpcomponents, apache-commons, zookeeper-client, etc.) > > Granted, I get that we want SolrJ to be as painless as possible for > its users, and jar-conflicts are a part of that and really suck. But > there are good tools out there for handling jar conflicts, and > reimplementing chunks of library functionality just to avoid a > gradle-dep strikes me as unsustainable from a maintenance/community > perspective. Not to mention that forking/mimicking Jackson code > deprives our users of any efficiency/security improvements that > Jackson might get tomorrow or next week or next year. > > Unless there's some reason specific to Jackson I guess I just don't > get it. But if I'm the minority opinion on that, fair enough I guess. > > On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com > <mailto:noble.p...@gmail.com>> wrote: > > > > The @JsonProperty annotations are added because it's a dependency in SolrJ > > as well > > If SolrJ has a Jackson dependency, it can have a conflict (different > > versions) with the Jackson used by the client app. > > > > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com > > <mailto:noble.p...@gmail.com>> wrote: > >> > >> Also keep in mind that the same endpoint can be accessed with a core name > >> and a collection name prefixes. > >> > >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com > >> <mailto:noble.p...@gmail.com>> wrote: > >>> > >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ > >>> HttpSolrCall. A lot of our initializations occur there. > >>> > >>> We need to make an API work on a per core basis and cores can come up and > >>> go down randomly. So we need to register these endpoints on a core. > >>> > >>> I'm not sure if any framework can achieve the same. > >>> > >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com > >>> <mailto:gus.h...@gmail.com>> wrote: > >>>> > >>>> IIRC last time I looked restlet had the unsavory property of existing > >>>> outside of the SolrDispatchFilter, unlike everything else which made for > >>>> special cases because several things that probably ought to be their own > >>>> siervlet filters are glommed into SolrDispatchFilter, like security, > >>>> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad > >>>> if one could just wrap such filters around it too... > >>>> > >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowsk...@gmail.com > >>>> <mailto:gerlowsk...@gmail.com>> wrote: > >>>>> > >>>>> > These are minor improvements compared to a full rewrite of the entire > >>>>> > framework > >>>>> > >>>>> If you think data type support is minor, fair enough. But to clarify > >>>>> I'm not suggesting a rewrite - I'm suggesting using something that > >>>>> already exists off the shelf. Jersey (e.g.) itself provides the > >>>>> framework - there would be no "rewrite". > >>>>> > >>>>> re: past restlet use > >>>>> > >>>>> > It was not playing well with our security framework. The framework > >>>>> > was not working well with Solr APIs > >>>>> > >>>>> Ah, very interesting! Security isn't something Eric or I tackled in > >>>>> our little spike branch, but it's definitely a concern. Do you > >>>>> remember the specific concerns? Or recall where any of the discussion > >>>>> around this happened? > >>>>> > >>>>> Without the context of that past discussion, it seems like the > >>>>> "PermissionNameProvider" interface could be implemented just as well > >>>>> by a class with (e.g.) Jersey annotations as one with our own custom > >>>>> annotations. Certainly there'd need to be some > >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but > >>>>> nothing that feels insurmountable. > >>>>> > >>>>> Maybe I can try spiking it out soon and find the issues myself, but > >>>>> it'd be much easier if someone happens to remember and can save me the > >>>>> trouble :-p > >>>>> > >>>>> Best, > >>>>> > >>>>> Jason > >>>>> > >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com > >>>>> <mailto:noble.p...@gmail.com>> wrote: > >>>>> > > >>>>> > The Annotations framework was written after playing with other > >>>>> > frameworks. There were many shortcomings which were hard to overcome. > >>>>> > > >>>>> > The best example is a per collection API . How do you register an > >>>>> > endpoint for a collection/core ? > >>>>> > > >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com > >>>>> > <mailto:noble.p...@gmail.com>> wrote: > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski > >>>>> >> <gerlowsk...@gmail.com <mailto:gerlowsk...@gmail.com>> wrote: > >>>>> >>> > >>>>> >>> > Is there some problem with our annotations that we hope to solve > >>>>> >>> > using third party dependencies? > >>>>> >>> > >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust > >>>>> >>> solutions, whereas our annotations still need support added now and > >>>>> >>> then for even primitive data types like "long" (see SOLR-15619). > >>>>> >> > >>>>> >> These are minor improvements compared to a full rewrite of the > >>>>> >> entire framework > >>>>> >> > >>>>> >> > >>>>> >>> > >>>>> >>> Every JIRA spent doing basic stuff like that is time away from > >>>>> >>> improving Solr in some other way. > >>>>> >>> > >>>>> >>> So there are feature-gap/capabilities arguments for moving to a > >>>>> >>> third-party dep, sure. But, even if our annotations did everything > >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it. > >>>>> >>> Every LOC in our code base brings along with it some maintenance > >>>>> >>> cost: > >>>>> >>> it might have bugs, needs tested, takes time for new contributors to > >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of > >>>>> >>> that. If off-the-shelf is available for some given functionality, > >>>>> >>> we > >>>>> >>> should need a compelling reason to NOT use it. > >>>>> >>> > >>>>> >>> Lastly, I think there's an "approachability" argument for using > >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with > >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar > >>>>> >>> with > >>>>> >>> Solr's custom annotations. Using a well-known technology like > >>>>> >>> Jersey > >>>>> >>> would make Solr all the easier to approach and contribute to for > >>>>> >>> that > >>>>> >>> pool of developers. > >>>>> >>> > >>>>> >>> > By the way, we have used Restlet in the past and that has been a > >>>>> >>> > regrettable decision. > >>>>> >> > >>>>> >> > >>>>> >>> > >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a > >>>>> >>> pointer > >>>>> >>> to related discussions, or remember what made this "regrettable"? > >>>>> >>> All > >>>>> >>> the theoretical benefits in the world don't matter much if we've > >>>>> >>> already tried something like this in the past and decided against > >>>>> >>> it. > >>>>> >> > >>>>> >> > >>>>> >> It was not playing well with our security framework. The framework > >>>>> >> was not working well with Solr APIs > >>>>> >> > >>>>> >>> > >>>>> >>> (Unrelated - Happy Thanksgiving all!) > >>>>> >>> > >>>>> >>> Best, > >>>>> >>> > >>>>> >>> Jason > >>>>> >>> > >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com > >>>>> >>> <mailto:noble.p...@gmail.com>> wrote: > >>>>> >>> > > >>>>> >>> > Have you gone through an API written using the @EndPoint > >>>>> >>> > annotation? > >>>>> >>> > > >>>>> >>> > I strongly recommend that you do > >>>>> >>> > > >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh > >>>>> >>> > <ep...@opensourceconnections.com > >>>>> >>> > <mailto:ep...@opensourceconnections.com>> wrote: > >>>>> >>> >> > >>>>> >>> >> I have found our V2 API code to be very impenetrable to > >>>>> >>> >> understand. Part of it is how the code is intertwined with > >>>>> >>> >> support for V1, however it’s also because there aren’t really > >>>>> >>> >> resources to go look at to understand how it should work! > >>>>> >>> >> Maintaining the API should be very simple work, as they just > >>>>> >>> >> exist as a translation. The home grown stuff may make sense if > >>>>> >>> >> you are a super knowledgable Solr developer, but if you are just > >>>>> >>> >> a new person, it’s a lot harder to contribute. > >>>>> >>> >> > >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of > >>>>> >>> >> projects use it very successfully, and if I want to implement > >>>>> >>> >> something, well, there are lots of blogs and resources out there! > >>>>> >>> >> > >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what > >>>>> >>> >> lessons learned there might apply to adopting Jersey for API > >>>>> >>> >> support? Looking at > >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14659 > >>>>> >>> >> <https://issues.apache.org/jira/browse/SOLR-14659>, it was > >>>>> >>> >> partly deprecated because we were not using it to support all > >>>>> >>> >> the API, only the ManagedResource ones, and > >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14766 > >>>>> >>> >> <https://issues.apache.org/jira/browse/SOLR-14766> suggests that > >>>>> >>> >> RESTlet maybe was no longer being updated? One reason why we > >>>>> >>> >> spiked out Jersey was because of the broad support in the Java > >>>>> >>> >> world! Looking at how much work we have to do in the V2 API > >>>>> >>> >> world, we need a much broader pool of developers contributing to > >>>>> >>> >> get there! > >>>>> >>> >> > >>>>> >>> >> Related, are there specific features/aspects of our annotations > >>>>> >>> >> that enable things in Solr that couldn’t be done otherwise? > >>>>> >>> >> > >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya > >>>>> >>> >> <ichattopadhy...@gmail.com <mailto:ichattopadhy...@gmail.com>> > >>>>> >>> >> wrote: > >>>>> >>> >> > >>>>> >>> >> Is there some problem with our annotations that we hope to solve > >>>>> >>> >> using third party dependencies? > >>>>> >>> >> By the way, we have used Restlet in the past and that has been a > >>>>> >>> >> regrettable decision. > >>>>> >>> >> > >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski > >>>>> >>> >> <gerlowsk...@gmail.com <mailto:gerlowsk...@gmail.com>> wrote: > >>>>> >>> >>> > >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', > >>>>> >>> >>> etc.) has > >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. > >>>>> >>> >>> Having them > >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use > >>>>> >>> >>> our own > >>>>> >>> >>> annotations here instead of something off the shelf? > >>>>> >>> >>> > >>>>> >>> >>> What we have works well enough, but anything homegrown comes > >>>>> >>> >>> with more > >>>>> >>> >>> maintenance burden than we'd have if we used something off the > >>>>> >>> >>> shelf. > >>>>> >>> >>> There are plenty of well-used, active projects out there whose > >>>>> >>> >>> whole > >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing > >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them? > >>>>> >>> >>> > >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this > >>>>> >>> >>> route? > >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any > >>>>> >>> >>> context on > >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations > >>>>> >>> >>> originally? > >>>>> >>> >>> > >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got > >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much > >>>>> >>> >>> trouble. > >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's > >>>>> >>> >>> still a lot > >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but > >>>>> >>> >>> we both > >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of > >>>>> >>> >>> all, > >>>>> >>> >>> APIs using our current homegrown annotation framework the > >>>>> >>> >>> switchover > >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey > >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that > >>>>> >>> >>> much. > >>>>> >>> >>> > >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up > >>>>> >>> >>> with the > >>>>> >>> >>> annotation setup we use today! > >>>>> >>> >>> > >>>>> >>> >>> Best, > >>>>> >>> >>> > >>>>> >>> >>> Jason > >>>>> >>> >>> > >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 > >>>>> >>> >>> <https://issues.apache.org/jira/browse/SOLR-15182> (and > >>>>> >>> >>> children) > >>>>> >>> >>> [2] > >>>>> >>> >>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E > >>>>> >>> >>> > >>>>> >>> >>> <http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E> > >>>>> >>> >>> [3] > >>>>> >>> >>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis > >>>>> >>> >>> > >>>>> >>> >>> <https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis>. > >>>>> >>> >>> > >>>>> >>> >>> --------------------------------------------------------------------- > >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> >>> >>> <mailto:dev-unsubscr...@solr.apache.org> > >>>>> >>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> >>> >>> <mailto:dev-h...@solr.apache.org> > >>>>> >>> >>> > >>>>> >>> >> > >>>>> >>> >> _______________________ > >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | > >>>>> >>> >> 434.466.1467 | http://www.opensourceconnections.com > >>>>> >>> >> <http://www.opensourceconnections.com/> | My Free/Busy > >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > >>>>> >>> >> This e-mail and all contents, including attachments, is > >>>>> >>> >> considered to be Company Confidential unless explicitly stated > >>>>> >>> >> otherwise, regardless of whether attachments are marked as such. > >>>>> >>> >> > >>>>> >>> > >>>>> >>> --------------------------------------------------------------------- > >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> >>> <mailto:dev-unsubscr...@solr.apache.org> > >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> >>> <mailto:dev-h...@solr.apache.org> > >>>>> >>> > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> ----------------------------------------------------- > >>>>> >> Noble Paul > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > ----------------------------------------------------- > >>>>> > Noble Paul > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> <mailto:dev-unsubscr...@solr.apache.org> > >>>>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> <mailto:dev-h...@solr.apache.org> > >>>>> > >>>> > >>>> > >>>> -- > >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) > >>>> http://www.the111shift.com <http://www.the111shift.com/> (play) > >>> > >>> > >>> > >>> -- > >>> ----------------------------------------------------- > >>> Noble Paul > >> > >> > >> > >> -- > >> ----------------------------------------------------- > >> Noble Paul > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <mailto:dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <mailto:dev-h...@solr.apache.org> >
