On Tue, Nov 30, 2021 at 8:44 PM Jason Gerlowski <gerlowsk...@gmail.com> wrote:
> > We need to make an API work on a per core basis and cores can come up > and go down randomly > > Ah, yeah. I'll admit I was overlooking some of the wrinkles around > custom request handlers etc. That is a problem for switching to some > off-the-shelf framework for sure. It may still be possible, but it's > definitely a big hurdle. I wonder how commonly users still take > advantage of the custom requestHandler stuff these days? Here's YASA using annotations: https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45 > I haven't > run across many customers who used them extensively, but maybe that's > self-selecting in some way? > > > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app > > SolrJ has a handful of dependencies - is there a reason we take this > approach with Jackson, but not with any of SolrJ's other deps? > (apache-httpcomponents, apache-commons, zookeeper-client, etc.) > > Granted, I get that we want SolrJ to be as painless as possible for > its users, and jar-conflicts are a part of that and really suck. But > there are good tools out there for handling jar conflicts, and > reimplementing chunks of library functionality just to avoid a > gradle-dep strikes me as unsustainable from a maintenance/community > perspective. Not to mention that forking/mimicking Jackson code > deprives our users of any efficiency/security improvements that > Jackson might get tomorrow or next week or next year. > > Unless there's some reason specific to Jackson I guess I just don't > get it. But if I'm the minority opinion on that, fair enough I guess. > > On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> wrote: > > > > The @JsonProperty annotations are added because it's a dependency in > SolrJ as well > > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app. > > > > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> wrote: > >> > >> Also keep in mind that the same endpoint can be accessed with a core > name and a collection name prefixes. > >> > >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> > wrote: > >>> > >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ > HttpSolrCall. A lot of our initializations occur there. > >>> > >>> We need to make an API work on a per core basis and cores can come up > and go down randomly. So we need to register these endpoints on a core. > >>> > >>> I'm not sure if any framework can achieve the same. > >>> > >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> wrote: > >>>> > >>>> IIRC last time I looked restlet had the unsavory property of existing > outside of the SolrDispatchFilter, unlike everything else which made for > special cases because several things that probably ought to be their own > siervlet filters are glommed into SolrDispatchFilter, like security, > tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if > one could just wrap such filters around it too... > >>>> > >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >>>>> > >>>>> > These are minor improvements compared to a full rewrite of the > entire framework > >>>>> > >>>>> If you think data type support is minor, fair enough. But to clarify > >>>>> I'm not suggesting a rewrite - I'm suggesting using something that > >>>>> already exists off the shelf. Jersey (e.g.) itself provides the > >>>>> framework - there would be no "rewrite". > >>>>> > >>>>> re: past restlet use > >>>>> > >>>>> > It was not playing well with our security framework. The framework > was not working well with Solr APIs > >>>>> > >>>>> Ah, very interesting! Security isn't something Eric or I tackled in > >>>>> our little spike branch, but it's definitely a concern. Do you > >>>>> remember the specific concerns? Or recall where any of the > discussion > >>>>> around this happened? > >>>>> > >>>>> Without the context of that past discussion, it seems like the > >>>>> "PermissionNameProvider" interface could be implemented just as well > >>>>> by a class with (e.g.) Jersey annotations as one with our own custom > >>>>> annotations. Certainly there'd need to be some > >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but > >>>>> nothing that feels insurmountable. > >>>>> > >>>>> Maybe I can try spiking it out soon and find the issues myself, but > >>>>> it'd be much easier if someone happens to remember and can save me > the > >>>>> trouble :-p > >>>>> > >>>>> Best, > >>>>> > >>>>> Jason > >>>>> > >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> > wrote: > >>>>> > > >>>>> > The Annotations framework was written after playing with other > frameworks. There were many shortcomings which were hard to overcome. > >>>>> > > >>>>> > The best example is a per collection API . How do you register an > endpoint for a collection/core ? > >>>>> > > >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com> > wrote: > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >>>>> >>> > >>>>> >>> > Is there some problem with our annotations that we hope to > solve using third party dependencies? > >>>>> >>> > >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust > >>>>> >>> solutions, whereas our annotations still need support added now > and > >>>>> >>> then for even primitive data types like "long" (see SOLR-15619). > >>>>> >> > >>>>> >> These are minor improvements compared to a full rewrite of the > entire framework > >>>>> >> > >>>>> >> > >>>>> >>> > >>>>> >>> Every JIRA spent doing basic stuff like that is time away from > >>>>> >>> improving Solr in some other way. > >>>>> >>> > >>>>> >>> So there are feature-gap/capabilities arguments for moving to a > >>>>> >>> third-party dep, sure. But, even if our annotations did > everything > >>>>> >>> Jersey+Jackson do today, I think switching would still be worth > it. > >>>>> >>> Every LOC in our code base brings along with it some maintenance > cost: > >>>>> >>> it might have bugs, needs tested, takes time for new > contributors to > >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch > of > >>>>> >>> that. If off-the-shelf is available for some given > functionality, we > >>>>> >>> should need a compelling reason to NOT use it. > >>>>> >>> > >>>>> >>> Lastly, I think there's an "approachability" argument for using > >>>>> >>> off-the-shelf. Thousands of developers out there are familiar > with > >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) > familiar with > >>>>> >>> Solr's custom annotations. Using a well-known technology like > Jersey > >>>>> >>> would make Solr all the easier to approach and contribute to for > that > >>>>> >>> pool of developers. > >>>>> >>> > >>>>> >>> > By the way, we have used Restlet in the past and that has been > a regrettable decision. > >>>>> >> > >>>>> >> > >>>>> >>> > >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a > pointer > >>>>> >>> to related discussions, or remember what made this > "regrettable"? All > >>>>> >>> the theoretical benefits in the world don't matter much if we've > >>>>> >>> already tried something like this in the past and decided > against it. > >>>>> >> > >>>>> >> > >>>>> >> It was not playing well with our security framework. The > framework was not working well with Solr APIs > >>>>> >> > >>>>> >>> > >>>>> >>> (Unrelated - Happy Thanksgiving all!) > >>>>> >>> > >>>>> >>> Best, > >>>>> >>> > >>>>> >>> Jason > >>>>> >>> > >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com> > wrote: > >>>>> >>> > > >>>>> >>> > Have you gone through an API written using the @EndPoint > annotation? > >>>>> >>> > > >>>>> >>> > I strongly recommend that you do > >>>>> >>> > > >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh < > ep...@opensourceconnections.com> wrote: > >>>>> >>> >> > >>>>> >>> >> I have found our V2 API code to be very impenetrable to > understand. Part of it is how the code is intertwined with support for > V1, however it’s also because there aren’t really resources to go look at > to understand how it should work! Maintaining the API should be very > simple work, as they just exist as a translation. The home grown stuff > may make sense if you are a super knowledgable Solr developer, but if you > are just a new person, it’s a lot harder to contribute. > >>>>> >>> >> > >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots > of projects use it very successfully, and if I want to implement something, > well, there are lots of blogs and resources out there! > >>>>> >>> >> > >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what > lessons learned there might apply to adopting Jersey for API support? > Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was > partly deprecated because we were not using it to support all the API, only > the ManagedResource ones, and > https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet > maybe was no longer being updated? One reason why we spiked out Jersey > was because of the broad support in the Java world! Looking at how much > work we have to do in the V2 API world, we need a much broader pool of > developers contributing to get there! > >>>>> >>> >> > >>>>> >>> >> Related, are there specific features/aspects of our > annotations that enable things in Solr that couldn’t be done otherwise? > >>>>> >>> >> > >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >>>>> >>> >> > >>>>> >>> >> Is there some problem with our annotations that we hope to > solve using third party dependencies? > >>>>> >>> >> By the way, we have used Restlet in the past and that has > been a regrettable decision. > >>>>> >>> >> > >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >>>>> >>> >>> > >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', > etc.) has > >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. > Having them > >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we > use our own > >>>>> >>> >>> annotations here instead of something off the shelf? > >>>>> >>> >>> > >>>>> >>> >>> What we have works well enough, but anything homegrown comes > with more > >>>>> >>> >>> maintenance burden than we'd have if we used something off > the shelf. > >>>>> >>> >>> There are plenty of well-used, active projects out there > whose whole > >>>>> >>> >>> purpose is facilitating the whole "annotation based API" > thing > >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them? > >>>>> >>> >>> > >>>>> >>> >>> Does anyone know of any technical reasons why we can't go > this route? > >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any > context on > >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty > annotations > >>>>> >>> >>> originally? > >>>>> >>> >>> > >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and > got > >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too > much trouble. > >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's > still a lot > >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), > but we both > >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best > of all, > >>>>> >>> >>> APIs using our current homegrown annotation framework the > switchover > >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like > Jersey > >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that > much. > >>>>> >>> >>> > >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up > with the > >>>>> >>> >>> annotation setup we use today! > >>>>> >>> >>> > >>>>> >>> >>> Best, > >>>>> >>> >>> > >>>>> >>> >>> Jason > >>>>> >>> >>> > >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and > children) > >>>>> >>> >>> [2] > http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E > >>>>> >>> >>> [3] > https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. > >>>>> >>> >>> > >>>>> >>> >>> > --------------------------------------------------------------------- > >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> >>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> >>> >>> > >>>>> >>> >> > >>>>> >>> >> _______________________ > >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | > 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy > >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > >>>>> >>> >> This e-mail and all contents, including attachments, is > considered to be Company Confidential unless explicitly stated otherwise, > regardless of whether attachments are marked as such. > >>>>> >>> >> > >>>>> >>> > >>>>> >>> > --------------------------------------------------------------------- > >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> >>> > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> ----------------------------------------------------- > >>>>> >> Noble Paul > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > ----------------------------------------------------- > >>>>> > Noble Paul > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >>>>> For additional commands, e-mail: dev-h...@solr.apache.org > >>>>> > >>>> > >>>> > >>>> -- > >>>> http://www.needhamsoftware.com (work) > >>>> http://www.the111shift.com (play) > >>> > >>> > >>> > >>> -- > >>> ----------------------------------------------------- > >>> Noble Paul > >> > >> > >> > >> -- > >> ----------------------------------------------------- > >> Noble Paul > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org > >
