> There are no known issues with the current system Maybe you're using hyperbole to emphasise a point, but let's steer this discussion away from straw-men and caricatures. I mentioned specific known issues in a previous reply as a direct response to your question about them. The incomplete support for data types (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider integration (SOLR-15823). If you consider those "minor", fair enough, and you're welcome to that opinion. But saying that they don't exist is a mischaracterization that wastes time and muddies water.
> Solr is not a general purpose web server where people write APIs every day Totally agree - 99% of the time someone opens an API file in Solr, they're reading, not writing. So if JAX-RS didn't help that case, it wouldn't be worth the effort of switching. But IMO it helps that "read" case a lot. JAX-RS defined APIs expose inputs as strongly-typed parameters (compare to the weakly-typed SolrQueryRequest/SolrParams used today). JAX-RS enumerates every input to the API in one place. JAX-RS APIs can return any serializable type as a response (compare to the near-omnipresent NamedList today). (The annotation framework's PayloadObj class gets us a bit of this benefit, but not all of it and only on POST requests afaik.) This is all nice stuff at API-write time, but where it really helps is in making the code clearer and simpler whenever someone goes to read it later. No wondering what parameters an API actually takes. No puzzling out what a bunch of NamedList operations spread throughout API execution actually produce in the end. etc. > The only reason why anyone would need to use this framework is to write a > custom request handler. ...except for the majority of people who come in contact with the framework by reading an existing Solr API? This ties into your point above about APIs being written infrequently but read all the time. Improving readability is the big gain here, and I think JAX-RS offers real improvements in that regard. Also, to clarify, I'm not asserting that no one configures custom requestHandlers. I was wondering aloud how common it was, as an open question. > We did not fork Jackson. We are using Jackson itself. Ah, maybe you're right, "fork" is probably an unfair word. I looked for the serialization code, and I'll admit there is less than I thought involved here and I misunderstood how it works. So that's my bad. The JSON serialization/deserialization discussion is prob a digression from the core contention about whether JAX-RS would be helpful. I was curious about your earlier response and couldn't resist following up here but I should've done so elsewhere to avoid the distraction. Best, Jason On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.p...@gmail.com> wrote: > > The annotation framework is just a single class and it is made to work well > with Solr instead of changing Solr to suit the needs of some external > framework. There are no known issues with the current system and there are > unknown issues that you'll face introducing a new framework. > > Solr is not a general purpose web server where people write APIs every day. > It's a finished product where people use the publicly available APIs . We > should focus our efforts on making our APIs work well . > > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowsk...@gmail.com> wrote: >> >> > We need to make an API work on a per core basis and cores can come up and >> > go down randomly >> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around >> custom request handlers etc. That is a problem for switching to some >> off-the-shelf framework for sure. It may still be possible, but it's >> definitely a big hurdle. I wonder how commonly users still take >> advantage of the custom requestHandler stuff these days? I haven't >> run across many customers who used them extensively, but maybe that's >> self-selecting in some way? > > > The only reason why anyone would need to use this framework is to write a > custom request handler. You're also saying nobody writes custom request > handlers. So, what problem are you trying to solve if you think nobody uses > them? > > >> >> > If SolrJ has a Jackson dependency, it can have a conflict (different >> > versions) with the Jackson used by the client app >> >> SolrJ has a handful of dependencies - is there a reason we take this >> approach with Jackson, but not with any of SolrJ's other deps? >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.) >> >> Granted, I get that we want SolrJ to be as painless as possible for >> its users, and jar-conflicts are a part of that and really suck. But >> there are good tools out there for handling jar conflicts, and >> reimplementing chunks of library functionality just to avoid a >> gradle-dep strikes me as unsustainable from a maintenance/community >> perspective. Not to mention that forking/mimicking Jackson code >> deprives our users of any efficiency/security improvements that >> Jackson might get tomorrow or next week or next year. > > > We did not fork Jackson. We are using Jackson itself. Jackson supports custom > annotations. There was a huge discussion why we should do it this way. >> >> >> Unless there's some reason specific to Jackson I guess I just don't >> get it. But if I'm the minority opinion on that, fair enough I guess. >> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> wrote: >> > >> > The @JsonProperty annotations are added because it's a dependency in SolrJ >> > as well >> > If SolrJ has a Jackson dependency, it can have a conflict (different >> > versions) with the Jackson used by the client app. >> > >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> wrote: >> >> >> >> Also keep in mind that the same endpoint can be accessed with a core name >> >> and a collection name prefixes. >> >> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> wrote: >> >>> >> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ >> >>> HttpSolrCall. A lot of our initializations occur there. >> >>> >> >>> We need to make an API work on a per core basis and cores can come up >> >>> and go down randomly. So we need to register these endpoints on a core. >> >>> >> >>> I'm not sure if any framework can achieve the same. >> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> wrote: >> >>>> >> >>>> IIRC last time I looked restlet had the unsavory property of existing >> >>>> outside of the SolrDispatchFilter, unlike everything else which made >> >>>> for special cases because several things that probably ought to be >> >>>> their own siervlet filters are glommed into SolrDispatchFilter, like >> >>>> security, tracing and MDC setup/teardown per request. Restlet wouldn't >> >>>> be so bad if one could just wrap such filters around it too... >> >>>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowsk...@gmail.com> >> >>>> wrote: >> >>>>> >> >>>>> > These are minor improvements compared to a full rewrite of the >> >>>>> > entire framework >> >>>>> >> >>>>> If you think data type support is minor, fair enough. But to clarify >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the >> >>>>> framework - there would be no "rewrite". >> >>>>> >> >>>>> re: past restlet use >> >>>>> >> >>>>> > It was not playing well with our security framework. The framework >> >>>>> > was not working well with Solr APIs >> >>>>> >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in >> >>>>> our little spike branch, but it's definitely a concern. Do you >> >>>>> remember the specific concerns? Or recall where any of the discussion >> >>>>> around this happened? >> >>>>> >> >>>>> Without the context of that past discussion, it seems like the >> >>>>> "PermissionNameProvider" interface could be implemented just as well >> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom >> >>>>> annotations. Certainly there'd need to be some >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but >> >>>>> nothing that feels insurmountable. >> >>>>> >> >>>>> Maybe I can try spiking it out soon and find the issues myself, but >> >>>>> it'd be much easier if someone happens to remember and can save me the >> >>>>> trouble :-p >> >>>>> >> >>>>> Best, >> >>>>> >> >>>>> Jason >> >>>>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> >> >>>>> wrote: >> >>>>> > >> >>>>> > The Annotations framework was written after playing with other >> >>>>> > frameworks. There were many shortcomings which were hard to overcome. >> >>>>> > >> >>>>> > The best example is a per collection API . How do you register an >> >>>>> > endpoint for a collection/core ? >> >>>>> > >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com> >> >>>>> > wrote: >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski >> >>>>> >> <gerlowsk...@gmail.com> wrote: >> >>>>> >>> >> >>>>> >>> > Is there some problem with our annotations that we hope to solve >> >>>>> >>> > using third party dependencies? >> >>>>> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust >> >>>>> >>> solutions, whereas our annotations still need support added now and >> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619). >> >>>>> >> >> >>>>> >> These are minor improvements compared to a full rewrite of the >> >>>>> >> entire framework >> >>>>> >> >> >>>>> >> >> >>>>> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from >> >>>>> >>> improving Solr in some other way. >> >>>>> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a >> >>>>> >>> third-party dep, sure. But, even if our annotations did everything >> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it. >> >>>>> >>> Every LOC in our code base brings along with it some maintenance >> >>>>> >>> cost: >> >>>>> >>> it might have bugs, needs tested, takes time for new contributors >> >>>>> >>> to >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of >> >>>>> >>> that. If off-the-shelf is available for some given functionality, >> >>>>> >>> we >> >>>>> >>> should need a compelling reason to NOT use it. >> >>>>> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument for using >> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar >> >>>>> >>> with >> >>>>> >>> Solr's custom annotations. Using a well-known technology like >> >>>>> >>> Jersey >> >>>>> >>> would make Solr all the easier to approach and contribute to for >> >>>>> >>> that >> >>>>> >>> pool of developers. >> >>>>> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and that has been a >> >>>>> >>> > regrettable decision. >> >>>>> >> >> >>>>> >> >> >>>>> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a >> >>>>> >>> pointer >> >>>>> >>> to related discussions, or remember what made this "regrettable"? >> >>>>> >>> All >> >>>>> >>> the theoretical benefits in the world don't matter much if we've >> >>>>> >>> already tried something like this in the past and decided against >> >>>>> >>> it. >> >>>>> >> >> >>>>> >> >> >>>>> >> It was not playing well with our security framework. The framework >> >>>>> >> was not working well with Solr APIs >> >>>>> >> >> >>>>> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!) >> >>>>> >>> >> >>>>> >>> Best, >> >>>>> >>> >> >>>>> >>> Jason >> >>>>> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com> >> >>>>> >>> wrote: >> >>>>> >>> > >> >>>>> >>> > Have you gone through an API written using the @EndPoint >> >>>>> >>> > annotation? >> >>>>> >>> > >> >>>>> >>> > I strongly recommend that you do >> >>>>> >>> > >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh >> >>>>> >>> > <ep...@opensourceconnections.com> wrote: >> >>>>> >>> >> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to >> >>>>> >>> >> understand. Part of it is how the code is intertwined with >> >>>>> >>> >> support for V1, however it’s also because there aren’t really >> >>>>> >>> >> resources to go look at to understand how it should work! >> >>>>> >>> >> Maintaining the API should be very simple work, as they just >> >>>>> >>> >> exist as a translation. The home grown stuff may make sense >> >>>>> >>> >> if you are a super knowledgable Solr developer, but if you are >> >>>>> >>> >> just a new person, it’s a lot harder to contribute. >> >>>>> >>> >> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of >> >>>>> >>> >> projects use it very successfully, and if I want to implement >> >>>>> >>> >> something, well, there are lots of blogs and resources out >> >>>>> >>> >> there! >> >>>>> >>> >> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what >> >>>>> >>> >> lessons learned there might apply to adopting Jersey for API >> >>>>> >>> >> support? Looking at >> >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14659, it was partly >> >>>>> >>> >> deprecated because we were not using it to support all the API, >> >>>>> >>> >> only the ManagedResource ones, and >> >>>>> >>> >> https://issues.apache.org/jira/browse/SOLR-14766 suggests that >> >>>>> >>> >> RESTlet maybe was no longer being updated? One reason why we >> >>>>> >>> >> spiked out Jersey was because of the broad support in the Java >> >>>>> >>> >> world! Looking at how much work we have to do in the V2 API >> >>>>> >>> >> world, we need a much broader pool of developers contributing >> >>>>> >>> >> to get there! >> >>>>> >>> >> >> >>>>> >>> >> Related, are there specific features/aspects of our annotations >> >>>>> >>> >> that enable things in Solr that couldn’t be done otherwise? >> >>>>> >>> >> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya >> >>>>> >>> >> <ichattopadhy...@gmail.com> wrote: >> >>>>> >>> >> >> >>>>> >>> >> Is there some problem with our annotations that we hope to >> >>>>> >>> >> solve using third party dependencies? >> >>>>> >>> >> By the way, we have used Restlet in the past and that has been >> >>>>> >>> >> a regrettable decision. >> >>>>> >>> >> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski >> >>>>> >>> >> <gerlowsk...@gmail.com> wrote: >> >>>>> >>> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', >> >>>>> >>> >>> etc.) has >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. >> >>>>> >>> >>> Having them >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use >> >>>>> >>> >>> our own >> >>>>> >>> >>> annotations here instead of something off the shelf? >> >>>>> >>> >>> >> >>>>> >>> >>> What we have works well enough, but anything homegrown comes >> >>>>> >>> >>> with more >> >>>>> >>> >>> maintenance burden than we'd have if we used something off the >> >>>>> >>> >>> shelf. >> >>>>> >>> >>> There are plenty of well-used, active projects out there whose >> >>>>> >>> >>> whole >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them? >> >>>>> >>> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this >> >>>>> >>> >>> route? >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any >> >>>>> >>> >>> context on >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty >> >>>>> >>> >>> annotations >> >>>>> >>> >>> originally? >> >>>>> >>> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much >> >>>>> >>> >>> trouble. >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's >> >>>>> >>> >>> still a lot >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but >> >>>>> >>> >>> we both >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best >> >>>>> >>> >>> of all, >> >>>>> >>> >>> APIs using our current homegrown annotation framework the >> >>>>> >>> >>> switchover >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like >> >>>>> >>> >>> Jersey >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that >> >>>>> >>> >>> much. >> >>>>> >>> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up >> >>>>> >>> >>> with the >> >>>>> >>> >>> annotation setup we use today! >> >>>>> >>> >>> >> >>>>> >>> >>> Best, >> >>>>> >>> >>> >> >>>>> >>> >>> Jason >> >>>>> >>> >>> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and >> >>>>> >>> >>> children) >> >>>>> >>> >>> [2] >> >>>>> >>> >>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E >> >>>>> >>> >>> [3] >> >>>>> >>> >>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. >> >>>>> >>> >>> >> >>>>> >>> >>> --------------------------------------------------------------------- >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >>>>> >>> >>> For additional commands, e-mail: dev-h...@solr.apache.org >> >>>>> >>> >>> >> >>>>> >>> >> >> >>>>> >>> >> _______________________ >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | >> >>>>> >>> >> 434.466.1467 | http://www.opensourceconnections.com | My >> >>>>> >>> >> Free/Busy >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed >> >>>>> >>> >> This e-mail and all contents, including attachments, is >> >>>>> >>> >> considered to be Company Confidential unless explicitly stated >> >>>>> >>> >> otherwise, regardless of whether attachments are marked as such. >> >>>>> >>> >> >> >>>>> >>> >> >>>>> >>> --------------------------------------------------------------------- >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org >> >>>>> >>> >> >>>>> >> >> >>>>> >> >> >>>>> >> -- >> >>>>> >> ----------------------------------------------------- >> >>>>> >> Noble Paul >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > -- >> >>>>> > ----------------------------------------------------- >> >>>>> > Noble Paul >> >>>>> >> >>>>> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >>>>> For additional commands, e-mail: dev-h...@solr.apache.org >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> http://www.needhamsoftware.com (work) >> >>>> http://www.the111shift.com (play) >> >>> >> >>> >> >>> >> >>> -- >> >>> ----------------------------------------------------- >> >>> Noble Paul >> >> >> >> >> >> >> >> -- >> >> ----------------------------------------------------- >> >> Noble Paul >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> For additional commands, e-mail: dev-h...@solr.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
