> We are discussing this as if moving to an external framework is going to be smooth. It's not.
Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance. On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <noble.p...@gmail.com> wrote: > > > On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > >> > There are no known issues with the current system >> >> Maybe you're using hyperbole to emphasise a point, but let's steer >> this discussion away from straw-men and caricatures. I mentioned >> specific known issues in a previous reply as a direct response to your >> question about them. The incomplete support for data types >> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider >> integration (SOLR-15823). If you consider those "minor", fair enough, >> and you're welcome to that opinion. But saying that they don't exist >> is a mischaracterization that wastes time and muddies water. >> > > We are discussing this as if moving to an external framework is going to > be smooth. It's not. I have seen how hard the Restlet integration was and > how it stood out like a sore thumb. The amount of code required to > integrate Solr APIs into restlet components was 10x more than the amount of > code written for the new framework and it still was incomplete > . Integrating anything to Solr is not simple and it's going to be an > ongoing effort to ensure that all of the functionalities of a > SolrRequestHandler are exposed to the new framework. Every single component > that we have today has ongoing improvements and bug fixes. Using that as an > excuse to do a complete rewrite is probably unwise. > > > >> > Solr is not a general purpose web server where people write APIs every >> day >> >> Totally agree - 99% of the time someone opens an API file in Solr, >> they're reading, not writing. So if JAX-RS didn't help that case, it >> wouldn't be worth the effort of switching. But IMO it helps that >> "read" case a lot. >> > > 99% of Solr users never see the java code that implements an API let alone > writing a new API. They wouldn't care if an API is implemented using a > Servlet API, a custom framework or something else. They only care about > the input and output. New end points are rare. > >> >> JAX-RS defined APIs expose inputs as strongly-typed parameters >> (compare to the weakly-typed SolrQueryRequest/SolrParams used today). >> JAX-RS enumerates every input to the API in one place. JAX-RS APIs >> can return any serializable type as a response (compare to the >> near-omnipresent NamedList today). >> >> (The annotation framework's PayloadObj class gets us a bit of this >> benefit, but not all of it and only on POST requests afaik.) >> >> This is all nice stuff at API-write time, but where it really helps is >> in making the code clearer and simpler whenever someone goes to read >> it later. No wondering what parameters an API actually takes. No >> puzzling out what a bunch of NamedList operations spread throughout >> API execution actually produce in the end. etc. >> >> > The only reason why anyone would need to use this framework is to write >> a custom request handler. >> >> ...except for the majority of people who come in contact with the >> framework by reading an existing Solr API? This ties into your point >> above about APIs being written infrequently but read all the time. >> Improving readability is the big gain here, and I think JAX-RS offers >> real improvements in that regard. >> > > We had no framework for the last 15+ years and V1 API (the only actually > used API) still uses no frameworks. Even the annotation framework is not > used much even though readability is a lot better in the annotation based > system. Readability has not been an overriding concern for V1 APIs for sure > > Also, to clarify, I'm not asserting that no one configures custom >> requestHandlers. I was wondering aloud how common it was, as an open >> question. >> >> Your point IS right. IRL, most of our users do not write request handlers > . They write search components , query parsers and other plugins. Because > we are a search engine and not a web server. I have never seen a user > complaining about the V1 API of writing a request handler. They may > actually complain about how hard it is to deploy one. Our users are much > smarter than we give them credit for. > > > >> > We did not fork Jackson. We are using Jackson itself. >> >> Ah, maybe you're right, "fork" is probably an unfair word. I looked >> for the serialization code, and I'll admit there is less than I >> thought involved here and I misunderstood how it works. So that's my >> bad. The JSON serialization/deserialization discussion is prob a >> digression from the core contention about whether JAX-RS would be >> helpful. I was curious about your earlier response and couldn't >> resist following up here but I should've done so elsewhere to avoid >> the distraction. >> >> Best, >> >> >> Jason >> >> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.p...@gmail.com> wrote: >> > >> > The annotation framework is just a single class and it is made to work >> well with Solr instead of changing Solr to suit the needs of some external >> framework. There are no known issues with the current system and there are >> unknown issues that you'll face introducing a new framework. >> > >> > Solr is not a general purpose web server where people write APIs every >> day. It's a finished product where people use the publicly available APIs . >> We should focus our efforts on making our APIs work well . >> > >> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowsk...@gmail.com> >> wrote: >> >> >> >> > We need to make an API work on a per core basis and cores can come >> up and go down randomly >> >> >> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around >> >> custom request handlers etc. That is a problem for switching to some >> >> off-the-shelf framework for sure. It may still be possible, but it's >> >> definitely a big hurdle. I wonder how commonly users still take >> >> advantage of the custom requestHandler stuff these days? I haven't >> >> run across many customers who used them extensively, but maybe that's >> >> self-selecting in some way? >> > >> > >> > The only reason why anyone would need to use this framework is to write >> a custom request handler. You're also saying nobody writes custom request >> handlers. So, what problem are you trying to solve if you think nobody uses >> them? >> > >> > >> >> >> >> > If SolrJ has a Jackson dependency, it can have a conflict (different >> versions) with the Jackson used by the client app >> >> >> >> SolrJ has a handful of dependencies - is there a reason we take this >> >> approach with Jackson, but not with any of SolrJ's other deps? >> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.) >> >> >> >> Granted, I get that we want SolrJ to be as painless as possible for >> >> its users, and jar-conflicts are a part of that and really suck. But >> >> there are good tools out there for handling jar conflicts, and >> >> reimplementing chunks of library functionality just to avoid a >> >> gradle-dep strikes me as unsustainable from a maintenance/community >> >> perspective. Not to mention that forking/mimicking Jackson code >> >> deprives our users of any efficiency/security improvements that >> >> Jackson might get tomorrow or next week or next year. >> > >> > >> > We did not fork Jackson. We are using Jackson itself. Jackson supports >> custom annotations. There was a huge discussion why we should do it this >> way. >> >> >> >> >> >> Unless there's some reason specific to Jackson I guess I just don't >> >> get it. But if I'm the minority opinion on that, fair enough I guess. >> >> >> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> >> wrote: >> >> > >> >> > The @JsonProperty annotations are added because it's a dependency in >> SolrJ as well >> >> > If SolrJ has a Jackson dependency, it can have a conflict (different >> versions) with the Jackson used by the client app. >> >> > >> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> >> wrote: >> >> >> >> >> >> Also keep in mind that the same endpoint can be accessed with a >> core name and a collection name prefixes. >> >> >> >> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> >> wrote: >> >> >>> >> >> >>> True Gus, Almost every framework works outside of >> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there. >> >> >>> >> >> >>> We need to make an API work on a per core basis and cores can come >> up and go down randomly. So we need to register these endpoints on a core. >> >> >>> >> >> >>> I'm not sure if any framework can achieve the same. >> >> >>> >> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> >> wrote: >> >> >>>> >> >> >>>> IIRC last time I looked restlet had the unsavory property of >> existing outside of the SolrDispatchFilter, unlike everything else which >> made for special cases because several things that probably ought to be >> their own siervlet filters are glommed into SolrDispatchFilter, like >> security, tracing and MDC setup/teardown per request. Restlet wouldn't be >> so bad if one could just wrap such filters around it too... >> >> >>>> >> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski < >> gerlowsk...@gmail.com> wrote: >> >> >>>>> >> >> >>>>> > These are minor improvements compared to a full rewrite of the >> entire framework >> >> >>>>> >> >> >>>>> If you think data type support is minor, fair enough. But to >> clarify >> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something >> that >> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the >> >> >>>>> framework - there would be no "rewrite". >> >> >>>>> >> >> >>>>> re: past restlet use >> >> >>>>> >> >> >>>>> > It was not playing well with our security framework. The >> framework was not working well with Solr APIs >> >> >>>>> >> >> >>>>> Ah, very interesting! Security isn't something Eric or I >> tackled in >> >> >>>>> our little spike branch, but it's definitely a concern. Do you >> >> >>>>> remember the specific concerns? Or recall where any of the >> discussion >> >> >>>>> around this happened? >> >> >>>>> >> >> >>>>> Without the context of that past discussion, it seems like the >> >> >>>>> "PermissionNameProvider" interface could be implemented just as >> well >> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own >> custom >> >> >>>>> annotations. Certainly there'd need to be some >> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, >> but >> >> >>>>> nothing that feels insurmountable. >> >> >>>>> >> >> >>>>> Maybe I can try spiking it out soon and find the issues myself, >> but >> >> >>>>> it'd be much easier if someone happens to remember and can save >> me the >> >> >>>>> trouble :-p >> >> >>>>> >> >> >>>>> Best, >> >> >>>>> >> >> >>>>> Jason >> >> >>>>> >> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul < >> noble.p...@gmail.com> wrote: >> >> >>>>> > >> >> >>>>> > The Annotations framework was written after playing with other >> frameworks. There were many shortcomings which were hard to overcome. >> >> >>>>> > >> >> >>>>> > The best example is a per collection API . How do you register >> an endpoint for a collection/core ? >> >> >>>>> > >> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul < >> noble.p...@gmail.com> wrote: >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski < >> gerlowsk...@gmail.com> wrote: >> >> >>>>> >>> >> >> >>>>> >>> > Is there some problem with our annotations that we hope to >> solve using third party dependencies? >> >> >>>>> >>> >> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more >> robust >> >> >>>>> >>> solutions, whereas our annotations still need support added >> now and >> >> >>>>> >>> then for even primitive data types like "long" (see >> SOLR-15619). >> >> >>>>> >> >> >> >>>>> >> These are minor improvements compared to a full rewrite of >> the entire framework >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >>> >> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away >> from >> >> >>>>> >>> improving Solr in some other way. >> >> >>>>> >>> >> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving >> to a >> >> >>>>> >>> third-party dep, sure. But, even if our annotations did >> everything >> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be >> worth it. >> >> >>>>> >>> Every LOC in our code base brings along with it some >> maintenance cost: >> >> >>>>> >>> it might have bugs, needs tested, takes time for new >> contributors to >> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole >> bunch of >> >> >>>>> >>> that. If off-the-shelf is available for some given >> functionality, we >> >> >>>>> >>> should need a compelling reason to NOT use it. >> >> >>>>> >>> >> >> >>>>> >>> Lastly, I think there's an "approachability" argument for >> using >> >> >>>>> >>> off-the-shelf. Thousands of developers out there are >> familiar with >> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) >> familiar with >> >> >>>>> >>> Solr's custom annotations. Using a well-known technology >> like Jersey >> >> >>>>> >>> would make Solr all the easier to approach and contribute to >> for that >> >> >>>>> >>> pool of developers. >> >> >>>>> >>> >> >> >>>>> >>> > By the way, we have used Restlet in the past and that has >> been a regrettable decision. >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >>> >> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have >> a pointer >> >> >>>>> >>> to related discussions, or remember what made this >> "regrettable"? All >> >> >>>>> >>> the theoretical benefits in the world don't matter much if >> we've >> >> >>>>> >>> already tried something like this in the past and decided >> against it. >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> It was not playing well with our security framework. The >> framework was not working well with Solr APIs >> >> >>>>> >> >> >> >>>>> >>> >> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!) >> >> >>>>> >>> >> >> >>>>> >>> Best, >> >> >>>>> >>> >> >> >>>>> >>> Jason >> >> >>>>> >>> >> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul < >> noble.p...@gmail.com> wrote: >> >> >>>>> >>> > >> >> >>>>> >>> > Have you gone through an API written using the @EndPoint >> annotation? >> >> >>>>> >>> > >> >> >>>>> >>> > I strongly recommend that you do >> >> >>>>> >>> > >> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh < >> ep...@opensourceconnections.com> wrote: >> >> >>>>> >>> >> >> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to >> understand. Part of it is how the code is intertwined with support for >> V1, however it’s also because there aren’t really resources to go look at >> to understand how it should work! Maintaining the API should be very >> simple work, as they just exist as a translation. The home grown stuff >> may make sense if you are a super knowledgable Solr developer, but if you >> are just a new person, it’s a lot harder to contribute. >> >> >>>>> >>> >> >> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen >> lots of projects use it very successfully, and if I want to implement >> something, well, there are lots of blogs and resources out there! >> >> >>>>> >>> >> >> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And >> what lessons learned there might apply to adopting Jersey for API support? >> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was >> partly deprecated because we were not using it to support all the API, only >> the ManagedResource ones, and >> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet >> maybe was no longer being updated? One reason why we spiked out Jersey >> was because of the broad support in the Java world! Looking at how much >> work we have to do in the V2 API world, we need a much broader pool of >> developers contributing to get there! >> >> >>>>> >>> >> >> >> >>>>> >>> >> Related, are there specific features/aspects of our >> annotations that enable things in Solr that couldn’t be done otherwise? >> >> >>>>> >>> >> >> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> wrote: >> >> >>>>> >>> >> >> >> >>>>> >>> >> Is there some problem with our annotations that we hope >> to solve using third party dependencies? >> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has >> been a regrettable decision. >> >> >>>>> >>> >> >> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski < >> gerlowsk...@gmail.com> wrote: >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', >> '@Command', etc.) has >> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] >> [2]. Having them >> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason >> we use our own >> >> >>>>> >>> >>> annotations here instead of something off the shelf? >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> What we have works well enough, but anything homegrown >> comes with more >> >> >>>>> >>> >>> maintenance burden than we'd have if we used something >> off the shelf. >> >> >>>>> >>> >>> There are plenty of well-used, active projects out there >> whose whole >> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" >> thing >> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of >> them? >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't >> go this route? >> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any >> context on >> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty >> annotations >> >> >>>>> >>> >>> originally? >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, >> and got >> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too >> much trouble. >> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and >> there's still a lot >> >> >>>>> >>> >>> of open questions (e.g. how would javabin be >> supported?), but we both >> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. >> Best of all, >> >> >>>>> >>> >>> APIs using our current homegrown annotation framework >> the switchover >> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look >> like Jersey >> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all >> that much. >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we >> ended up with the >> >> >>>>> >>> >>> annotation setup we use today! >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> Best, >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> Jason >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 >> (and children) >> >> >>>>> >>> >>> [2] >> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E >> >> >>>>> >>> >>> [3] >> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. >> >> >>>>> >>> >>> >> >> >>>>> >>> >>> >> --------------------------------------------------------------------- >> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >> >>>>> >>> >>> For additional commands, e-mail: >> dev-h...@solr.apache.org >> >> >>>>> >>> >>> >> >> >>>>> >>> >> >> >> >>>>> >>> >> _______________________ >> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | >> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy >> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed >> >> >>>>> >>> >> This e-mail and all contents, including attachments, is >> considered to be Company Confidential unless explicitly stated otherwise, >> regardless of whether attachments are marked as such. >> >> >>>>> >>> >> >> >> >>>>> >>> >> >> >>>>> >>> >> --------------------------------------------------------------------- >> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >> >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org >> >> >>>>> >>> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> -- >> >> >>>>> >> ----------------------------------------------------- >> >> >>>>> >> Noble Paul >> >> >>>>> > >> >> >>>>> > >> >> >>>>> > >> >> >>>>> > -- >> >> >>>>> > ----------------------------------------------------- >> >> >>>>> > Noble Paul >> >> >>>>> >> >> >>>>> >> --------------------------------------------------------------------- >> >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >> >>>>> For additional commands, e-mail: dev-h...@solr.apache.org >> >> >>>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> http://www.needhamsoftware.com (work) >> >> >>>> http://www.the111shift.com (play) >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> ----------------------------------------------------- >> >> >>> Noble Paul >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> ----------------------------------------------------- >> >> >> Noble Paul >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> >> For additional commands, e-mail: dev-h...@solr.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> For additional commands, e-mail: dev-h...@solr.apache.org >> >> > > -- > ----------------------------------------------------- > Noble Paul >
