On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <gerlowsk...@gmail.com> wrote:
> > There are no known issues with the current system > > Maybe you're using hyperbole to emphasise a point, but let's steer > this discussion away from straw-men and caricatures. I mentioned > specific known issues in a previous reply as a direct response to your > question about them. The incomplete support for data types > (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider > integration (SOLR-15823). If you consider those "minor", fair enough, > and you're welcome to that opinion. But saying that they don't exist > is a mischaracterization that wastes time and muddies water. > We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise. > > Solr is not a general purpose web server where people write APIs every > day > > Totally agree - 99% of the time someone opens an API file in Solr, > they're reading, not writing. So if JAX-RS didn't help that case, it > wouldn't be worth the effort of switching. But IMO it helps that > "read" case a lot. > 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare. > > JAX-RS defined APIs expose inputs as strongly-typed parameters > (compare to the weakly-typed SolrQueryRequest/SolrParams used today). > JAX-RS enumerates every input to the API in one place. JAX-RS APIs > can return any serializable type as a response (compare to the > near-omnipresent NamedList today). > > (The annotation framework's PayloadObj class gets us a bit of this > benefit, but not all of it and only on POST requests afaik.) > > This is all nice stuff at API-write time, but where it really helps is > in making the code clearer and simpler whenever someone goes to read > it later. No wondering what parameters an API actually takes. No > puzzling out what a bunch of NamedList operations spread throughout > API execution actually produce in the end. etc. > > > The only reason why anyone would need to use this framework is to write > a custom request handler. > > ...except for the majority of people who come in contact with the > framework by reading an existing Solr API? This ties into your point > above about APIs being written infrequently but read all the time. > Improving readability is the big gain here, and I think JAX-RS offers > real improvements in that regard. > We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure Also, to clarify, I'm not asserting that no one configures custom > requestHandlers. I was wondering aloud how common it was, as an open > question. > > Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for. > > We did not fork Jackson. We are using Jackson itself. > > Ah, maybe you're right, "fork" is probably an unfair word. I looked > for the serialization code, and I'll admit there is less than I > thought involved here and I misunderstood how it works. So that's my > bad. The JSON serialization/deserialization discussion is prob a > digression from the core contention about whether JAX-RS would be > helpful. I was curious about your earlier response and couldn't > resist following up here but I should've done so elsewhere to avoid > the distraction. > > Best, > > > Jason > > On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.p...@gmail.com> wrote: > > > > The annotation framework is just a single class and it is made to work > well with Solr instead of changing Solr to suit the needs of some external > framework. There are no known issues with the current system and there are > unknown issues that you'll face introducing a new framework. > > > > Solr is not a general purpose web server where people write APIs every > day. It's a finished product where people use the publicly available APIs . > We should focus our efforts on making our APIs work well . > > > > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > >> > >> > We need to make an API work on a per core basis and cores can come up > and go down randomly > >> > >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around > >> custom request handlers etc. That is a problem for switching to some > >> off-the-shelf framework for sure. It may still be possible, but it's > >> definitely a big hurdle. I wonder how commonly users still take > >> advantage of the custom requestHandler stuff these days? I haven't > >> run across many customers who used them extensively, but maybe that's > >> self-selecting in some way? > > > > > > The only reason why anyone would need to use this framework is to write > a custom request handler. You're also saying nobody writes custom request > handlers. So, what problem are you trying to solve if you think nobody uses > them? > > > > > >> > >> > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app > >> > >> SolrJ has a handful of dependencies - is there a reason we take this > >> approach with Jackson, but not with any of SolrJ's other deps? > >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.) > >> > >> Granted, I get that we want SolrJ to be as painless as possible for > >> its users, and jar-conflicts are a part of that and really suck. But > >> there are good tools out there for handling jar conflicts, and > >> reimplementing chunks of library functionality just to avoid a > >> gradle-dep strikes me as unsustainable from a maintenance/community > >> perspective. Not to mention that forking/mimicking Jackson code > >> deprives our users of any efficiency/security improvements that > >> Jackson might get tomorrow or next week or next year. > > > > > > We did not fork Jackson. We are using Jackson itself. Jackson supports > custom annotations. There was a huge discussion why we should do it this > way. > >> > >> > >> Unless there's some reason specific to Jackson I guess I just don't > >> get it. But if I'm the minority opinion on that, fair enough I guess. > >> > >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> > wrote: > >> > > >> > The @JsonProperty annotations are added because it's a dependency in > SolrJ as well > >> > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app. > >> > > >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> > wrote: > >> >> > >> >> Also keep in mind that the same endpoint can be accessed with a core > name and a collection name prefixes. > >> >> > >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> > wrote: > >> >>> > >> >>> True Gus, Almost every framework works outside of > SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there. > >> >>> > >> >>> We need to make an API work on a per core basis and cores can come > up and go down randomly. So we need to register these endpoints on a core. > >> >>> > >> >>> I'm not sure if any framework can achieve the same. > >> >>> > >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> > wrote: > >> >>>> > >> >>>> IIRC last time I looked restlet had the unsavory property of > existing outside of the SolrDispatchFilter, unlike everything else which > made for special cases because several things that probably ought to be > their own siervlet filters are glommed into SolrDispatchFilter, like > security, tracing and MDC setup/teardown per request. Restlet wouldn't be > so bad if one could just wrap such filters around it too... > >> >>>> > >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >> >>>>> > >> >>>>> > These are minor improvements compared to a full rewrite of the > entire framework > >> >>>>> > >> >>>>> If you think data type support is minor, fair enough. But to > clarify > >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that > >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the > >> >>>>> framework - there would be no "rewrite". > >> >>>>> > >> >>>>> re: past restlet use > >> >>>>> > >> >>>>> > It was not playing well with our security framework. The > framework was not working well with Solr APIs > >> >>>>> > >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled > in > >> >>>>> our little spike branch, but it's definitely a concern. Do you > >> >>>>> remember the specific concerns? Or recall where any of the > discussion > >> >>>>> around this happened? > >> >>>>> > >> >>>>> Without the context of that past discussion, it seems like the > >> >>>>> "PermissionNameProvider" interface could be implemented just as > well > >> >>>>> by a class with (e.g.) Jersey annotations as one with our own > custom > >> >>>>> annotations. Certainly there'd need to be some > >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, > but > >> >>>>> nothing that feels insurmountable. > >> >>>>> > >> >>>>> Maybe I can try spiking it out soon and find the issues myself, > but > >> >>>>> it'd be much easier if someone happens to remember and can save > me the > >> >>>>> trouble :-p > >> >>>>> > >> >>>>> Best, > >> >>>>> > >> >>>>> Jason > >> >>>>> > >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> > wrote: > >> >>>>> > > >> >>>>> > The Annotations framework was written after playing with other > frameworks. There were many shortcomings which were hard to overcome. > >> >>>>> > > >> >>>>> > The best example is a per collection API . How do you register > an endpoint for a collection/core ? > >> >>>>> > > >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul < > noble.p...@gmail.com> wrote: > >> >>>>> >> > >> >>>>> >> > >> >>>>> >> > >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >> >>>>> >>> > >> >>>>> >>> > Is there some problem with our annotations that we hope to > solve using third party dependencies? > >> >>>>> >>> > >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more > robust > >> >>>>> >>> solutions, whereas our annotations still need support added > now and > >> >>>>> >>> then for even primitive data types like "long" (see > SOLR-15619). > >> >>>>> >> > >> >>>>> >> These are minor improvements compared to a full rewrite of the > entire framework > >> >>>>> >> > >> >>>>> >> > >> >>>>> >>> > >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from > >> >>>>> >>> improving Solr in some other way. > >> >>>>> >>> > >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to > a > >> >>>>> >>> third-party dep, sure. But, even if our annotations did > everything > >> >>>>> >>> Jersey+Jackson do today, I think switching would still be > worth it. > >> >>>>> >>> Every LOC in our code base brings along with it some > maintenance cost: > >> >>>>> >>> it might have bugs, needs tested, takes time for new > contributors to > >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole > bunch of > >> >>>>> >>> that. If off-the-shelf is available for some given > functionality, we > >> >>>>> >>> should need a compelling reason to NOT use it. > >> >>>>> >>> > >> >>>>> >>> Lastly, I think there's an "approachability" argument for > using > >> >>>>> >>> off-the-shelf. Thousands of developers out there are > familiar with > >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) > familiar with > >> >>>>> >>> Solr's custom annotations. Using a well-known technology > like Jersey > >> >>>>> >>> would make Solr all the easier to approach and contribute to > for that > >> >>>>> >>> pool of developers. > >> >>>>> >>> > >> >>>>> >>> > By the way, we have used Restlet in the past and that has > been a regrettable decision. > >> >>>>> >> > >> >>>>> >> > >> >>>>> >>> > >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a > pointer > >> >>>>> >>> to related discussions, or remember what made this > "regrettable"? All > >> >>>>> >>> the theoretical benefits in the world don't matter much if > we've > >> >>>>> >>> already tried something like this in the past and decided > against it. > >> >>>>> >> > >> >>>>> >> > >> >>>>> >> It was not playing well with our security framework. The > framework was not working well with Solr APIs > >> >>>>> >> > >> >>>>> >>> > >> >>>>> >>> (Unrelated - Happy Thanksgiving all!) > >> >>>>> >>> > >> >>>>> >>> Best, > >> >>>>> >>> > >> >>>>> >>> Jason > >> >>>>> >>> > >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul < > noble.p...@gmail.com> wrote: > >> >>>>> >>> > > >> >>>>> >>> > Have you gone through an API written using the @EndPoint > annotation? > >> >>>>> >>> > > >> >>>>> >>> > I strongly recommend that you do > >> >>>>> >>> > > >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh < > ep...@opensourceconnections.com> wrote: > >> >>>>> >>> >> > >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to > understand. Part of it is how the code is intertwined with support for > V1, however it’s also because there aren’t really resources to go look at > to understand how it should work! Maintaining the API should be very > simple work, as they just exist as a translation. The home grown stuff > may make sense if you are a super knowledgable Solr developer, but if you > are just a new person, it’s a lot harder to contribute. > >> >>>>> >>> >> > >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen > lots of projects use it very successfully, and if I want to implement > something, well, there are lots of blogs and resources out there! > >> >>>>> >>> >> > >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And > what lessons learned there might apply to adopting Jersey for API support? > Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was > partly deprecated because we were not using it to support all the API, only > the ManagedResource ones, and > https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet > maybe was no longer being updated? One reason why we spiked out Jersey > was because of the broad support in the Java world! Looking at how much > work we have to do in the V2 API world, we need a much broader pool of > developers contributing to get there! > >> >>>>> >>> >> > >> >>>>> >>> >> Related, are there specific features/aspects of our > annotations that enable things in Solr that couldn’t be done otherwise? > >> >>>>> >>> >> > >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >> >>>>> >>> >> > >> >>>>> >>> >> Is there some problem with our annotations that we hope to > solve using third party dependencies? > >> >>>>> >>> >> By the way, we have used Restlet in the past and that has > been a regrettable decision. > >> >>>>> >>> >> > >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski < > gerlowsk...@gmail.com> wrote: > >> >>>>> >>> >>> > >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', > '@Command', etc.) has > >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] > [2]. Having them > >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason > we use our own > >> >>>>> >>> >>> annotations here instead of something off the shelf? > >> >>>>> >>> >>> > >> >>>>> >>> >>> What we have works well enough, but anything homegrown > comes with more > >> >>>>> >>> >>> maintenance burden than we'd have if we used something > off the shelf. > >> >>>>> >>> >>> There are plenty of well-used, active projects out there > whose whole > >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" > thing > >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of > them? > >> >>>>> >>> >>> > >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go > this route? > >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any > context on > >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty > annotations > >> >>>>> >>> >>> originally? > >> >>>>> >>> >>> > >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, > and got > >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too > much trouble. > >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and > there's still a lot > >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), > but we both > >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. > Best of all, > >> >>>>> >>> >>> APIs using our current homegrown annotation framework the > switchover > >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like > Jersey > >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all > that much. > >> >>>>> >>> >>> > >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended > up with the > >> >>>>> >>> >>> annotation setup we use today! > >> >>>>> >>> >>> > >> >>>>> >>> >>> Best, > >> >>>>> >>> >>> > >> >>>>> >>> >>> Jason > >> >>>>> >>> >>> > >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 > (and children) > >> >>>>> >>> >>> [2] > http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E > >> >>>>> >>> >>> [3] > https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. > >> >>>>> >>> >>> > >> >>>>> >>> >>> > --------------------------------------------------------------------- > >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >> >>>>> >>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >> >>>>> >>> >>> > >> >>>>> >>> >> > >> >>>>> >>> >> _______________________ > >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | > 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy > >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > >> >>>>> >>> >> This e-mail and all contents, including attachments, is > considered to be Company Confidential unless explicitly stated otherwise, > regardless of whether attachments are marked as such. > >> >>>>> >>> >> > >> >>>>> >>> > >> >>>>> >>> > --------------------------------------------------------------------- > >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >> >>>>> >>> For additional commands, e-mail: dev-h...@solr.apache.org > >> >>>>> >>> > >> >>>>> >> > >> >>>>> >> > >> >>>>> >> -- > >> >>>>> >> ----------------------------------------------------- > >> >>>>> >> Noble Paul > >> >>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> > -- > >> >>>>> > ----------------------------------------------------- > >> >>>>> > Noble Paul > >> >>>>> > >> >>>>> > --------------------------------------------------------------------- > >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >> >>>>> For additional commands, e-mail: dev-h...@solr.apache.org > >> >>>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> http://www.needhamsoftware.com (work) > >> >>>> http://www.the111shift.com (play) > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> ----------------------------------------------------- > >> >>> Noble Paul > >> >> > >> >> > >> >> > >> >> -- > >> >> ----------------------------------------------------- > >> >> Noble Paul > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > >> For additional commands, e-mail: dev-h...@solr.apache.org > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org > > -- ----------------------------------------------------- Noble Paul
