FYI, I misspelt "Amazon Ion" to "Amazon Eon". I meant this: https://en.wikipedia.org/wiki/Ion_(serialization_format)
On Mon, Dec 6, 2021 at 6:56 PM Eric Pugh <ep...@opensourceconnections.com> wrote: > I was actually poking around trying to find out who actually in the Ruby > world uses the wt=ruby, and all the projects that intersect with Solr seems > to do wt=json ;-) > > > On Dec 6, 2021, at 8:17 AM, Jan Høydahl <jan....@cominvent.com> wrote: > > I'm happy to see that there is agreement in a shift towards standard.. > > Bwt - in v3, lets get rid of "wt" parameter and instead use standard HTTP > Accept header to tell Solr whether you want the response as JSON or XML. > Maybe we are even ready to deprecate XML, PYTHON, XSLT, Excel.. > See https://issues.apache.org/jira/browse/SOLR-10998 > > Jan > > 3. des. 2021 kl. 17:31 skrev Timothy Potter <thelabd...@gmail.com>: > > Why do we keep talking about Restlet here? Restlet is a stale (at > best) project, so not sure why it keeps coming up in a discussion > about modernizing our API? Integration with it in Solr was introduced > nearly 10 years ago, time to move on and stop using that as an excuse > to block adoption of other API frameworks. > > Surely anyone committed to this effort, after doing even the most > cursory due diligence, will not land on Restlet as the way forward. > Jersey is a likely candidate, I'm sure there are others? > > I actually don't care all that much about which annotations we use to > describe our APIs (of course standards are better). What's more > important for me, is that we can auto-generate an OpenAPI spec for our > APIs. That opens up a world of automated tooling for our users to > interact with our APIs. Using JAX-RS / Jersey probably helps get us > closer to an automated way to generate an OpenAPI spec but there are > other ways to get there as well. > > In general, it sounds like the objections to modernizing our API fall > into three buckets: 1) our bad experience with Restlet, 2) performance > is better with our homegrown code, 3) security is too complicated with > another framework. > > Performance overhead added by a framework like Jersey (if there is > any) for Admin APIs is mostly irrelevant. You think Jersey is going to > be slower at mapping requests to Java method calls? Pretty sure they > have that part figured out at this point. > > As for Solr's security, it also needs to be replaced with a modern > framework like Shiro or pac4j or similar. The permissions are > basically impossible to understand, even for most committers. This > whole idea of allow by default instead of deny by default goes against > every security best practice in the industry. So arguing that we need > some homegrown API framework to support Solr's obscure authz scheme > seems a bit backwards to me. Once you start using standard API > frameworks like JAX-RS, you start getting security integration too, > e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up > with all the innovation around security; heck, SolrJ doesn't even > support OIDC yet. I'm all for scraping both homegrown frameworks and > starting fresh with mature open source frameworks that have all the > kinks worked out over many years with many projects using them. > > If you want to talk about stress, try building a non-trivial app with > Solr's current API's, that'll induce some stress for sure! > > Tim > > > > On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <noble.p...@gmail.com> wrote: > > > I do not deny that a standardized framework may be helpful. I'm already > suffering from the PTSD of restlet integration. The integration code was > extremely complex and a security nightmare.It was added without any perf > tests too. It took many years to get rid of that > > > > On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > 99% of Solr users never see the java code that implements an API let alone > writing a new API. > > > We may be talking past each other here a bit Noble. The JAX-RS > benefits I've been trying to describe are to benefit us developers, > not Solr users. I agree that 99% of our users won't know or care > about it, but who cares. The goal is to make _our_ lives as > maintainers easier going into the future. Jan described it in a word > better than I've been struggling to: it's about "modernization". > > Best, > > Jason > > On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <jan....@cominvent.com> wrote: > > > Agree, Ishan. > > Let's welcome and cheer initiatives to modernize this old codebase. > Diving in deep and un-tangling DispatchFilter like Gus is attempting is > brave. > And testing JAX-RS to modernize APIs and make the codebas easier to grok > is also not for the faint hearted. > > I hope both initiatives end up as a breath of fresh air for Solr > developers. And perhaps users may get their hands on auto-generated OpenAPI > docs for Solr as well! > Theoretically foundations like this may make it feasible in the future to > auto-generate client code from the spec,so that full Admin API client > support in e.g. Go or Rust may be achievable without hand-coding. But that > is still far away :) > > Jan > > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya < > ichattopadhy...@gmail.com>: > > We are discussing this as if moving to an external framework is going to > be smooth. It's not. > > > Noble, we won't know before someone has given it a try. My hunch suggests > it won't be smooth, but we can decide on that once we see a concrete > solution that 1) works well for all existing APIs (per core or global) > without hacks/workarounds, and 2) has no security implications, and 3) > doesn't clutter our codebase, and 4) doesn't affect our runtime performance. > > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <noble.p...@gmail.com> wrote: > > > > > On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > There are no known issues with the current system > > > Maybe you're using hyperbole to emphasise a point, but let's steer > this discussion away from straw-men and caricatures. I mentioned > specific known issues in a previous reply as a direct response to your > question about them. The incomplete support for data types > (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider > integration (SOLR-15823). If you consider those "minor", fair enough, > and you're welcome to that opinion. But saying that they don't exist > is a mischaracterization that wastes time and muddies water. > > > > We are discussing this as if moving to an external framework is going to > be smooth. It's not. I have seen how hard the Restlet integration was and > how it stood out like a sore thumb. The amount of code required to > integrate Solr APIs into restlet components was 10x more than the amount of > code written for the new framework and it still was incomplete . > Integrating anything to Solr is not simple and it's going to be an ongoing > effort to ensure that all of the functionalities of a SolrRequestHandler > are exposed to the new framework. Every single component that we have today > has ongoing improvements and bug fixes. Using that as an excuse to do a > complete rewrite is probably unwise. > > > > Solr is not a general purpose web server where people write APIs every day > > > Totally agree - 99% of the time someone opens an API file in Solr, > they're reading, not writing. So if JAX-RS didn't help that case, it > wouldn't be worth the effort of switching. But IMO it helps that > "read" case a lot. > > > > 99% of Solr users never see the java code that implements an API let alone > writing a new API. They wouldn't care if an API is implemented using a > Servlet API, a custom framework or something else. They only care about > the input and output. New end points are rare. > > > > JAX-RS defined APIs expose inputs as strongly-typed parameters > (compare to the weakly-typed SolrQueryRequest/SolrParams used today). > JAX-RS enumerates every input to the API in one place. JAX-RS APIs > can return any serializable type as a response (compare to the > near-omnipresent NamedList today). > > (The annotation framework's PayloadObj class gets us a bit of this > benefit, but not all of it and only on POST requests afaik.) > > This is all nice stuff at API-write time, but where it really helps is > in making the code clearer and simpler whenever someone goes to read > it later. No wondering what parameters an API actually takes. No > puzzling out what a bunch of NamedList operations spread throughout > API execution actually produce in the end. etc. > > The only reason why anyone would need to use this framework is to write a > custom request handler. > > > ...except for the majority of people who come in contact with the > framework by reading an existing Solr API? This ties into your point > above about APIs being written infrequently but read all the time. > Improving readability is the big gain here, and I think JAX-RS offers > real improvements in that regard. > > > > We had no framework for the last 15+ years and V1 API (the only actually > used API) still uses no frameworks. Even the annotation framework is not > used much even though readability is a lot better in the annotation based > system. Readability has not been an overriding concern for V1 APIs for sure > > Also, to clarify, I'm not asserting that no one configures custom > requestHandlers. I was wondering aloud how common it was, as an open > question. > > Your point IS right. IRL, most of our users do not write request handlers > . They write search components , query parsers and other plugins. Because > we are a search engine and not a web server. I have never seen a user > complaining about the V1 API of writing a request handler. They may > actually complain about how hard it is to deploy one. Our users are much > smarter than we give them credit for. > > > > We did not fork Jackson. We are using Jackson itself. > > > Ah, maybe you're right, "fork" is probably an unfair word. I looked > for the serialization code, and I'll admit there is less than I > thought involved here and I misunderstood how it works. So that's my > bad. The JSON serialization/deserialization discussion is prob a > digression from the core contention about whether JAX-RS would be > helpful. I was curious about your earlier response and couldn't > resist following up here but I should've done so elsewhere to avoid > the distraction. > > Best, > > > Jason > > On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.p...@gmail.com> wrote: > > > The annotation framework is just a single class and it is made to work > well with Solr instead of changing Solr to suit the needs of some external > framework. There are no known issues with the current system and there are > unknown issues that you'll face introducing a new framework. > > Solr is not a general purpose web server where people write APIs every > day. It's a finished product where people use the publicly available APIs . > We should focus our efforts on making our APIs work well . > > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > We need to make an API work on a per core basis and cores can come up and > go down randomly > > > Ah, yeah. I'll admit I was overlooking some of the wrinkles around > custom request handlers etc. That is a problem for switching to some > off-the-shelf framework for sure. It may still be possible, but it's > definitely a big hurdle. I wonder how commonly users still take > advantage of the custom requestHandler stuff these days? I haven't > run across many customers who used them extensively, but maybe that's > self-selecting in some way? > > > > The only reason why anyone would need to use this framework is to write a > custom request handler. You're also saying nobody writes custom request > handlers. So, what problem are you trying to solve if you think nobody uses > them? > > > > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app > > > SolrJ has a handful of dependencies - is there a reason we take this > approach with Jackson, but not with any of SolrJ's other deps? > (apache-httpcomponents, apache-commons, zookeeper-client, etc.) > > Granted, I get that we want SolrJ to be as painless as possible for > its users, and jar-conflicts are a part of that and really suck. But > there are good tools out there for handling jar conflicts, and > reimplementing chunks of library functionality just to avoid a > gradle-dep strikes me as unsustainable from a maintenance/community > perspective. Not to mention that forking/mimicking Jackson code > deprives our users of any efficiency/security improvements that > Jackson might get tomorrow or next week or next year. > > > > We did not fork Jackson. We are using Jackson itself. Jackson supports > custom annotations. There was a huge discussion why we should do it this > way. > > > > Unless there's some reason specific to Jackson I guess I just don't > get it. But if I'm the minority opinion on that, fair enough I guess. > > On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> wrote: > > > The @JsonProperty annotations are added because it's a dependency in SolrJ > as well > If SolrJ has a Jackson dependency, it can have a conflict (different > versions) with the Jackson used by the client app. > > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> wrote: > > > Also keep in mind that the same endpoint can be accessed with a core name > and a collection name prefixes. > > On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> wrote: > > > True Gus, Almost every framework works outside of SolrDispatchFilter+ > HttpSolrCall. A lot of our initializations occur there. > > We need to make an API work on a per core basis and cores can come up and > go down randomly. So we need to register these endpoints on a core. > > I'm not sure if any framework can achieve the same. > > On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> wrote: > > > IIRC last time I looked restlet had the unsavory property of existing > outside of the SolrDispatchFilter, unlike everything else which made for > special cases because several things that probably ought to be their own > siervlet filters are glommed into SolrDispatchFilter, like security, > tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if > one could just wrap such filters around it too... > > On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > These are minor improvements compared to a full rewrite of the entire > framework > > > If you think data type support is minor, fair enough. But to clarify > I'm not suggesting a rewrite - I'm suggesting using something that > already exists off the shelf. Jersey (e.g.) itself provides the > framework - there would be no "rewrite". > > re: past restlet use > > It was not playing well with our security framework. The framework was not > working well with Solr APIs > > > Ah, very interesting! Security isn't something Eric or I tackled in > our little spike branch, but it's definitely a concern. Do you > remember the specific concerns? Or recall where any of the discussion > around this happened? > > Without the context of that past discussion, it seems like the > "PermissionNameProvider" interface could be implemented just as well > by a class with (e.g.) Jersey annotations as one with our own custom > annotations. Certainly there'd need to be some > RuleBasedAuthorizationPlugin changes or other integration code, but > nothing that feels insurmountable. > > Maybe I can try spiking it out soon and find the issues myself, but > it'd be much easier if someone happens to remember and can save me the > trouble :-p > > Best, > > Jason > > On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> wrote: > > > The Annotations framework was written after playing with other frameworks. > There were many shortcomings which were hard to overcome. > > The best example is a per collection API . How do you register an endpoint > for a collection/core ? > > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com> wrote: > > > > > On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > Is there some problem with our annotations that we hope to solve using > third party dependencies? > > > I guess so yeah. Third-party deps are just fuller, more robust > solutions, whereas our annotations still need support added now and > then for even primitive data types like "long" (see SOLR-15619). > > > These are minor improvements compared to a full rewrite of the entire > framework > > > > Every JIRA spent doing basic stuff like that is time away from > improving Solr in some other way. > > So there are feature-gap/capabilities arguments for moving to a > third-party dep, sure. But, even if our annotations did everything > Jersey+Jackson do today, I think switching would still be worth it. > Every LOC in our code base brings along with it some maintenance cost: > it might have bugs, needs tested, takes time for new contributors to > "grok", etc. Using off-the-shelf here would nuke a whole bunch of > that. If off-the-shelf is available for some given functionality, we > should need a compelling reason to NOT use it. > > Lastly, I think there's an "approachability" argument for using > off-the-shelf. Thousands of developers out there are familiar with > (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with > Solr's custom annotations. Using a well-known technology like Jersey > would make Solr all the easier to approach and contribute to for that > pool of developers. > > By the way, we have used Restlet in the past and that has been a > regrettable decision. > > > > > Ah, yeah, that's just the context I'm missing. Anyone have a pointer > to related discussions, or remember what made this "regrettable"? All > the theoretical benefits in the world don't matter much if we've > already tried something like this in the past and decided against it. > > > > It was not playing well with our security framework. The framework was not > working well with Solr APIs > > > (Unrelated - Happy Thanksgiving all!) > > Best, > > Jason > > On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com> wrote: > > > Have you gone through an API written using the @EndPoint annotation? > > I strongly recommend that you do > > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> > wrote: > > > I have found our V2 API code to be very impenetrable to understand. Part > of it is how the code is intertwined with support for V1, however it’s also > because there aren’t really resources to go look at to understand how it > should work! Maintaining the API should be very simple work, as they just > exist as a translation. The home grown stuff may make sense if you are a > super knowledgable Solr developer, but if you are just a new person, it’s a > lot harder to contribute. > > I was interested in the Jersey stuff because I’ve seen lots of projects > use it very successfully, and if I want to implement something, well, there > are lots of blogs and resources out there! > > Can anyone recap briefly why we dropped RESTlet? And what lessons > learned there might apply to adopting Jersey for API support? Looking at > https://issues.apache.org/jira/browse/SOLR-14659, it was partly > deprecated because we were not using it to support all the API, only the > ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 > suggests that RESTlet maybe was no longer being updated? One reason why > we spiked out Jersey was because of the broad support in the Java world! > Looking at how much work we have to do in the V2 API world, we need a > much broader pool of developers contributing to get there! > > Related, are there specific features/aspects of our annotations that > enable things in Solr that couldn’t be done otherwise? > > On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > > Is there some problem with our annotations that we hope to solve using > third party dependencies? > By the way, we have used Restlet in the past and that has been a > regrettable decision. > > On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has > cropped up a few times over the past week or two. [1] [2]. Having them > on top of mind, I've been wondering - is there a reason we use our own > annotations here instead of something off the shelf? > > What we have works well enough, but anything homegrown comes with more > maintenance burden than we'd have if we used something off the shelf. > There are plenty of well-used, active projects out there whose whole > purpose is facilitating the whole "annotation based API" thing > (Jersey, Restlet, RESTEasy, etc.) - why not use one of them? > > Does anyone know of any technical reasons why we can't go this route? > Or have any subjective reasons why we shouldn't? Or any context on > why we wrote our own Endpoint, Command, JsonProperty annotations > originally? > > FWIW, Eric Pugh and I spiked out a small POC recently, and got > Jersey+Jackson working for a few simple APIs without too much trouble. > [3] Obviously nothing production-ready there, and there's still a lot > of open questions (e.g. how would javabin be supported?), but we both > came away convinced that it seemed feasible, at least. Best of all, > APIs using our current homegrown annotation framework the switchover > seems blessedly straightforward, and it doesn't look like Jersey > (which we chose mostly arbitrarily) bloats our dist all that much. > > Curious if anyone has thoughts or context on how we ended up with the > annotation setup we use today! > > Best, > > Jason > > [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children) > [2] > http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E > [3] > https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > _______________________ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com | My Free/Busy > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless of > whether attachments are marked as such. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > > -- > ----------------------------------------------------- > Noble Paul > > > > > -- > ----------------------------------------------------- > Noble Paul > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > > -- > http://www.needhamsoftware.com (work) > http://www.the111shift.com (play) > > > > > -- > ----------------------------------------------------- > Noble Paul > > > > > -- > ----------------------------------------------------- > Noble Paul > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > > -- > ----------------------------------------------------- > Noble Paul > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > > -- > ----------------------------------------------------- > Noble Paul > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > <dev-unsubscr...@solr.apache.org> > For additional commands, e-mail: dev-h...@solr.apache.org > <dev-h...@solr.apache.org> > > > _______________________ > *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467 > | http://www.opensourceconnections.com | My Free/Busy > <http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless > of whether attachments are marked as such. > >
