FYI, I misspelt "Amazon Ion" to "Amazon Eon". I meant this:
https://en.wikipedia.org/wiki/Ion_(serialization_format)

On Mon, Dec 6, 2021 at 6:56 PM Eric Pugh <ep...@opensourceconnections.com>
wrote:

> I was actually poking around trying to find out who actually in the Ruby
> world uses the wt=ruby, and all the projects that intersect with Solr seems
> to do wt=json ;-)
>
>
> On Dec 6, 2021, at 8:17 AM, Jan Høydahl <jan....@cominvent.com> wrote:
>
> I'm happy to see that there is agreement in a shift towards standard..
>
> Bwt - in v3, lets get rid of "wt" parameter and instead use standard HTTP
> Accept header to tell Solr whether you want the response as JSON or XML.
> Maybe we are even ready to deprecate XML, PYTHON, XSLT, Excel..
> See https://issues.apache.org/jira/browse/SOLR-10998
>
> Jan
>
> 3. des. 2021 kl. 17:31 skrev Timothy Potter <thelabd...@gmail.com>:
>
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> I do not deny that a standardized framework may be helpful. I'm already
> suffering from the PTSD of restlet integration. The integration code was
> extremely complex and a security nightmare.It was added without any perf
> tests too. It took many years to get rid of that
>
>
>
> On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> 99% of Solr users never see the java code that implements an API let alone
> writing a new API.
>
>
> We may be talking past each other here a bit Noble.  The JAX-RS
> benefits I've been trying to describe are to benefit us developers,
> not Solr users.  I agree that 99% of our users won't know or care
> about it, but who cares.  The goal is to make _our_  lives as
> maintainers easier going into the future.  Jan described it in a word
> better than I've been struggling to: it's about "modernization".
>
> Best,
>
> Jason
>
> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <jan....@cominvent.com> wrote:
>
>
> Agree, Ishan.
>
> Let's welcome and cheer initiatives to modernize this old codebase.
> Diving in deep and un-tangling DispatchFilter like Gus is attempting is
> brave.
> And testing JAX-RS to modernize APIs and make the codebas easier to grok
> is also not for the faint hearted.
>
> I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated OpenAPI
>  docs for Solr as well!
> Theoretically foundations like this may make it feasible in the future to
> auto-generate client code from the spec,so that full Admin API client
> support in e.g. Go or Rust may be achievable without hand-coding. But that
> is still far away :)
>
> Jan
>
> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> We are discussing this as if moving to an external framework is going to
> be smooth. It's not.
>
>
> Noble, we won't know before someone has given it a try. My hunch suggests
> it won't be smooth, but we can decide on that once we see a concrete
> solution that 1) works well for all existing APIs (per core or global)
> without hacks/workarounds, and 2) has no security implications, and 3)
> doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>
> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <noble.p...@gmail.com> wrote:
>
>
>
>
> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> There are no known issues with the current system
>
>
> Maybe you're using hyperbole to emphasise a point, but let's steer
> this discussion away from straw-men and caricatures.  I mentioned
> specific known issues in a previous reply as a direct response to your
> question about them.  The incomplete support for data types
> (SOLR-15619 and SOLR-15796).  The inflexible PermissionNameProvider
> integration (SOLR-15823).  If you consider those "minor", fair enough,
> and you're welcome to that opinion.  But saying that they don't exist
> is a mischaracterization that wastes time and muddies water.
>
>
>
> We are discussing this as if moving to an external framework is going to
> be smooth. It's not.  I have seen how hard the Restlet integration was and
> how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
>
>
>
> Solr is not a general purpose web server where people write APIs every day
>
>
> Totally agree - 99% of the time someone opens an API file in Solr,
> they're reading, not writing.  So if JAX-RS didn't help that case, it
> wouldn't be worth the effort of switching.  But IMO it helps that
> "read" case a lot.
>
>
>
> 99% of Solr users never see the java code that implements an API let alone
> writing a new API. They wouldn't care if an API is implemented using a
> Servlet API, a custom framework or something else.  They only care about
> the input and output. New end points are rare.
>
>
>
> JAX-RS defined APIs expose inputs as strongly-typed parameters
> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
> JAX-RS enumerates every input to the API in one place.  JAX-RS APIs
> can return any serializable type as a response (compare to the
> near-omnipresent NamedList today).
>
> (The annotation framework's PayloadObj class gets us a bit of this
> benefit, but not all of it and only on POST requests afaik.)
>
> This is all nice stuff at API-write time, but where it really helps is
> in making the code clearer and simpler whenever someone goes to read
> it later.  No wondering what parameters an API actually takes.  No
> puzzling out what a bunch of NamedList operations spread throughout
> API execution actually produce in the end. etc.
>
> The only reason why anyone would need to use this framework is to write a
> custom request handler.
>
>
> ...except for the majority of people who come in contact with the
> framework by reading an existing Solr API?  This ties into your point
> above about APIs being written infrequently but read all the time.
> Improving readability is the big gain here, and I think JAX-RS offers
> real improvements in that regard.
>
>
>
> We had no framework for the last 15+ years and V1 API (the only actually
> used API) still uses no frameworks. Even the annotation framework is not
> used much even though readability is a lot better in the annotation based
> system. Readability has not been an overriding concern for V1 APIs for sure
>
> Also, to clarify, I'm not asserting that no one configures custom
> requestHandlers.  I was wondering aloud how common it was, as an open
> question.
>
> Your point IS right. IRL, most of our users do not write request handlers
> . They write search components , query parsers and other plugins. Because
> we are a search engine and not a web server. I have never seen a user
> complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
>
>
>
> We did not fork Jackson. We are using Jackson itself.
>
>
> Ah, maybe you're right, "fork" is probably an unfair word.  I looked
> for the serialization code, and I'll admit there is less than I
> thought involved here and I misunderstood how it works.  So that's my
> bad.  The JSON serialization/deserialization discussion is prob a
> digression from the core contention about whether JAX-RS would be
> helpful.  I was curious about your earlier response and couldn't
> resist following up here but I should've done so elsewhere to avoid
> the distraction.
>
> Best,
>
>
> Jason
>
> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> The annotation framework is just a single class and it is made to work
> well with Solr instead of changing Solr to suit the needs of some external
> framework. There are no known issues with the current system and there are
> unknown issues that you'll face introducing a new framework.
>
> Solr is not a general purpose web server where people write APIs every
> day. It's a finished product where people use the publicly available APIs .
> We should focus our efforts on making our APIs work well .
>
> On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> We need to make an API work on a per core basis and cores can come up and
> go down randomly
>
>
> Ah, yeah.  I'll admit I was overlooking some of the wrinkles around
> custom request handlers etc.  That is a problem for switching to some
> off-the-shelf framework for sure.  It may still be possible, but it's
> definitely a big hurdle.  I wonder how commonly users still take
> advantage of the custom requestHandler stuff these days?  I haven't
> run across many customers who used them extensively, but maybe that's
> self-selecting in some way?
>
>
>
> The only reason why anyone would need to use this framework is to write a
> custom request handler. You're also saying nobody writes custom request
> handlers. So, what problem are you trying to solve if you think nobody uses
> them?
>
>
>
> If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app
>
>
> SolrJ has a handful of dependencies - is there a reason we take this
> approach with Jackson, but not with any of SolrJ's other deps?
> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>
> Granted, I get that we want SolrJ to be as painless as possible for
> its users, and jar-conflicts are a part of that and really suck.  But
> there are good tools out there for handling jar conflicts, and
> reimplementing chunks of library functionality just to avoid a
> gradle-dep strikes me as unsustainable from a maintenance/community
> perspective.  Not to mention that forking/mimicking Jackson code
> deprives our users of any efficiency/security improvements that
> Jackson might get tomorrow or next week or next year.
>
>
>
> We did not fork Jackson. We are using Jackson itself. Jackson supports
> custom annotations. There was a huge discussion why we should do it this
> way.
>
>
>
> Unless there's some reason specific to Jackson I guess I just don't
> get it.  But if I'm the minority opinion on that, fair enough I guess.
>
> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> The @JsonProperty annotations are added because it's a dependency in SolrJ
> as well
> If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app.
>
> On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> Also keep in mind that the same endpoint can be accessed with a core name
> and a collection name prefixes.
>
> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> True Gus, Almost every framework works outside of SolrDispatchFilter+
> HttpSolrCall. A lot of our initializations occur there.
>
> We need to make an API work on a per core basis and cores can come up and
> go down randomly. So we need to register these endpoints on a core.
>
> I'm not sure if any framework can achieve the same.
>
> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.h...@gmail.com> wrote:
>
>
> IIRC last time I looked restlet had the unsavory property of existing
> outside of the SolrDispatchFilter, unlike everything else which made for
> special cases because several things that probably ought to be their own
> siervlet filters are glommed into SolrDispatchFilter, like security,
> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
> one could just wrap such filters around it too...
>
> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> These are minor improvements compared to a full rewrite of the entire
> framework
>
>
> If you think data type support is minor, fair enough.  But to clarify
> I'm not suggesting a rewrite - I'm suggesting using something that
> already exists off the shelf.  Jersey (e.g.) itself provides the
> framework - there would be no "rewrite".
>
> re: past restlet use
>
> It was not playing well with our security framework. The framework was not
> working well with Solr APIs
>
>
> Ah, very interesting!  Security isn't something Eric or I tackled in
> our little spike branch, but it's definitely a concern.  Do you
> remember the specific concerns?  Or recall where any of the discussion
> around this happened?
>
> Without the context of that past discussion, it seems like the
> "PermissionNameProvider" interface could be implemented just as well
> by a class with (e.g.) Jersey annotations as one with our own custom
> annotations.  Certainly there'd need to be some
> RuleBasedAuthorizationPlugin changes or other integration code, but
> nothing that feels insurmountable.
>
> Maybe I can try spiking it out soon and find the issues myself, but
> it'd be much easier if someone happens to remember and can save me the
> trouble :-p
>
> Best,
>
> Jason
>
> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> The Annotations framework was written after playing with other frameworks.
> There were many shortcomings which were hard to overcome.
>
> The best example is a per collection API . How do you register an endpoint
> for a collection/core ?
>
> On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.p...@gmail.com> wrote:
>
>
>
>
> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
>
>
> I guess so yeah.  Third-party deps are just fuller, more robust
> solutions, whereas our annotations still need support added now and
> then for even primitive data types like "long" (see SOLR-15619).
>
>
> These are minor improvements compared to a full rewrite of the entire
> framework
>
>
>
> Every JIRA spent doing basic stuff like that is time away from
> improving Solr in some other way.
>
> So there are feature-gap/capabilities arguments for moving to a
> third-party dep, sure.  But, even if our annotations did everything
> Jersey+Jackson do today, I think switching would still be worth it.
> Every LOC in our code base brings along with it some maintenance cost:
> it might have bugs, needs tested, takes time for new contributors to
> "grok", etc.  Using off-the-shelf here would nuke a whole bunch of
> that.  If off-the-shelf is available for some given functionality, we
> should need a compelling reason to NOT use it.
>
> Lastly, I think there's an "approachability" argument for using
> off-the-shelf.  Thousands of developers out there are familiar with
> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> Solr's custom annotations.  Using a well-known technology like Jersey
> would make Solr all the easier to approach and contribute to for that
> pool of developers.
>
> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
>
>
>
> Ah, yeah, that's just the context I'm missing.  Anyone have a pointer
> to related discussions, or remember what made this "regrettable"?  All
> the theoretical benefits in the world don't matter much if we've
> already tried something like this in the past and decided against it.
>
>
>
> It was not playing well with our security framework. The framework was not
> working well with Solr APIs
>
>
> (Unrelated - Happy Thanksgiving all!)
>
> Best,
>
> Jason
>
> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.p...@gmail.com> wrote:
>
>
> Have you gone through an API written using the @EndPoint annotation?
>
> I strongly recommend that you do
>
> On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com>
> wrote:
>
>
> I have found our V2 API code to be very impenetrable to understand.   Part
> of it is how the code is intertwined with support for V1, however it’s also
> because there aren’t really resources to go look at to understand how it
> should work!  Maintaining the API should be very simple work, as they just
> exist as a translation.   The home grown stuff may make sense if you are a
> super knowledgable Solr developer, but if you are just a new person, it’s a
> lot harder to contribute.
>
> I was interested in the Jersey stuff because I’ve seen lots of projects
> use it very successfully, and if I want to implement something, well, there
> are lots of blogs and resources out there!
>
> Can anyone recap briefly why we dropped RESTlet?   And what lessons
> learned there might apply to adopting Jersey for API support?   Looking at
> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
> deprecated because we were not using it to support all the API, only the
> ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766
> suggests that RESTlet maybe was no longer being updated?   One reason why
> we spiked out Jersey was because of the broad support in the Java world!
>   Looking at how much work we have to do in the V2 API world, we need a
> much broader pool of developers contributing to get there!
>
> Related, are there specific features/aspects of our annotations that
> enable things in Solr that couldn’t be done otherwise?
>
> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <gerlowsk...@gmail.com>
> wrote:
>
>
> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> cropped up a few times over the past week or two. [1] [2]. Having them
> on top of mind, I've been wondering - is there a reason we use our own
> annotations here instead of something off the shelf?
>
> What we have works well enough, but anything homegrown comes with more
> maintenance burden than we'd have if we used something off the shelf.
> There are plenty of well-used, active projects out there whose whole
> purpose is facilitating the whole "annotation based API" thing
> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>
> Does anyone know of any technical reasons why we can't go this route?
> Or have any subjective reasons why we shouldn't?  Or any context on
> why we wrote our own Endpoint, Command, JsonProperty annotations
> originally?
>
> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> Jersey+Jackson working for a few simple APIs without too much trouble.
> [3]  Obviously nothing production-ready there, and there's still a lot
> of open questions (e.g. how would javabin be supported?), but we both
> came away convinced that it seemed feasible, at least.  Best of all,
> APIs using our current homegrown annotation framework the switchover
> seems blessedly straightforward, and it doesn't look like Jersey
> (which we chose mostly arbitrarily) bloats our dist all that much.
>
> Curious if anyone has thoughts or context on how we ended up with the
> annotation setup we use today!
>
> Best,
>
> Jason
>
> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com | My Free/Busy
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> <dev-unsubscr...@solr.apache.org>
> For additional commands, e-mail: dev-h...@solr.apache.org
> <dev-h...@solr.apache.org>
>
>
> _______________________
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Reply via email to