Gus: Very thoughtful post. I think you raise an _extremely_ important point about “how critical is the UI?” And by extension other packages. If they’re critical to Solr, the question of how to keep them in sync becomes paramount.
I agree that the admin UI is important, if we have a mechanism to insure it’s kept in sync with the release that would be near the to of my list. Best, Erick > On Apr 9, 2020, at 12:06 PM, Gus Heck <gus.h...@gmail.com> wrote: > > In my view this brings us up against a bit of an existential question. How do > we ensure quality of packages that are key to Solr? I'm sure that there are > folks who don't find the UI very useful, but it's important to others. The > rationale that "elastic keeps their's separate" has to be tempered by the > actual real differences between Elastic and Solr. Elastic has a corporate > sponsor, a coordinated road map, and explicitly ensures that all the bits > that are maintained separately work together (so that they don't have > excessive support costs or bad first experiences that impact their bottom > line etc.). > > Solr is in a different place however, and we need to carefully examine the > question of whether something that works for Elastic works for us. Lucidworks > and several other large companies do spend a lot of money on developers that > contribute to Solr, but there is no organization around multiple components > that MUST work together. Another example of this is Solr and Lucene, and our > defense against a lack of coordination of components in that case has been to > unify them into a single release package. > > So IMHO we need to answer 2 questions: > 1.) Do we consider the UI important. If I'm alone or in a minority in feeling > it's important, then so be it and it doesn't really matter what we do. (maybe > a vote?) > 2.) If we make it "a package" how do we ensure that important packages such > as the UI are never broken by a new release. > > IMHO I don't thing we should tolerate a situation where things we consider > important are broken frequently. > > For my part obviously my answer to 1 is "yes" :). As for 2, one thing that > comes to mind is what the Ant project did (may still do?) with (and my memory > from 15 years ago is foggy here, so forgive me if something I say is not > quite perfect) the GUMP build server that ran ant builds for a bunch of > different projects that depended on ant to ensure early detections of changes > that would break existing projects. If we have a good UI test suite and > commit to that being part of the release build package that might be a > solution. I honestly don't actually care where it lives, but I do think it > hurts us if it becomes broken and unusable, or hard to install. > > My worry is that "Solr developers are not UI developers" is really code-words > for "I want to be able to break it and let others clean up the mess, because > I'm not a UI developer". I have this worry with respect to all "packages", > but I may be missing info from discussions about the package system, which > initiated during a very busy period for me so background links that I should > have read are welcome if I've got something wrong here :) I went looking for > a SIP but didn't see it... I have found a google doc linked from SOLR-13661 > > Finally to a detail about one of the above suggestions the option to > automatically download and install the UI could be good, but that then > requires that packages be available from somewhere that never goes away like > maven central, or that Apache commits to hosting a repository server > indefinitely, but again that's surely been discussed WRT packages already... > Using Github in such a way is subject to being broken arbitrarily when Github > decides to restrict things for cost reasons (ask Bower about that one WRT > rate limiting...) or the "repository" has to be something local and therefore > must be included part of the distribution... at which point it's still a > thing we distribute and since we're distributing it and we probably don't > mean to distribute broken stuff we still need UI developers... > > Also, I thought the package loading stuff was supposed to be disabled by > default for security, that seems to conflict with or at least complicate the > notion of easily installing as a package. > > So "package" is a good for modularizing code, or for 3rd party (possibly > paid) plugins (I have a client that might find that interesting in the > future) but we have to ensure that it doesn't lead to a lack of maintenance > for things that are critical. > > Incidentally though I've said I favor Angular CLI, (significantly because > I've got some start on learning it) it also occurs to me that perhaps > anything "modern" is a difficulty because those things all have a learning > curve, and maximizing accessibility and ease of modifications for folks not > steeped in UI development might be our priority (different from the > priorities a commercial site would have). The flip side argument is that with > a popular framework, it would be easier for UI focused folks to contribute... > but will they? and does that leave us perennially rewriting the UI in > whatever is popular? (maybe that's ok?) I think in all our decisions here we > need to be very careful to distinguish how our needs may differ in unusual > ways from the needs of commercial web development. > > -Gus > > On Thu, Apr 9, 2020 at 8:14 AM Erick Erickson <erickerick...@gmail.com> wrote: > Marcus: > > re-reading the thread, it looks to me like the consensus from Noble and Ishan > and Jan is that as long as the new, nifty UI is a separate package, go ahead > and knock yourself out ;). The objection is to making it part of the Solr > code base… We’ll all be thrilled with if we can rip the current admin UI out > ;) > > That said, I suspect it’ll be one of the tighter packages. It’d be super-cool > if we could run the UI tests on Jenkins say once a day just to keep it up to > date. > > The admin UI has always been somewhat awkwardly bolted on the side of Solr, > it’d be great to have it have a more elegant architecture. > > The other exciting thing would be that clients could then use the package > code as something they can incorporate/fork/whatever. Practically every > client I’ve worked with at large installations has rolled their own > dashboard. If they could use a package as a starting point, it’d be welcome. > > Best, > Erick > > > On Apr 9, 2020, at 3:07 AM, Marcus Eagan <marcusea...@gmail.com> wrote: > > > > Hey Noble, > > > > -1 is a definitive, so I want to clarify that you are saying you do not > > wish to remove the EOL front end and replace it with another one in the > > longer term? > > > > I hear you! As a product manager in my day job, my primary goal is to find > > features to cut! I spend a lot of time thinking about non-essential vs used > > heavily vs causes more problems than it's worth. I can tell you from > > watching the many people in the field at Lucidworks, there are a lot of > > people who know quite a bit about Solr, but rely on the Admin UI heavily > > because they feel comfortable there. Those people in effect help us stay > > employed despite never contributing or being capable of contributing to > > Solr. So hear me out. I've got a proposal: > > > > To start, I can work on this app as an optional package for your awesome > > new package manager. It will be the second one I've worked on in my > > evenings and weekends btw. The first was a package validator that I hope to > > eventually open source, but its complexity and lack of popularity because > > it is security ;( will likely make it the second one I open source/finish. > > I'm also collaborating with a couple members of the Lucidworks security > > team on that one, but I have built the basics already for them to build > > upon. > > > > Back to the new UI discussion and my update that I promised. > > > > My update was going to be that after evaluating the projects Jan posted, > > the most recent project that Jan listed created a pretty good base to build > > on. After lots of auditing of the packages and a bit of refactoring because > > the UI world moves fast, I was able to get it to transpile and run again > > (as I'm sure it previously did) and from (2290 vulns): > > <image.png> > > no npm fix doesn't magically fix, I wish it did > > > > to (2 low sev, non-productions vulns): > > <image.png> > > > > These two issues will not affect production and really only the unit tests. > > Besides, I plan to remove them before we get to a stage even that matters. > > I've also started investigating the level of effort for me to get it to > > feature parity with the current app. The preliminary answer is not very > > much compared to other work I've done in shorter time — working with a jerk > > for a boss (years ago, don't worry Hatcher). I'm building a couple of the > > missing features as we speak. From the beginning, it will have infinitely > > more test coverage and will be a lot more approachable to contemporary UI > > developers. It will also make the Solr experience for new developers > > simpler. The major design changes that I have been thinking about would be > > to the cloud view and the query view. Both of those are important, the > > first to more experienced users, and the second to less advanced users > > though occasionally an advanced debugger or demo presenter in my experience. > > > > In the end Noble, this is about making Solr more approachable to new users > > not experts like you. The growth of Solr adoption only benefits you, so I > > would ask you to revisit your -1 at some point in the near future when you > > see the progress and breadth of improvement. We have had customers complain > > about the Admin UI and the community has even complained about it. I think > > this is the right thing to do. If you still consider effectively upgrading > > the current Admin UI as feature creep, I can revisit the package manager > > compromise or move the efforts elsewhere. I respect your position. A search > > service is nothing without a strong and diverse set of skills and > > capabilities behind it and making it accessible to everyone who needs it. > > > > For those who care, here's my 4 Node cluster of tech products running > > locally. > > > > <image.png> > > > > The shoulders of the homie that put that scaffold together are broad! Props > > to him. I will volunteer to put together extensive docs for JS devs who > > want to contribute and make it better once we get it to a place where it > > replaces and improves upon the current option. I'll even sponsor some > > prizes for college kids or people recently out of work to get cranking on > > this bad boy. > > > > Thank you Noble and everyone else, > > > > Marcus > > > > On Wed, Apr 8, 2020 at 11:20 PM Noble Paul <noble.p...@gmail.com> wrote: > > Solr developers are not UI experts. We are a search service and such a > > service should have nice clean APIs + documentation. Is a UI useful ? yes. > > The last thing we want today is another complex component in Solr codebase > > that nobody understands or cannot maintain. > > > > So, Solr UI can be hosted outside our codebase and we can have an option to > > install UI from that remote repo > > > > something like "bin/solr install-gui" > > > > I'm -1 on anymore feature creep. > > > > --Noble > > > > > > On Thu, Apr 9, 2020 at 12:22 PM Marcus Eagan <marcusea...@gmail.com> wrote: > > Thanks again Gus. > > > > Lots of people indeed misuse REST so we could go on and on about whether > > requests are stateless or not in another thread. Let's spare the group. > > > > I think most everyone on this channel would be in agreement with you on > > separate app. I'll be opening a new ticket and a PR that will document a > > few things to make it easy for UI devs who know little to no Java how to > > get started. > > > > Ishan, there's some significant UI expertise in the team. Erickson finds > > his way to open every cookie jar. Erik Hatcher wrote the first version of > > Blacklight. I've seen Pugh do lots of work on Quepid's UI. Jan and Kevin > > have done a lot of work, and so have many others. The list goes on, and > > *likes to work on UI* is a different discussion. > > > > Beyond committers because I'm not a committer, I have UI expertise that I > > can polish off and improve for the sake of my interest and commitment to > > the community and I like to do it. I've also led UI teams. I can help to > > steward the effort overall and keep things up to date up to the point where > > I need to ask one of the committers to help me get changes merged. I'll > > probably even hire a developer to work on it once we are to that point. ;-) > > > > Expertise is not something that should block us but motivate us to expand > > this community and/or our own skillsets long term. > > > > Thank you both and everyone else, > > > > Marcus > > > > On Wed, Apr 8, 2020, 10:21 AM Gus Heck <gus.h...@gmail.com> wrote: > > While running it in an external node does ensure separability, I don't > > think it does a good job of addressing my other point of not needing to > > manage a 3rd server. It's still a server if it's started by java, and one > > still has to ensure it exists, and it will be extra hard to figure out how > > to configure it if started by Solr. > > > > I'm strongly in favor of us having a UI from my perspective as a consultant > > it makes discovery of things like their startup parameters and directories > > and such very easy (just go to front page of the admin screen), and it's so > > much easier to get a customer with security concerns and strict controls on > > who can access what (think banks, military, etc) to share a web session > > where they drive the UI than to get direct access to machines. It'll be a > > lot slower and much lower service to be making people wait while I craft > > curl statements to paste into the web session (and then fix the inevitable > > typos, or detect when they missed the last char of what I pasted, etc...). > > > > I definitely against Solr spawning some other server (node or otherwise) on > > it's own and thereby requiring additional system dependencies, or creating > > a second process that needs to be configured and properly secured. To me > > that's even worse than requiring the UI to run outside of Solr. We have a > > perfectly good web container already, and furthermore there's a much > > greater likelihood that maintainers will be facile with java/j2ee than > > anything else (IMHO). It's great if the framework we choose uses little or > > no JSP/Servlet and is modernized with a 100% javascript, templated etc. > > front end, but the back end should be java/jetty because we've got lots of > > java folks. > > > > If the back end matters deeply then you're not really programming to > > MVC/REST style... > > > > So there's another $0.02 :) and if you're not careful I'll give you an > > entire nickle's worth of ways people misuse/misunderstand the term REST :) > > > > -Gus > > > > On Tue, Apr 7, 2020 at 9:06 PM Marcus Eagan <marcusea...@gmail.com> wrote: > > Gus, > > > > Your $.02 are worth a lot more than $.02 USD, so thank you. > > > > By separate app, I think I mean to endorse managed by a Node.js process > > started by NPM. I don’t think that conflicts with what you have proposed. > > The NPM command should be issued by Java || or Bash but I don’t think it > > would add significant overhead. Also, seems like on CI and or precommit > > hooks front end could be sizzled in parallel without adding much overhead. > > > > As for the front end framework, the most important things to consider in my > > view are simplicity and maintainability. We need to do a thorough analysis > > on the ecosystem and issues like the size of a React project vs Angular > > project vs Vue project, but React and Vue certainly have the velocity and > > the hearts if the front end community more than Angular. React is MIT > > license now and for the foreseeable > > future thanks to the power and reach of its developers. > > > > <gus.h...@gmail.com> wrote: > > +1 for Angular CLI / Typescript since I've fiddled with this in a minor way > > recently, Also MIT license is super friendly. > > > > > > As a disenfranchised volunteer to the project, I also assume voters on > > specific choices like frameworks will be helping build in some respect at > > some point now or in the future. Is that a fair or misguided assumption? > > > > Marcus > > > > On Tue, Apr 7, 2020 at 17:15 Gus Heck <gus.h...@gmail.com> wrote: > > +1 for Angular CLI / Typescript since I've fiddled with this in a minor way > > recently, Also MIT license is super friendly. > > > > Separate App - hmm... that's got some attraction, but also gives my stomach > > some churning when I think about solr now requiring management of 3 > > different servers (solr, something to serve UI and zookeeper). Adding more > > infrastructure gives me pause with respect to all the smaller > > installations. I've had several small self funded startup clients and a few > > clients with existing initial installs that they are outgrowing in places > > where procuring new machines and new software is a 6-12 mo endeavor and > > both types seem to squirm when I make suggestions such as running zookeeper > > separately, (let alone 3 of them). I think separate looks good for medium > > to large folks or very large companies that **already have** a solr expert > > on hand, but hurts the small clients and the departments in large orgs that > > got started with insufficient advice/expertise, so maybe > > > > - The UI should be installed by default > > - it should be easy to remove it, or start with it disabled > > - it should be self contained and separately downloadable. > > > > My recent fiddling included figuring out how to make angular CLI play nice > > in a J2ee war file structure seen here: https://github.com/nsoft/ns-login > > > > By play nice I mean, > > - build creates a war file that "just works" when installed > > - Angluar CLI commands work > > - Angular serve command works (for auto-reloading ui changes, running on > > port 4200; note the use of proxy to allow it to talk to an already running > > web container) > > > > My $0.02, > > > > -Gus > > > > On Mon, Apr 6, 2020 at 11:03 AM Jörn Franke <jornfra...@gmail.com> wrote: > > I think standalone would be very useful. > > I propose Angular with Typescript - it fits to a more data centric approach > > with data types etc. > > Maybe even two types of UIs - Admin UI and a simple Search UI. > > > > > >> Am 06.04.2020 um 16:53 schrieb Jan Høydahl <jan....@cominvent.com>: > >> > >> Thanks for kickstarting this and bringing some fresh blood and enthusiasm > >> :) > >> > >> Looks like others have had similar wish for a standalone Solr Admin App, > >> here’s a quick GitHub search for inspiration: > >> > >> https://github.com/savantly-net/solr-admin (Angular, nice screenshots, > >> 1y old) > >> https://github.com/kezhenxu94/yasa (vuejs, impressive screenshots, 2y > >> old) > >> https://github.com/thereactleague/galaxy (React, no screenshots, 4y old) > >> > >> They all seem abandoned but perhaps a new official effort could bring > >> their developers in as contributors again? > >> > >>> the people who work on the Admin UI do not need to be expected to know > >>> the Java workflow, necessarily. This reality widens the net for who can > >>> contribute. > >> > >> Agree. Frontend devs have been a shortage in this project, and if we can > >> make it easier to attract UI committers who feel at home and productive > >> with the UI code, that would be a win. On the other hand, if we expect > >> that the UI will be maintained by regular Java committers, then anything > >> that makes it easier for them/us to contribute is also a win, like perhaps > >> strongly-typed. > >> > >> Again, thanks Marcus for reviving this topic. Let us all try not to be > >> overly ambitious here or shoot the initiative down with bikeshedding. It > >> is far more important to fuel the energy and momentum and get something > >> built than to remain stuck :) > >> > >> Jan > >> > >> > >>> 6. apr. 2020 kl. 13:47 skrev Marcus Eagan <m...@marcuseagan.com>: > >>> > >>> Coming back to these existential questions from my phone: > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Jan Høydahl > >>> Added 1 hour ago > >>> There are many opinions around admin UI. So I think the best place to > >>> start would be a new mail-thread in dev@ to discuss the way forward. > >>> Before we start a major re-work, we should probably ask ourselves a few > >>> existential questions: > >>> • Should we turn Amin UI into a standalone app instead of embedded in > >>> Solr? > >>> > >>> I think it should be a standalone app. There are many advantages gained > >>> from a separation of such concerns. Some of the ones include, the people > >>> who work on the Admin UI do not need to be expected to know the Java > >>> workflow, necessarily. This reality widens the net for who can > >>> contribute. > >>> > >>> Testing becomes a lot easier because JS developers are accustomed to > >>> building tests for static assets and self-contained node apps. They > >>> generally know less about testing a bit of JS within a massive Java > >>> project. The test could also run independently for changes that only > >>> affect the front end. Adding test coverage without adding time to tests > >>> sounds awesome. > >>> > >>> There are quite a few tickets over the years that have seemed to suggest > >>> that people want more fine-grained control over the Solr admin UI > >>> overall. Two recent tickets discussed topics like running a Solr Admin > >>> app on only one node and disabling it al together for whatever reason. > >>> See: https://issues.apache.org/jira/browse/SOLR-14014. > >>> > >>> • What UI framework? Guess anything is better than current EOL, but > >>> will largely depend on who is willing to do the job! > >>> I’m happy to take this on (and willing to follow through on completing in > >>> my nights and weekends), but I am mostly framework agnostic. My stronge > >>> preference would be React, provided the license is kosher. There was one > >>> blip of “practically unusable for most orgs” a couple years back, but > >>> Facebook made it right really soon after. However, I’m flexible. Angular > >>> (not JS) and Vue are also great. I would recommend we consider > >>> Typescript also because of the size of project and number of > >>> strongly-typed devs on this mailing list. My only reservation with > >>> TypeScript, though it may not apply in this case, is that the supersets > >>> of JS have changed a lot more than the frameworks. While CoffeeScript was > >>> an unnecessary layer of abstraction from my limited perspective, > >>> TypeScript might make JS more embraceable to a list of Java hackers. > >>> > >>> • Current UI has no test coverage, can we do better with the new UI? > >>> > >>> It’s imperative.React, Angular, and Vue each make it easy to include > >>> tests. > >>> > >>> > >>> > >>> https://issues.apache.org/jira/browse/SOLR-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076204#comment-17076204 > >>> > >>> > >>> > >> > > > > > > -- > > http://www.needhamsoftware.com (work) > > http://www.the111shift.com (play) > > -- > > Marcus Eagan > > > > > > > > -- > > http://www.needhamsoftware.com (work) > > http://www.the111shift.com (play) > > > > > > -- > > ----------------------------------------------------- > > Noble Paul > > > > > > -- > > Marcus Eagan > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > -- > http://www.needhamsoftware.com (work) > http://www.the111shift.com (play) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
