Responses inline. More potentially flimsy excuses coming your way.
On 9/15/2016 9:56 PM, Aaron Greenspan wrote:
> My two cents: I’m glad to see the discussion over improved documentation, but
> if you give me a choice between better docs and better UI, I’ll choose a
> better UI every time. If contributors are going to spend real time on the
> concerns raised in this thread, spend the time on making the software better
> to the point where more docs are unnecessary. All sorts of things could
> improve that would make the product far more intuitive (and I know, there are
> probably JIRA entries on most of these already…).
The UI is not really intended to be the way that Solr gets accessed.
It's mostly just a way for an admin to peer into what Solr is doing. At
this time, it's not really a good tool for configuration. The admin UI
in 1.x and 3.x only had one or two ways to change the system ... and one
of those ways was enable/disable on the ping handler. Everything else
the UI did back then was informational.
In newer versions of Solr, the UI does have some capability to make
changes to the system state -- which it does by accessing the HTTP API,
same as any program you would write that uses Solr.
> - The psuedo-frames in the web UI are the source of all kinds of problems,
> with lots of weird horizontal scrolling I’ve noticed over the years. It makes
> the Logging screen in particular infuriating to use. When I click on certain
> log entries an arbitrary-seeming "false" flips to "true" under the "WARN"
> statement in the Level column. But on other log entries, it all just goes
> haywire all over the screen because it’s too big both horizontally and
> vertically, and then re-condenses as though I’d never clicked, as I mentioned
If the UI problems you've found are filed as bugs (or as a single bug
listing the problems), we can take it from there. My free time is
pretty small, or I would do it myself.
Checking the actual logfile has always been a better option than the
Logging tab in the UI. The info in ERROR messages is typically too
large for effective display in the UI, and the UI excludes messages that
are at a severity of INFO or less. The default level for the logfile is
INFO, and is typically very verbose.
> - The top menu on the left is in plain English. The core menu on the bottom
> is written as though it’s being viewed by a person who only speaks UNIX. For
> example, there is no space between "Data" and "Import" in "DataImport" and
> "Segments info" could just be "Segments". Is "Plugins / Stats" two menus in
The Solr feature that this accesses is contained in a class named
DataImportHandler. The most commonly configured URL path for the
handler is "/dataimport". Is it more important to be grammatically
correct, or connect the wording in the admin UI with what the user
actually sees in solrconfig.xml? You're probably right that it should
have that space, so it doesn't grate on the nerves of those who like
things to be correct.
On "Segments Info" ... brevity counts for a lot in an interface. That
sounds like a good change.
> - "Ping" in the menu takes you nowhere in particular and shouldn’t really be
> a menu item. It should be part of the main dashboard with all of the other
> tech stats (which I do like) or a menu called "Status". (Why would one core
> ping faster than another anyway? If this is really for "cloud" installations
> where cores can be split up on different servers, why am I seeing it when
> everything is local and immediate?)
Good point. I'm not sure why the time for the ping is so prominently
displayed. Ping isn't really about speed -- it's about whether or not
the server is up and functional. It's also a legacy feature that sort
of works with Cloud, but isn't really aware of Cloud. Plenty of room
for improvement in the UI, in the ping feature itself, and the docs.
> - On the Data Import page, the expandable icons are [-] when they’re expanded
> and still [-] when they’re collapsed. Extremely confusing.
That's definitely a bug. The priority of that bug might be debatable.
> - The Data Import UI makes no mention anywhere of the ability to import from
> MySQL, which is 99% of what I want to do with this product. It doesn’t tell
> me how to set up the MySQL connector, doesn’t give me a button that turns it
> on in some modular fashion, doesn’t tell me if the server connection is
> successful, doesn’t let me easily enter or edit credentials, doesn’t let me
> edit my queries anywhere, and doesn’t let me test out a new query and see how
> it might fit into the Solr schema. These deficiencies are presumably also
> true for any database data source, e.g. Postgres/DB2/ODBC/whatever—which also
> are not listed, were I curious to know what Solr can do just by looking at
> the product itself.
> - Nor does the Data Import UI have another section for picking a folder on
> the filesystem that might contain PDFs I want to import with Tika.
The dataimport section of the UI isn't about configuring dataimport or
telling it where to grab data from, it's more starting an import with
the already-configured settings and about visibility into the current
status of an ongoing import. Configuring the feature must still be done
with the XML config file right now. Having configuration options for
DIH in the UI would be awesome.
> - There is no field picker on the Query screen, but I just spent all of that
> time defining my fields in those XML files I can’t edit or auto-generate
> through the UI. That means I have to do extra work to remember them all. But
> then there is a field picker on the Analysis screen?
That's a cool idea. It would require a rework of the query screen. The
"q" box would still need to be there -- a field picker works great when
a query is only on one field, but the full query syntax lets you query
for information on multiple fields, and writing a UI to deal with THAT
would very challenging.
The analysis page only works on one field at a time, and even works on a
fieldType, which is also available in that screen's dropdown.
> - How do I restart Solr from the UI? Or change memory allocation settings?
> Can I?
You can't. Solr is still implemented as a java webapp. Although
containers do have the ability to hot deploy their applications, which
is like a restart, using this feature is typically plagued by memory
leaks and other problems. Restarting the actual *java* process is even
further out of reach for the UI, at least currently.
> - How do I change the port the UI is running on from the UI? Or limit the IP
> addresses Jetty is binding to? Can I?
This would require changing the system-level config and restarting Java,
which can't be done through the UI currently.
One of the ideas that has been floated is to rewrite Solr as *two* java
processes -- one that is a controller with extremely minimal memory
requirements that starts the *main* java process, and can completely
control every aspect. That's about the only way that settings like
memory allocation and listen port could be changed in the admin UI.
Implementing this feature would not be trivial, but I really like the idea.
> - How do I change log settings through the UI? Can I?
Which log settings? The logging level for each class can already be
changed. All other settings are configured with log4j.properties,
including permanent adjustments to logging levels for classes. Many
settings can be changed programmatically, but if we introduce code for
configuring a specific logging framework, then users will not be able to
change the logging framework that Solr uses. In the future, I think we
might take away the choice of logging framework and make additional
config options for logging available in the admin UI, but for now, the
user can still change the framework.
> - For those places where technical terms really are necessary (and I’d argue
> that should be nowhere), tooltips should be pervasive to explain what
> everything means. q? fq? fl? df?
Agreed. We've recently changed to a completely new UI that has far more
cloud capability included, and it's possible that some of this has
regressed. I know the old UI had quite a few tooltips on the Query
screen. The old UI is still accessible in 6.x versions, at least for now.
> - The Data Import UI currently broadcasts your MySQL database password to the
> world for whoever is logged in. In a best-case scenario, the legitimate user
> might have search admin permissions but ideally not full MySQL permissions.
> In a worst-case scenario, they’re a random stranger who used nmap to find an
> open port 8983 and just got closer to rooting your server or at least taking
> all of your data. This feature—showing the password—seems unnecessary. Though
> I take issue with the kludge of shoving a configuration file in an IFRAME or
> something like it in the first place, while it’s still part of the product,
> at least replace whatever comes after the string "password=" and before the
> next space with some asterisks.
The password can be encrypted. The capability is NOT well documented,
and I had a hell of a time figuring out how to generate the password,
which I think I did with openssl. Worse, the *encryption* password is
sitting in the DIH config file in cleartext. This would defeat a casual
observer, but not a determined attacker with some brains.
The inclusion of the full config is one of the ways that the UI's status
as an add-on, instead of a fully functional configuration point, is
> - The Data Import UI always checks the "clean" box by default, which means
> every time I try to do something I almost erase my entire core, which takes a
> full day and a lot of CPU cycles to rebuild. I know this has been in JIRA for
> some time, and it’s still not fixed. And it’s a bug that destroys data!
Yep. Another problem that has apparently escaped our immediate
attention. Make some noise on the Jira. Anyone who's on the dev list
automatically gets a copy of all Jira activity.
Saving the status of that checkbox across browser sessions might be
challenging, but I think that the status within a session could be
handled with cookies.
> - Revealing my true ignorance here, I have no idea how to use the Analysis
It's deceptively simple, and once you DO grok it, it will be your best
friend. With it, you can determine whether your field analysis will
generate matches when you make specific queries.
Basic usage -- pick a field. You probably want "verbose" checked. Type
some query terms into the "query" side. Type the actual text that will
be indexed into a document for that field into the "index" side.
For the query side, it doesn't have the full query parser capability, so
you can't include operators like "AND". If it's a phrase query, don't
include the quotes in what you type. It also can't deal with field
names in the query. These are things that we'd really like to have, but
the analysis screen accesses a back-end analysis request handler, and
that request handler doesn't deal with those things, at least not yet.
When you press the "Analyze" button, Solr will show you how both sides
are analyzed. Any terms from the query side that show up on the index
side will be highlighted with a purple color. You'll have to look at
the last line of "index" analysis output for purple matches and apply
some of your own intelligence to determine whether matches on that
screen actually will result in a query match -- default operator,
operators included in the actual query, etc. If the query is a phrase
query (includes quotes) then the relative term positions will matter as
well as having the highlighted matches.
Determining whether a multi-field query will result in a match will
require using the analysis screen for each field in the query.
> - The Files screen doesn’t say the name of the parent folder at the top, so
> it’s not entirely obvious where I even am on my filesystem or why I want the
> ability to view, but not edit, files through the browser.
Everything in "Files" is found in the core's "conf" directory, or in the
case of SolrCloud, in the configuration that's stored in Zookeeper.
This could be more clear, definitely. The location of the directory or
zookeeper path should be shown there.
At one time, we had the ability to edit the config files in the UI. A
big player (Redhat, if I remember right) filed a security bug against
Solr because of this, so we took it out.
> My general feeling is that the UI does mostly things I don’t understand or
> want, and not the few basic things that I do want. Even with a great sample
> environment, this would still be true. This is why the documentation ends up
> being so important. Again, I would point everyone to successful server
> products that are already out there that have done similar tasks well for
> decades now. If you don’t like WordPress, or the now aged Netscape Enterprise
> Server, then there’s phpMyAdmin, Oracle Application Server, Microsoft IIS,
> etc. They all have web UIs that are for the most part self-explanatory. Good
> software doesn’t force users to learn how it works. It hides the inner
> workings under the interface, so that people never even have to worry about
> it at all.
As already said above, the UI is not the true access point for Solr.
It's a helper that's been added on. The true access point is the
various request handlers forming the HTTP API -- /select, /update, etc
... and the HTTP API lacks complete documentation.
I would like to see a future where the admin UI is more than just an
addon ... but even then, I think the HTTP API will *still* be the most
important piece of the system.