Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

>> Good software doesn’t force users to learn how it works. It hides the
inner workings under the interface, so that people never even have to
worry about it at all.

I agree - AND the fact is that the amount of effort involved in expressing
(In a really well-made UI) every possible permutation of every command line
option of a really complex tool like SOLR is tremendously high and unlikely
to ever get done unless sponsored by a corporation.

While it's true that the ideal would be UI that makes it all easier (and
such UI is theoretically possible) it rarely gets built because of the cost
involved.  There can be tremendous complexity in expressing that many
moving parts successfully...

My .02 worth anyway...


On Sat, Sep 17, 2016 at 1:52 PM, Bram Van Dam  wrote:

> > I would like to see a future where the admin UI is more than just an
> > addon ... but even then, I think the HTTP API will *still* be the most
> > important piece of the system.
>
> In 4 years of heavily using (many instances and many versions of) Solr,
> the only times when I've used the admin UI has been as a
> debugging/diagnostics tool. For instance to quickly check memory usage
> or to verify data has been loaded.
>
> My (and by extension my employer's and our customers') Solr usage
> probably isn't typical, but I can't imagine anyone relying on the admin
> UI for day-to-day Solr operations.
>
> >> Good software doesn’t force users to learn how it works. It hides the
> inner workings under the interface, so that people never even have to
> worry about it at all.
>
> Administering a system you know nothing about is a recipe for disaster.
> This is just as true for WordPress, MySQL or Oracle as it is for Solr.
> I'm not saying things can't/shouldn't be as easy and clear as possible.
> But some effort to understand the system should be expected by the
> sysadmin/operator.
>
>  - Bram
>
>

Re: Miserable Experience Using Solr. Again.

[Bram Van Dam]

> I would like to see a future where the admin UI is more than just an
> addon ... but even then, I think the HTTP API will *still* be the most
> important piece of the system.

In 4 years of heavily using (many instances and many versions of) Solr,
the only times when I've used the admin UI has been as a
debugging/diagnostics tool. For instance to quickly check memory usage
or to verify data has been loaded.

My (and by extension my employer's and our customers') Solr usage
probably isn't typical, but I can't imagine anyone relying on the admin
UI for day-to-day Solr operations.

>> Good software doesn’t force users to learn how it works. It hides the
inner workings under the interface, so that people never even have to
worry about it at all.

Administering a system you know nothing about is a recipe for disaster.
This is just as true for WordPress, MySQL or Oracle as it is for Solr.
I'm not saying things can't/shouldn't be as easy and clear as possible.
But some effort to understand the system should be expected by the
sysadmin/operator.

 - Bram

Re: Miserable Experience Using Solr. Again.

[Alexandre Rafalovitch]

On 16 September 2016 at 18:30, Stefan Matheis  wrote:
>> … choice between better docs and better UI, I’ll choose a better UI every 
>> time
>
> Aaron, you (as well as all others) are more than welcome to help out - no 
> matter what you do / how you do it.
>
> While we’d obviously would love to get some more hands helping out with the 
> coding parts - improving the UI in terms of wording (as you just pointed out) 
> does help equally as much, if not even more.

And just to back it up with facts:

Let's run this query against my ugly-but-interesting Solr-to-Github
collection ( https://github.com/arafalov/git-to-solr):

http://localhost:8983/solr/git/select?facet.field=committer=10=on=-message:Merge=commitTime:[NOW/YEAR-5YEAR%20TO%20*]=on={!parent%20which=type:commit}fileExt:(css%20js)=10=json

That should be (approximately): "Give me breakdown on committer names
that committed more than 10 times something that includes a css or js
file. Limit this to last 5 years". And you'd get:


 "facet_counts":{
"facet_queries":{},
"facet_fields":{
  "committer":[
"Stefan Matheis",97,
"Upayavira",38,
"Ryan McKinley",33,
"Erik Hatcher",13,
"Jan Høydahl",12,
"Erick Erickson",11,
"Steven Rowe",10]},
...

That's it... In last 5 years! UI design is hard. Especially when the
core developer team are the hard core backend guys. Solr only got the
Reference Guide (as opposed to WIKI) because LucidWorks did the bulk
of initial (and possibly ongoing) work. The pretty website was also
sponsored. We don't have in-the-team UI/HTML/CSS/JS expertise.

However, if you don't mind installing a third party project, Cloudera
Hue is a free (open-source if I remember correctly) interface to a
bunch of big-data components, including Solr. They do nice videos too:
http://gethue.com/search-dashboards/

Or there is a commercial one from LucidWorks themselves.

A lot more people know how to contribute to the documentation and
workflows around that are better. And hopefully will be better yet
again once the migration away from Confluence happens.

Or perhaps somebody will sponsor a full-blown front-end developer to
help out. Of course, _they_ not being a part of the current team would
need to figure out what is possible and what APIs to use. So
documentation would come useful there too.

Regards,
   Alex.


Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: Miserable Experience Using Solr. Again.

[Shawn Heisey]

Responses inline.  More potentially flimsy excuses coming your way.

On 9/15/2016 9:56 PM, Aaron Greenspan wrote:
> My two cents: I’m glad to see the discussion over improved documentation, but 
> if you give me a choice between better docs and better UI, I’ll choose a 
> better UI every time. If contributors are going to spend real time on the 
> concerns raised in this thread, spend the time on making the software better 
> to the point where more docs are unnecessary. All sorts of things could 
> improve that would make the product far more intuitive (and I know, there are 
> probably JIRA entries on most of these already…).

The UI is not really intended to be the way that Solr gets accessed. 
It's mostly just a way for an admin to peer into what Solr is doing.  At
this time, it's not really a good tool for configuration.  The admin UI
in 1.x and 3.x only had one or two ways to change the system ... and one
of those ways was enable/disable on the ping handler.  Everything else
the UI did back then was informational.

In newer versions of Solr, the UI does have some capability to make
changes to the system state -- which it does by accessing the HTTP API,
same as any program you would write that uses Solr.

> - The psuedo-frames in the web UI are the source of all kinds of problems, 
> with lots of weird horizontal scrolling I’ve noticed over the years. It makes 
> the Logging screen in particular infuriating to use. When I click on certain 
> log entries an arbitrary-seeming "false" flips to "true" under the "WARN" 
> statement in the Level column. But on other log entries, it all just goes 
> haywire all over the screen because it’s too big both horizontally and 
> vertically, and then re-condenses as though I’d never clicked, as I mentioned 
> before.

If the UI problems you've found are filed as bugs (or as a single bug
listing the problems), we can take it from there.  My free time is
pretty small, or I would do it myself.

Checking the actual logfile has always been a better option than the
Logging tab in the UI.  The info in ERROR messages is typically too
large for effective display in the UI, and the UI excludes messages that
are at a severity of INFO or less.  The default level for the logfile is
INFO, and is typically very verbose.

> - The top menu on the left is in plain English. The core menu on the bottom 
> is written as though it’s being viewed by a person who only speaks UNIX. For 
> example, there is no space between "Data" and "Import" in "DataImport" and 
> "Segments info" could just be "Segments". Is "Plugins / Stats" two menus in 
> one?

The Solr feature that this accesses is contained in a class named
DataImportHandler.  The most commonly configured URL path for the
handler is "/dataimport".  Is it more important to be grammatically
correct, or connect the wording in the admin UI with what the user
actually sees in solrconfig.xml?  You're probably right that it should
have that space, so it doesn't grate on the nerves of those who like
things to be correct.

On "Segments Info" ... brevity counts for a lot in an interface.  That
sounds like a good change.

> - "Ping" in the menu takes you nowhere in particular and shouldn’t really be 
> a menu item. It should be part of the main dashboard with all of the other 
> tech stats (which I do like) or a menu called "Status". (Why would one core 
> ping faster than another anyway? If this is really for "cloud" installations 
> where cores can be split up on different servers, why am I seeing it when 
> everything is local and immediate?)

Good point.  I'm not sure why the time for the ping is so prominently
displayed.  Ping isn't really about speed -- it's about whether or not
the server is up and functional.  It's also a legacy feature that sort
of works with Cloud, but isn't really aware of Cloud.  Plenty of room
for improvement in the UI, in the ping feature itself, and the docs.

> - On the Data Import page, the expandable icons are [-] when they’re expanded 
> and still [-] when they’re collapsed. Extremely confusing.

That's definitely a bug.  The priority of that bug might be debatable.

> - The Data Import UI makes no mention anywhere of the ability to import from 
> MySQL, which is 99% of what I want to do with this product. It doesn’t tell 
> me how to set up the MySQL connector, doesn’t give me a button that turns it 
> on in some modular fashion, doesn’t tell me if the server connection is 
> successful, doesn’t let me easily enter or edit credentials, doesn’t let me 
> edit my queries anywhere, and doesn’t let me test out a new query and see how 
> it might fit into the Solr schema. These deficiencies are presumably also 
> true for any database data source, e.g. Postgres/DB2/ODBC/whatever—which also 
> are not listed, were I curious to know what Solr can do just by looking at 
> the product itself.
>
> - Nor does the Data Import UI have another section for picking a folder on 
> the filesystem that might contain PDFs I 

Re: Miserable Experience Using Solr. Again.

[Stefan Matheis]

> … choice between better docs and better UI, I’ll choose a better UI every time

Aaron, you (as well as all others) are more than welcome to help out - no 
matter what you do / how you do it.

While we’d obviously would love to get some more hands helping out with the 
coding parts - improving the UI in terms of wording (as you just pointed out) 
does help equally as much, if not even more.

When i've started this whole new Admin UI thing, my intensions were primarily 
to make it look like it was from recent times and not a century ago. Afterwards 
Upayavira joined in to upgrade the frontend architecture to the current state 
of art - which by now didn’t help as much as we’d expected for others to 
contribute. I’m running out of ideas what else could help. We are here in 
backend country and not that attractive for capable frontend developers.

We both came up with whatever we could - neither of us is a designer, at most a 
random guy with two eyes. In certain situations i’m the same as you, i’m the 
first person to critize this and that - i often see what others could improve, 
but as often i do not realize that i could do the very same for projects where 
i’m involved. and that’s for a variety of reasons.

To sum it up: if you (again, as well as others) do not speak up - our hands are 
tied. of course it’s easier to report a specific bug that gets fixed, but 
nobody said it’s the only thing you can do. as helpful and needed it is to have 
people working on the documentation instead of contributing code - as important 
are suggestions on the ui itself. you don’t have to actually do it, especially 
not if it’s an area where you can’t help .. but you are one of many using it 
day in, day out. 

And that goes for all the things .. wordings, usability as well as and 
especially design. the ui still looks like (actually is) my first 
work-in-progress draft from years ago - and the reason therefore is certainly 
not because we all love it to death and refuse the change the smallest bits.

those were a bit more than my two cents, but they needed to get off my chest.

-Stefan


On September 16, 2016 at 5:56:34 AM, Aaron Greenspan 
(aaron.greens...@plainsite.org) wrote:
> Hi again,
>  
> My two cents: I’m glad to see the discussion over improved documentation, but 
> if you give  
> me a choice between better docs and better UI, I’ll choose a better UI every 
> time. If contributors  
> are going to spend real time on the concerns raised in this thread, spend the 
> time on making  
> the software better to the point where more docs are unnecessary. All sorts 
> of things  
> could improve that would make the product far more intuitive (and I know, 
> there are probably  
> JIRA entries on most of these already…).
>  
> - The psuedo-frames in the web UI are the source of all kinds of problems, 
> with lots of weird  
> horizontal scrolling I’ve noticed over the years. It makes the Logging screen 
> in particular  
> infuriating to use. When I click on certain log entries an arbitrary-seeming 
> "false"  
> flips to "true" under the "WARN" statement in the Level column. But on other 
> log entries,  
> it all just goes haywire all over the screen because it’s too big both 
> horizontally and  
> vertically, and then re-condenses as though I’d never clicked, as I mentioned 
> before.  
>  
> - The top menu on the left is in plain English. The core menu on the bottom 
> is written as though  
> it’s being viewed by a person who only speaks UNIX. For example, there is no 
> space between  
> "Data" and "Import" in "DataImport" and "Segments info" could just be 
> "Segments". Is  
> "Plugins / Stats" two menus in one?
>  
> - "Ping" in the menu takes you nowhere in particular and shouldn’t really be 
> a menu item.  
> It should be part of the main dashboard with all of the other tech stats 
> (which I do like)  
> or a menu called "Status". (Why would one core ping faster than another 
> anyway? If this  
> is really for "cloud" installations where cores can be split up on different 
> servers,  
> why am I seeing it when everything is local and immediate?)
>  
> - On the Data Import page, the expandable icons are [-] when they’re expanded 
> and still  
> [-] when they’re collapsed. Extremely confusing.
>  
> - The Data Import UI makes no mention anywhere of the ability to import from 
> MySQL, which  
> is 99% of what I want to do with this product. It doesn’t tell me how to set 
> up the MySQL connector,  
> doesn’t give me a button that turns it on in some modular fashion, doesn’t 
> tell me if the  
> server connection is successful, doesn’t let me easily enter or edit 
> credentials, doesn’t  
> let me edit my queries anywhere, and doesn’t let me test out a new query and 
> see how it might  
> fit into the Solr schema. These deficiencies are presumably also true for any 
> database  
> data source, e.g. Postgres/DB2/ODBC/whatever—which also are not listed, were 
> I curious  
> to know what Solr can do just 

Re: Miserable Experience Using Solr. Again.

[Aaron Greenspan]

Hi again,

My two cents: I’m glad to see the discussion over improved documentation, but 
if you give me a choice between better docs and better UI, I’ll choose a better 
UI every time. If contributors are going to spend real time on the concerns 
raised in this thread, spend the time on making the software better to the 
point where more docs are unnecessary. All sorts of things could improve that 
would make the product far more intuitive (and I know, there are probably JIRA 
entries on most of these already…).

- The psuedo-frames in the web UI are the source of all kinds of problems, with 
lots of weird horizontal scrolling I’ve noticed over the years. It makes the 
Logging screen in particular infuriating to use. When I click on certain log 
entries an arbitrary-seeming "false" flips to "true" under the "WARN" statement 
in the Level column. But on other log entries, it all just goes haywire all 
over the screen because it’s too big both horizontally and vertically, and then 
re-condenses as though I’d never clicked, as I mentioned before.

- The top menu on the left is in plain English. The core menu on the bottom is 
written as though it’s being viewed by a person who only speaks UNIX. For 
example, there is no space between "Data" and "Import" in "DataImport" and 
"Segments info" could just be "Segments". Is "Plugins / Stats" two menus in one?

- "Ping" in the menu takes you nowhere in particular and shouldn’t really be a 
menu item. It should be part of the main dashboard with all of the other tech 
stats (which I do like) or a menu called "Status". (Why would one core ping 
faster than another anyway? If this is really for "cloud" installations where 
cores can be split up on different servers, why am I seeing it when everything 
is local and immediate?)

- On the Data Import page, the expandable icons are [-] when they’re expanded 
and still [-] when they’re collapsed. Extremely confusing.

- The Data Import UI makes no mention anywhere of the ability to import from 
MySQL, which is 99% of what I want to do with this product. It doesn’t tell me 
how to set up the MySQL connector, doesn’t give me a button that turns it on in 
some modular fashion, doesn’t tell me if the server connection is successful, 
doesn’t let me easily enter or edit credentials, doesn’t let me edit my queries 
anywhere, and doesn’t let me test out a new query and see how it might fit into 
the Solr schema. These deficiencies are presumably also true for any database 
data source, e.g. Postgres/DB2/ODBC/whatever—which also are not listed, were I 
curious to know what Solr can do just by looking at the product itself.

- Nor does the Data Import UI have another section for picking a folder on the 
filesystem that might contain PDFs I want to import with Tika.

- There is no field picker on the Query screen, but I just spent all of that 
time defining my fields in those XML files I can’t edit or auto-generate 
through the UI. That means I have to do extra work to remember them all. But 
then there is a field picker on the Analysis screen?

- How do I restart Solr from the UI? Or change memory allocation settings? Can 
I?

- How do I change the port the UI is running on from the UI? Or limit the IP 
addresses Jetty is binding to? Can I?

- How do I change log settings through the UI? Can I?

- For those places where technical terms really are necessary (and I’d argue 
that should be nowhere), tooltips should be pervasive to explain what 
everything means. q? fq? fl? df?

- The Data Import UI currently broadcasts your MySQL database password to the 
world for whoever is logged in. In a best-case scenario, the legitimate user 
might have search admin permissions but ideally not full MySQL permissions. In 
a worst-case scenario, they’re a random stranger who used nmap to find an open 
port 8983 and just got closer to rooting your server or at least taking all of 
your data. This feature—showing the password—seems unnecessary. Though I take 
issue with the kludge of shoving a configuration file in an IFRAME or something 
like it in the first place, while it’s still part of the product, at least 
replace whatever comes after the string "password=" and before the next space 
with some asterisks.

- The Data Import UI always checks the "clean" box by default, which means 
every time I try to do something I almost erase my entire core, which takes a 
full day and a lot of CPU cycles to rebuild. I know this has been in JIRA for 
some time, and it’s still not fixed. And it’s a bug that destroys data!

- Revealing my true ignorance here, I have no idea how to use the Analysis 
screen.

- The Files screen doesn’t say the name of the parent folder at the top, so 
it’s not entirely obvious where I even am on my filesystem or why I want the 
ability to view, but not edit, files through the browser.

My general feeling is that the UI does mostly things I don’t understand or 
want, and not the few basic things that I do want. Even with a great 

Re: Miserable Experience Using Solr. Again.

[Nick Vasilyev]

Just wanted to chime in on the technical set-up of the Solr "petting zoo",
I think I can help here; just let me know what you need.

Here is the idea; just have a vagrant box with ansible provisioning Zoo
keepers and Solr, creating collections, and etc That way anyone
starting out can just clone the repo, 'vagrant up' and have a fully
functional environment in no time. Setting up Solr is not the hard part and
I think it takes a little something from the experience, but if it would
help someone get started. Just send me an e-mail off line and let me know.

I do some work on an open source Solr python library and I use a similar
instance to run through unit tests on supported versions of python with
some of the latest versions of Solr; it works great and most of the work is
already done.


On Thu, Sep 15, 2016 at 2:39 PM, Shawn Heisey  wrote:

> On 9/15/2016 8:24 AM, Alexandre Rafalovitch wrote:
> > The WIKI may be an official community-contributing forum, but its
> > technological implementation has gotten so bad it is impossible to
> > update. Every time I change the page, it takes minutes (and feels like
> > hours) for the update to come through. No clue what to do about that
> > though.
>
> Interestingly, even though it takes several minutes for the change
> request to finish, the wiki actually updates almost immediately after
> pushing the button.  The page load (and the resulting email to the
> mailing list) just takes forever.  I discovered this by looking at the
> page in another tab while waiting for the page load to get done.
>
> As I understand it, MoinMoin is entirely filesystem-based, a typical
> config doesn't use a database.  Apache has a LOT of MoinMoin installs
> running on wiki.apache.org.  I think the performance woes are a case of
> a technology that's not scalable enough for how it's being used.
>
> > I feel that it would be cool to have a live tutorial. Perhaps a
> > special collection that, when bootstrapped from, provides tutorial,
> > supporting data, smart interface to play with that data against that
> > same instance, etc. It could also have a static read-only export, but
> > the default experience should be interactive ("bin/solr start -e
> > tutorial" or even "bin/solr start -e
> > http://www.example.com/tutorial;).
>
> That is an interesting idea.  I can envision a tutorial example, a
> canned source directory for indexing data into it, and a third volume of
> documentation, specifically for learning with that index.  It could
> include a section on changing the schema, reindexing, and seeing how
> those changes affect indexing and queries.
>
> > And it should be something that very strongly focuses on teaching new
> > users to fish, not just use the variety of seafood Solr comes with. A
> > narrative showing how different parts of Solr come together and how to
> > troubleshoot those, as opposed to taking each element (e.g. Query
> > Parser) individually and covering them super-comprehensively. That
> > last one is perfect in the reference guide, but less than friendly to
> > a beginner.
>
> Yes, yes, yes.
>
> Thanks,
> Shawn
>
>

Re: Miserable Experience Using Solr. Again.

[Shawn Heisey]

On 9/15/2016 8:24 AM, Alexandre Rafalovitch wrote:
> The WIKI may be an official community-contributing forum, but its
> technological implementation has gotten so bad it is impossible to
> update. Every time I change the page, it takes minutes (and feels like
> hours) for the update to come through. No clue what to do about that
> though. 

Interestingly, even though it takes several minutes for the change
request to finish, the wiki actually updates almost immediately after
pushing the button.  The page load (and the resulting email to the
mailing list) just takes forever.  I discovered this by looking at the
page in another tab while waiting for the page load to get done.

As I understand it, MoinMoin is entirely filesystem-based, a typical
config doesn't use a database.  Apache has a LOT of MoinMoin installs
running on wiki.apache.org.  I think the performance woes are a case of
a technology that's not scalable enough for how it's being used.

> I feel that it would be cool to have a live tutorial. Perhaps a
> special collection that, when bootstrapped from, provides tutorial,
> supporting data, smart interface to play with that data against that
> same instance, etc. It could also have a static read-only export, but
> the default experience should be interactive ("bin/solr start -e
> tutorial" or even "bin/solr start -e
> http://www.example.com/tutorial;). 

That is an interesting idea.  I can envision a tutorial example, a
canned source directory for indexing data into it, and a third volume of
documentation, specifically for learning with that index.  It could
include a section on changing the schema, reindexing, and seeing how
those changes affect indexing and queries.

> And it should be something that very strongly focuses on teaching new
> users to fish, not just use the variety of seafood Solr comes with. A
> narrative showing how different parts of Solr come together and how to
> troubleshoot those, as opposed to taking each element (e.g. Query
> Parser) individually and covering them super-comprehensively. That
> last one is perfect in the reference guide, but less than friendly to
> a beginner.

Yes, yes, yes.

Thanks,
Shawn

Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

Maybe it's too much to manage without a corporate sponsor, as you say...

But what about a cloneable AWS instance which people can then take
responsibility for themselves?  Or a set of VM's that could be downloaded?
Or a Docker?

I haven't done this so there may be roadblocks I'm unaware of - and I
haven't looked at the cost of keeping the VMs somewhere for download...

Although, quite honestly if someone had told me it was $25 to ship a usb
drive with the VM's on it I would have paid it without hesitation if I
could have prevented the weeks of hair-pulling I went through...

but my thought is:


   - Build AWS Instance so that it's cloneable and set up correctly (at
   free level if possible, at minimal viable level if not)
   - Turn it off but make the ability to clone it available publicly
   - Interested parties can then clone and run their own copy
   -   (hopefully at the free level, or for a small amount of money
   since they'll only be running it for a few hours at a time)
   - If necessary - add a tutorial that explains exactly how to clone and
   get it running...  (Yes, it adds AWS to the mix which adds complexity and
   so may not be good)


Alternatively - provide a VirtualBox VM (or set of VMs) that are
downloadable (don't care if it takes overnight) -- Or a Docker image...

I get that it's a little complex - especially if you want a "real" minimal
cloud setup which I would consider to be 3 Zookeeper VM's plus 2-3 Solr
VMs...

It may be too much, but that's what I've been wondering about for the last
year or so...



On Thu, Sep 15, 2016 at 9:09 AM, Alexandre Rafalovitch 
wrote:

> On 15 September 2016 at 21:47, John Bickerstaff
>  wrote:
> > One thing I'd like to suggest is that I believe the ideal tutorial does
> not
> > require someone to even install the software.
>
> Well, if somebody would just agree to run a hosted read-only instance
> of Solr we could totally do that by doing a tutorial that links to a
> separate pre-built collection for each step. That would be extra
> awesome because people could run alternative live queries against
> those instances. And with collections being read-only, there is no
> need to reset them or do any other management. Just initial setup and
> ongoing bandwidth.
>
> Unfortunately, I do not want to run that as an individual and no
> corporate sponsors have come through yet (I talked to several).
>
> Regards,
>Alex.
>
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>

Re: Miserable Experience Using Solr. Again.

[Alexandre Rafalovitch]

On 15 September 2016 at 21:47, John Bickerstaff
 wrote:
> One thing I'd like to suggest is that I believe the ideal tutorial does not
> require someone to even install the software.

Well, if somebody would just agree to run a hosted read-only instance
of Solr we could totally do that by doing a tutorial that links to a
separate pre-built collection for each step. That would be extra
awesome because people could run alternative live queries against
those instances. And with collections being read-only, there is no
need to reset them or do any other management. Just initial setup and
ongoing bandwidth.

Unfortunately, I do not want to run that as an individual and no
corporate sponsors have come through yet (I talked to several).

Regards,
   Alex.


Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

YES, YES, YES!!!

I think nearly everyone on this list will agree that getting started with
almost any open-source project is agony - it's just that we've all gotten
used to sucking it up and getting past it.  Solr, given it's many moving
parts and multiple ways of doing things is "worse" than several other
projects I've dealt with in this sense.

Fortunately, I got paid to learn because my employer expected this from
open-source projects, so it was OK with me - although I did have many
hair-pulling moments in the first couple of months that would have been
totally eliminated by something like this.

One thing I'd like to suggest is that I believe the ideal tutorial does not
require someone to even install the software.  So many open source projects
have tutorials that begin with:

"OK, now that you've figured out the incredibly complex and poorly
documented method of installing the software and getting it running, we'll
begin to explain how to use it..."

I'd suggest that a really good tutorial comes self-contained inside a VM or
Docker or AWS image (that can be cloned) so that interested people do not
have to even go through the install to start learning.  I will also suggest
that running everything on the same machine on loopback is not really
helpful for learning how to make something work on separate servers or VM's
in production.

Of course, you want a "Solr Install Module" as part of the training - but
especially for people who want to learn about using SOLR and have a
production system, forcing them through a complete setup may not be the
best way...

My two cents worth anyway...

I have a number of years (long ago) in Tech writing and instructional
design.  I enjoy building training materials.  I'm happy to volunteer to be
a point person on this and build the docs and tutorials (with support from
the community) since I don't know it all by any means.

We could start with a proposed "table of contents" about what should be
covered and work from there.



On Thu, Sep 15, 2016 at 8:24 AM, Alexandre Rafalovitch 
wrote:

> The WIKI may be an official community-contributing forum, but its
> technological implementation has gotten so bad it is impossible to
> update. Every time I change the page, it takes minutes (and feels like
> hours) for the update to come through. No clue what to do about that
> though.
>
> Creating the short, slim Solr User guide I feel should be in the same
> discussion as rewriting tutorials (shipped vs website ones) and
> shipped examples. One discussion, multiple ducks.
>
> I feel that it would be cool to have a live tutorial. Perhaps a
> special collection that, when bootstrapped from, provides tutorial,
> supporting data, smart interface to play with that data against that
> same instance, etc. It could also have a static read-only export, but
> the default experience should be interactive ("bin/solr start -e
> tutorial"  or even "bin/solr start -e
> http://www.example.com/tutorial;).
>
> And it should be something that very strongly focuses on teaching new
> users to fish, not just use the variety of seafood Solr comes with. A
> narrative showing how different parts of Solr come together and how to
> troubleshoot those, as opposed to taking each element (e.g. Query
> Parser) individually and covering them super-comprehensively. That
> last one is perfect in the reference guide, but less than friendly to
> a beginner.
>
> Regards,
> Alex.
>
>
>
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 15 September 2016 at 20:24, john saylor  wrote:
> > hey
> >
> > On 09/15/16 04:35, Jan Høydahl wrote:
> >>
> >> and the official user-contributed
> >> docs is at http://wiki.apache.org/solr/ 
> >>
> >> But I wonder if we should consider creating an official, slim Solr User
> >> Guide
> >> as well, for end users, structured as a getting-started guide and with
> >> focus
> >> on how you achieve a task, not documenting all 99 parameters a plugin
> can
> >> take.
> >
> >
> > this sounds like a need to be filled.
> >
> > in another context (electronic music), the csound community created a
> floss
> > manual that i think has fulfilled many of these needs for them:
> > http://floss.booktype.pro/csound/preface/
> >
> > so maybe something like this could work for solr? i don't know a whole
> lot
> > about the software underlying the foss site, just a somewhat tangential
> > familiarity based upon consulting this resource and finding it answered
> my
> > questions pretty well.
> >
> > one size does not fit everyone.
>

Re: Miserable Experience Using Solr. Again.

[Alexandre Rafalovitch]

The WIKI may be an official community-contributing forum, but its
technological implementation has gotten so bad it is impossible to
update. Every time I change the page, it takes minutes (and feels like
hours) for the update to come through. No clue what to do about that
though.

Creating the short, slim Solr User guide I feel should be in the same
discussion as rewriting tutorials (shipped vs website ones) and
shipped examples. One discussion, multiple ducks.

I feel that it would be cool to have a live tutorial. Perhaps a
special collection that, when bootstrapped from, provides tutorial,
supporting data, smart interface to play with that data against that
same instance, etc. It could also have a static read-only export, but
the default experience should be interactive ("bin/solr start -e
tutorial"  or even "bin/solr start -e
http://www.example.com/tutorial;).

And it should be something that very strongly focuses on teaching new
users to fish, not just use the variety of seafood Solr comes with. A
narrative showing how different parts of Solr come together and how to
troubleshoot those, as opposed to taking each element (e.g. Query
Parser) individually and covering them super-comprehensively. That
last one is perfect in the reference guide, but less than friendly to
a beginner.

Regards,
Alex.




Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 15 September 2016 at 20:24, john saylor  wrote:
> hey
>
> On 09/15/16 04:35, Jan Høydahl wrote:
>>
>> and the official user-contributed
>> docs is at http://wiki.apache.org/solr/ 
>>
>> But I wonder if we should consider creating an official, slim Solr User
>> Guide
>> as well, for end users, structured as a getting-started guide and with
>> focus
>> on how you achieve a task, not documenting all 99 parameters a plugin can
>> take.
>
>
> this sounds like a need to be filled.
>
> in another context (electronic music), the csound community created a floss
> manual that i think has fulfilled many of these needs for them:
> http://floss.booktype.pro/csound/preface/
>
> so maybe something like this could work for solr? i don't know a whole lot
> about the software underlying the foss site, just a somewhat tangential
> familiarity based upon consulting this resource and finding it answered my
> questions pretty well.
>
> one size does not fit everyone.

Re: Miserable Experience Using Solr. Again.

[john saylor]

hey

On 09/15/16 04:35, Jan Høydahl wrote:

and the official user-contributed
docs is at http://wiki.apache.org/solr/ 

But I wonder if we should consider creating an official, slim Solr User Guide
as well, for end users, structured as a getting-started guide and with focus
on how you achieve a task, not documenting all 99 parameters a plugin can
take.


this sounds like a need to be filled.

in another context (electronic music), the csound community created a 
floss manual that i think has fulfilled many of these needs for them:

http://floss.booktype.pro/csound/preface/

so maybe something like this could work for solr? i don't know a whole 
lot about the software underlying the foss site, just a somewhat 
tangential familiarity based upon consulting this resource and finding 
it answered my questions pretty well.


one size does not fit everyone.

Re: Miserable Experience Using Solr. Again.

[Shawn Heisey]

On 9/15/2016 2:35 AM, Jan Høydahl wrote:
> But I wonder if we should consider creating an official, slim Solr
> User Guide as well, for end users, structured as a getting-started
> guide and with focus on how you achieve a task, not documenting all 99
> parameters a plugin can take. 

Yes, I like this idea.  In this guide, start with a short section that
defines some terminology, clarifies some concepts, and has enough of an
architecture description that a reader can understand how the major
pieces fit together.  Emphasis on "short".

If we provide beginners with the right building blocks, they'll be able
to figure out more on their own, and when they DO come here, there's a
greater chance they'll be able to ask coherent questions.

Thanks,
Shawn

Re: Miserable Experience Using Solr. Again.

[Jan Høydahl]

I was not proposing StackOverflow as some official docs.
Solr’s only official doc is the RefGuide, and the official user-contributed
docs is at http://wiki.apache.org/solr/ 

But it may be that some of you Solr users want to contribute to StackOverflow’s
Solr topic, both in requesting topics, writing short snippets and fixing what is
wrong.

But I wonder if we should consider creating an official, slim Solr User Guide
as well, for end users, structured as a getting-started guide and with focus
on how you achieve a task, not documenting all 99 parameters a plugin can
take.

Regarding the links in the admin, the “Documentation” link should perhaps
link to http://lucene.apache.org/solr/resources.html#documentation 

The other links seem ok though.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 14. sep. 2016 kl. 23.11 skrev Gus Heck :
> 
> While stack overflow is a great place, and the more good info that exists
> there, the merrier, I think Solr should have it's own complete docs, in
> addition to anything found on 3rd party sites. Each hop to a new location
> is a chance for the user to get lost, and the content on 3rd party sites
> could be wrong, out of date without folks here being aware of it as
> quickly.
> 
> Also, I just noticed that there seem to be some links at the bottom of the
> admin UI, but they often run off the bottom and can be easily missed. The
> "documentation" link doesn't actually lead to the documentation... Maybe
> those should be along the top where they would be easily seen and the
> link(s?) fixed up?
> 
> -Gus
> 
> On Wed, Sep 14, 2016 at 3:27 PM, Jan Høydahl  wrote:
> 
>>> If you could decide, what kind of documentation would you want from the
>> project? A very short “Solr Quick start guide”? with step-by-step
>> instructions for the most common tasks from a User perspective?
>> 
>> I just became aware of StackOverflow’s Documentation project, which also
>> has a solr topic:
>> http://stackoverflow.com/documentation/solr > documentation/solr>
>> Perhaps that could also be a good place to contribute HOWTOs and more
>> end-user focused docs?
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>> 
> 
> 
> -- 
> http://www.the111shift.com

Re: Miserable Experience Using Solr. Again.

[Gus Heck]

While stack overflow is a great place, and the more good info that exists
there, the merrier, I think Solr should have it's own complete docs, in
addition to anything found on 3rd party sites. Each hop to a new location
is a chance for the user to get lost, and the content on 3rd party sites
could be wrong, out of date without folks here being aware of it as
quickly.

Also, I just noticed that there seem to be some links at the bottom of the
admin UI, but they often run off the bottom and can be easily missed. The
"documentation" link doesn't actually lead to the documentation... Maybe
those should be along the top where they would be easily seen and the
link(s?) fixed up?

-Gus

On Wed, Sep 14, 2016 at 3:27 PM, Jan Høydahl  wrote:

> > If you could decide, what kind of documentation would you want from the
> project? A very short “Solr Quick start guide”? with step-by-step
> instructions for the most common tasks from a User perspective?
>
> I just became aware of StackOverflow’s Documentation project, which also
> has a solr topic:
> http://stackoverflow.com/documentation/solr  documentation/solr>
> Perhaps that could also be a good place to contribute HOWTOs and more
> end-user focused docs?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>


-- 
http://www.the111shift.com

Re: Miserable Experience Using Solr. Again.

[Jan Høydahl]

> If you could decide, what kind of documentation would you want from the 
> project? A very short “Solr Quick start guide”? with step-by-step 
> instructions for the most common tasks from a User perspective?

I just became aware of StackOverflow’s Documentation project, which also has a 
solr topic:
http://stackoverflow.com/documentation/solr 

Perhaps that could also be a good place to contribute HOWTOs and more end-user 
focused docs?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Re: Miserable Experience Using Solr. Again.

[Shawn Heisey]

On 9/13/2016 5:42 PM, Aaron Greenspan wrote:
> I get this on digest mode (and wasn’t even sure my initial message
> went through to the list), so please forgive the delay in responding. 

I've added you as BCC so you'll get this as soon as I send it.  I wrote
most of it last night, and left it to complete in the morning -- and now
I see that Jan has replied with similar information.

> I think the various reactions to my post suggest that a sizable number
> of users (and by "users" I mean those who are not affiliated with
> Apache and who are not core contributors) find Solr difficult to use.
> For me, this was confirmed many months ago when a family friend—a
> non-technical CEO twice my age of a company recently acquired for a
> very sizable sum—came over for dinner and without any prompting from
> anyone began complaining about this impossible program at work called
> Solr that none of his engineers could get to work. By his telling, he
> had several experienced engineers working on it. 

I've been using Solr for about six years now.  When I first got started,
I spent a HUGE amount of time figuring out the most basic things, and I
asked plenty of dumb questions right here on this list.  I think it took
me about three days to get from that initial download of the 1.4.0
archive to a working server that had something besides "collection1" on
it.  It took another month or so beyond that before I could demonstrate
anything usable to my team, and after that had to start writing tools
that would actually create the index without manual intervention.  One
of those tools was an init script.  Now Solr will install an init script
on Unix-like operating systems.

My active production indexes are running on a couple of different 4.x
versions.  I have production 5.x indexes on servers serving a hot
standby role, but they have not been fully vetted, so the primaries
remain on older versions.  It'll be a while before I get around to 6.x.

> I’m aware that issues with Java are not Solr’s fault. But most
> programs still manage to gracefully fail when they are missing a
> dependency, and then clearly report what’s missing. If you’re not
> actually a Java programmer, which I am not, "major.minor 52.0" (for
> example) is meaningless gibberish. "Please download and install JRE
> 1.8 to run this software" would be considerably clearer. How is it
> that Solr can search through millions of files, but it can’t do that? 

I know that in the 5.x days, we had Java version detection in the start
script, so that the start would complain if certain buggy versions of
Java 7 were detected.  I think it would even refuse to start if the
version wasn't new enough.  If we have lost that with 6.x, that needs to
go back in, and we will look at that problem immediately.

On password security:  I hear you.  Part of the issue is that Solr can't
*directly* do security.  It's sitting behind another piece of software
that handles the network and HTTP -- Jetty.  Until recently, Solr really
didn't touch the servlet container, allowing it to do its thing
according to its config files.  Part of this was due to the fact that
before 5.0, we did not know what container was being used -- the user
had the option of deploying in several different containers, and none of
them handled security in quite the same way.  Since 5.0, the only
officially supported container is the Jetty that Solr includes, so we
CAN put container-specific code into Solr.  This is why 5.3 and later
have good support for authentication.

TL;DR info:  When you password protect Solr, the admin UI actually
doesn't get protected.  It is nothing more than static HTML, CSS,
Javascript, and images.  The admin UI actually runs in your browser, not
on the server.  What gets password protection is the HTTP API used for
information, queries, and updates.

You're absolutely right that our documentation and error messages are
completely inadequate for a novice user.  The error messages sometimes
aren't even adequate for an experienced Java developer to know what went
wrong, at least not without examining the source code.

> As for Bram Van Dam’s question about how a settings database would
> work, I don’t think it’s worth getting too specific here, but my
> general response would be, if you need a good model for how to widely
> deploy software—not a perfect model, but a good one—look at WordPress.
> A lot of people use WordPress. Like any software, it has its flaws.
> But average people are able to sign in, with a password (!), change
> their admin settings, and save those settings I’m pretty certain to a
> MySQL schema. I’d love to be able to do that with Solr. 

I concur with what Alexandre said about Wordpress compared to Solr.  The
target audience and deployment method are quite different ... but I take
your point too -- we can learn a lot from projects like WordPress, which
has had to address "first contact" issues in their documentation.

The addition of Zookeeper capability to Solr in 

Re: Miserable Experience Using Solr. Again.

[Jan Høydahl]

> 14. sep. 2016 kl. 01.42 skrev Aaron Greenspan :

First of all, thanks for spending some time to give feedback and opening JIRAs 
(even if some get closed because it is a question, not a bug report).
This list is exactly the right forum to bring up frustrations newbie users 
might have with Solr, and I think we should LISTEN carefully and identify low 
hanging fruits, as Alexandre is also focusing on!

> I’m aware that issues with Java are not Solr’s fault. But most programs still 
> manage to gracefully fail when they are missing a dependency, and then 
> clearly report what’s missing. If you’re not actually a Java programmer, 
> which I am not, "major.minor 52.0" (for example) is meaningless gibberish. 
> "Please download and install JRE 1.8 to run this software" would be 
> considerably clearer. How is it that Solr can search through millions of 
> files, but it can’t do that?

I agree that it would help big time if bin/solr would validate correct Java 
version. Found https://issues.apache.org/jira/browse/SOLR-8080 
 for this, will try to cook up 
something :) 
Also, over in https://issues.apache.org/jira/browse/SOLR-9508 
 I added a check for Java, so 
it will prompt you to install Java before you can install Solr. Should perhaps 
check for min-version here as well.

> 1. I did. The documentation is severely lacking, apparently having been 
> written by project contributors who have vastly different goals than their 
> users. 

Yea, the ref-guide is a huge beast and aims to list every single setting.
Then we have the tutorials that aim to walk new users through installing, 
indexing and searching. But they don’t cover upgrading etc of course.
Then of course you have all the books - which is perhaps the best option right 
now to get quickly up to speed..

If you could decide, what kind of documentation would you want from the 
project? A very short “Solr Quick start guide”? with step-by-step instructions 
for the most common tasks from a User perspective?

> Note the red section at the bottom (which originally wasn’t even there): "No 
> Solr API, including the Admin UI, is designed to be exposed to non-trusted 
> parties. Tune your firewall so that only trusted computers and people are 
> allowed access." If one of my employees tried to pull this I would fire them. 
> Admin UIs in every other product I’ve ever seen are password-protected. 
> Always. Netscape Enterprise Server in 1996 had a password for its admin UI.

The warning is an honest way to tell admins that Solr is not designed to be an 
internet-facing program, like httpd or nginx. That is not to say that you 
cannot secure Solr pretty well with what we already got, but there will 
probably be a bunch of security holes since an internet-facing service is not 
the goal of Solr. It is not either an excuse for not having a password 
protection that is easier to understand.

Still, the truth is that you CAN add authentication to all of Solr, including 
the UI. What is confusing though, is that the static (non-sensitive) parts of 
the UI will load and display, but as soon as the UI attempts to request any 
kind of information from the Solr APIs, it will fail.

In my opinion this is perceived by our users as the Admin UI being insecure, 
and even if technically not true, we should continue work on 
https://issues.apache.org/jira/browse/SOLR-7896 
 (Add a login page for Solr 
Administrative Interface). 

> 2. I have filed several reports on JIRA. Here’s the kind of response I have 
> received in the past:

Again, thanks for contributing all of this, without users who care and suggest 
stuff there would be no progress…
And I apologise if we as a community have not been understanding or had a 
welcoming attitude to the suggestions.

> https://issues.apache.org/jira/browse/SOLR-7896?focusedCommentId=14661324=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14661324

Security is a special case I would say, since the project had an “official” 
attitude that we’d not add any kind of security whatsoever, to not mislead 
people into exposing Solr publicly. But then things changed in 5.2 and all the 
new security stuff got no vetoes anymore, and we’re now in a pretty good 
position on the API level of security. Wrt SOLR-7896, I re-opened that one 
after it got prematurely closed. But apparently not enough users have felt the 
need for it for someone to invest the time or money it takes to implement it. 
And that’s how open source works, as I’m sure you know.

> Lastly, I run a non-profit foundation devoted to transparency, and I think 
> Solr could do a lot to help further my foundation's goals. That’s why I’m 
> using it at all. It’s the kind of project I’d be willing to fund (since I 
> don’t think I can write the code myself in this instance)—except that 

Re: Miserable Experience Using Solr. Again.

[Alexandre Rafalovitch]

On 14 September 2016 at 06:42, Aaron Greenspan
 wrote:
> This is a potential solution, but not one I choose to pursue. For one thing, 
> I am not an idiot. I’ve managed Linux systems for about 18 years now and I’ve 
> been programming for 20. I have learned that I am rarely the best at 
> anything, so sure, I fully admit that there will always be others with much 
> better skills than my own. But I’m an intelligent person with experience in 
> software trying to leave constructive feedback, and being told that my 
> feedback essentially reflects on my own stupidity kind of misses the point. 
> I’m providing feedback because things need fixing.


And that's the part I am confused about. Managing LInux systems is a
real pain in the ... Config files locations differ between
distributions. Upgrades are confusing, error messages are truly
critic. Debugging by dmesg and truss/strace is dark arts. Reading the
logs is nearly an art.

But with that experience, Zookeeper port is an lsof away. Or a ps away
if you want to read it from the parameters. Or a netstat away. Binding
anything to a local subnet is something of a standard firewall
operation. Couple of other things are output as part of "bin/solr
start --help" as well as part of log messages of running examples. I
understand the frustration. I truly do. And my own - committer - focus
is on improving beginner's experience (no, not looking for funding).
But the problems you list are definitely should be minor, not major
pain points to a Linux system administrator.

Solr is not like a WordPress. WordPress is designed for external users
and so is optimized for ease of used at expense of everything else.
Not correctness, not internal security (passwords are plain text,
plugins have access to everything), not ease of "good" development.
And WordPress is a _small_ product. It is a wrong comparison to the
point that apples and oranges are in the same category of fruit.

Solr is like a MySQL at least. And, frankly, changing a root password
in MySQL is also quite a pain. Or BEA weblogic (which I could...)

As to JIRA, the question was of a very high granularity on a use case
that is complicated and is not a bug/feature distinction. It also has
been discussed multiple times on the User list.



Anyway, I see one JIRA in this so far (Admin UI reclosing the log
message). If nobody else opens it, I will and have a go at it in the
next couple of days.

Regards,
   Alex
P.s. Aaron, perhaps you missed it with the digest mode, but I JUST
asked for feedback on an example reading group idea. I would have
expected you to jump on an opportunity as that would mean a direct
access to somebody contributing their contributor (mine!) time to
improve your understanding of Solr at whatever level of knowledge you
currently have. And - if that's not obvious - to see what kind of
things people find difficult to feed it into the next version of Solr.
Several people already showed interest, but you are not among them.
Hopefully, you will see that email and join us on your next read
through the digest. For easy reference, the survey link again is:
https://www.surveymonkey.com/r/JH8S666


Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: Miserable Experience Using Solr. Again.

[Aaron Greenspan]

Hello again…

I get this on digest mode (and wasn’t even sure my initial message went through 
to the list), so please forgive the delay in responding.

I think the various reactions to my post suggest that a sizable number of users 
(and by "users" I mean those who are not affiliated with Apache and who are not 
core contributors) find Solr difficult to use. For me, this was confirmed many 
months ago when a family friend—a non-technical CEO twice my age of a company 
recently acquired for a very sizable sum—came over for dinner and without any 
prompting from anyone began complaining about this impossible program at work 
called Solr that none of his engineers could get to work. By his telling, he 
had several experienced engineers working on it.

I’m aware that issues with Java are not Solr’s fault. But most programs still 
manage to gracefully fail when they are missing a dependency, and then clearly 
report what’s missing. If you’re not actually a Java programmer, which I am 
not, "major.minor 52.0" (for example) is meaningless gibberish. "Please 
download and install JRE 1.8 to run this software" would be considerably 
clearer. How is it that Solr can search through millions of files, but it can’t 
do that?

As for the suggestions that I should (1) read the documentation; (2) file 
reports on JIRA; and (3) hire a consultant if my own skills aren’t up to snuff:

1. I did. The documentation is severely lacking, apparently having been written 
by project contributors who have vastly different goals than their users. 
Example #1: the security issue (I still can’t figure out how to 
password-protect the Solr web UI, a convenience perhaps, but a convenience I 
depend upon because I cannot spend all day handling the combination of the 
command line and Java, neither can most people, and I still can’t figure out 
how to install or use "Zookeeper") is documented here:

https://cwiki.apache.org/confluence/display/solr/Securing+Solr

Note the red section at the bottom (which originally wasn’t even there): "No 
Solr API, including the Admin UI, is designed to be exposed to non-trusted 
parties. Tune your firewall so that only trusted computers and people are 
allowed access." If one of my employees tried to pull this I would fire them. 
Admin UIs in every other product I’ve ever seen are password-protected. Always. 
Netscape Enterprise Server in 1996 had a password for its admin UI. (See 
https://docs.oracle.com/cd/E19957-01/816-5654-10/816-5654-10.pdf, page 62.) 
Cobalt RaQs in 1999 had passwords on their Admin UIs. Home routers have had 
passwords since time immemorial. It’s 2016. Solr is on major release version 6. 
Don’t tell me how to configure my firewall, and certainly don’t tell me that 
firewalls can programmatically block access to "people". (My understanding of 
firewalls is that they block access to IP addresses and/or ports—if there was a 
product that could magically always block certain people, who would bother with 
firewalls?) Even if I configure my firewall to restrict [IP]:8983 to one IP, 
many people may use that IP, especially at a large organization with enough 
data to merit a product like Solr! Fix your dangerously insecure product, give 
it an install script that handles the SSL cert generation, and if I for some 
reason need to do something to turn that fix on, please tell me how.

Note also, going back to Zookeeper, that on 
https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin, 
the documentation states, "Run the following command to upload it to Zookeeper. 
(ensure that the Zookeeper port is correct)". First, what is it talking about? 
I’ve never heard of Zookeeper outside of an actual zoo. Second, it runs on a 
port? Third, which port? Is it 9983, as is only cryptically alluded to below? 
What if it’s not running yet? How do I start it? Is it secure? Is it part of 
Java? Is it part of Solr? Is it even installed? Why is this my problem? Can you 
imagine if any other piece of software involving a password worked this way? 
(Yes, I have read the Apache Zookeeper Wikipedia article. My point about flaws 
in the documentation stands. It is confusing to new users—those who need it 
most.)

Example #2: the potential bug involving the fieldType error message. I searched 
for documentation on the fieldType. Something about the Solr API 
(https://lucene.apache.org/solr/6_2_0/solr-core/org/apache/solr/schema/FieldType.html)
 came up which was not relevant or helpful. It’s easy to say RTFM, but what if 
the product is full of bugs? Those tend not to be in manuals.

After doing this dance enough times I’ve learned that the Solr documentation is 
most often out-of-date or unhelpful. So here I am.

2. I have filed several reports on JIRA. Here’s the kind of response I have 
received in the past:

https://issues.apache.org/jira/browse/SOLR-7896?focusedCommentId=14661324=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14661324

"Please bring this kind 

Re: Miserable Experience Using Solr. Again.

[Shawn Heisey]

On 9/12/2016 3:48 PM, Aaron Greenspan wrote:
> I have been on this list for some time because I know that any time I
> try to do anything related to Solr I’m going to have to spend hours on
> it, wondering why everything has to be so awful, and I just want
> somewhere to provide feedback with the dim hope that the product might
> improve one day. (So far, for my purposes, it hasn’t.) Sure enough, I
> still absolutely hate using Solr, and I have more feedback. 

First, let me thank you for mentioning your experiences.  It's harsh
feedback, but I still welcome it.  I'm going to say some things that may
sound to you like excuses ... and you aren't really wrong to think that,
but we *do* take your comments seriously, and in many cases, we already
know that there are problems we need to solve.

As others have said, and as I'm sure you probably know, open source is
created by people trying to solve a problem for themselves, and is done
on  a volunteer basis.  If a project is lucky, it attracts interested
volunteers and truly magical things happen for the project users.  I
think Solr is a good project, with an awesome community.

Beginner documentation is one of the hardest things to write.  It's
difficult for people who live and breathe the software to view the
system from the perspective of someone who has never touched it at all,
and to write something that explains to that novice exactly how to make
it work.

> I started with a confusing error on the web console, which I still
> can’t figure out how to password protect without going through an
> insanely process involving "ZooKeeper," which I don’t know anything
> about, or have, to the best of my knowledge: Problem accessing /solr/.
> Reason: Forbidden 

This particular part of your message involves an old fight in the Solr
project:  Security.  Those of us who have been in the industry forever
have learned that many of the security features that people expect for
Internet-facing services are not at all helpful for the security of
internal systems like Solr.  The best thing you can do to prevent
problems is to place Solr in a location where it cannot be reached by
people who cannot be trusted.  If you can trust those who have access,
then there's no need for intrinsic security features.

Any security that you layer on top of Solr (encryption, authentication,
etc) is useless if somebody compromises the system that talks to Solr,
which already has all the keys/passwords/etc required to get right in.

As evidenced by the fact that authentication has come to Solr, we *are*
listening to our users that demand security features.  The
authentication feature that you are trying to use, which involves basic
username/password authentication of the API calls that the admin UI
makes (*not* the admin UI itself), was originally developed for
SolrCloud -- which utilizes Zookeeper as a central configuration
database.  Work is underway right now to bring basic authentication to
standalone Solr, but it is not going to be available until at least 6.3,
and may take even longer to finish.  It will also require separate
configuration on every host, which is not required for SolrCloud.

> According to logs, this apparently meant that a MySQL query had failed
> due to a field name change. Since I would have to change my XML
> configuration files, I decided to use the opportunity to upgrade from
> Solr 5.1.4 to 6.2.0. It broke everything. 

For almost ANY software, but especially for an open source package,
upgrading to a new major version is a good way to cause problems, not
usually a good way to solve them.  Solr does maintain compatibility with
configs that are completely current for the later versions in the
previous major release ... but there are a LOT of configs out in the
world (even in the latest versions of their software!) that were
originally designed for Solr 3.x, 4.x, or *early* 5.x.  Solr makes zero
guarantees about configs designed for software that old.

For an example of a similar situation in a different software package: 
If you try to copy configs for the Apache webserver (httpd) from a 2.2
install to a 2.4 install, chances are excellent that you're going to
have to change those configs before Apache will even start, much less
operate as expected.  Upgrading Apache from 2.2 to 2.4 is technically a
"minor" version upgrade, but in terms of capability and configuration,
is similar to a major Solr version upgrade.

> First I was getting errors about "Unsupported major.minor version
> 52.0", so I needed to install the Linux x64 JRE 1.8.0, which I managed
> on CentOS 6 with... yum install openjdk-1.8.0 ...going to Oracle’s web
> site, downloading the latest JRE 1.8 build, and then running... yum
> localinstall jre-8u101-linux-x64.rpm So far so good. But I didn’t have
> JAVA_HOME set properly apparently, so I needed to do the
> not-exactly-intuitive… export
> JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/

Others have already covered this 

Re: Miserable Experience Using Solr. Again.

[Alexandre Rafalovitch]

On 13 September 2016 at 16:46, Alessandro Benedetti
 wrote:
>>  It didn’t say which field type. Buried in the logs I found a reference in
>> the Java stack trace—which *disappears* (and distorts the viewing window
>> horribly) after a few seconds when you try to view it in the web log UI—to
>> the string "units="degrees"".
>>
>
> This si a bug, and it is really annoying, not sure anyone already raised
> it, if not I suggest you to do that :)

I don't remember seeing it opened. So this could be a great practice
for somebody who is good at finding bugs and issues. :-)

I love a good rant. I used to produce those myself. I love even more a
good rant that includes  specific granular improvements others can get
behind. The bug report as suggested above would be a great one example
of such a granular thing.

I would also point out that most of the contributors to Lucene/Solr
open source are able to contribute because somebody _pays_ them to
develop something on top of/with those projects and they hit
limitations they cannot solve in other easier ways. Those _usually_
are the cutting edge features such as CDCR, new performance
improvements, etc. We could always do with _more_ people who will
focus on the more user-oriented features or on making those new
cutting edge features more easily accessible.

Regards,
   Alex.
P.s. And if anybody wants to rant and will be at Lucene/Solr
revolution, I will be more than happy to sit and listen to you during
any of the food breaks. And I'll help figuring out what those granular
improvement suggestions could be. Feel free to reach out directly if
you want to have a rant scheduled too, instead of catching me
organically :-)


Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: Miserable Experience Using Solr. Again.

[Yago Riveiro]

I stuck in 5.3.1 because if upgrade to 5.5 or 6.x my cluster dies.  
  
Doing a rolling upgrade, when I upgrade the second node to 5.5 both die in the
per-sync phase, I don't know what changes in 5.5 but it's demanding a huge
quantity of memory to check if the replica it's in sync.  
  
This kind of stuff and the full re-index (12T)  between major releases are
indeed a pain.  
  
Cryptical errors and a deficient system to get metrics from what it's going on
inside the cluster is another issue, I'm unable to get the throughput in a
collection as a whole, the number of http connection in each node, the
utilization of the jetty thread pool and stuff like that.  
  
Solr is a great tool, but it's hard, too hard to get in.  
\--

  

/Yago Riveiro

![](https://link.nylas.com/open/m7fkqw0yim04itb62itnp7r9/local-
89046b47-a272?r=c29sci11c2VyQGx1Y2VuZS5hcGFjaGUub3Jn)

  
On Sep 13 2016, at 10:46 am, Alessandro Benedetti 
wrote:  

> First of all I second Bram, I am sorry you had a bad experience with Solr,  
but I think that:  
\- without a minimum study and documentation  
\- without trying to follow the best practices  
I think you are going to have a "miserable" experience with any software,  
don't you ?

>

> In addition to Bram :

>

> On Mon, Sep 12, 2016 at 10:48 PM, Aaron Greenspan <  
aaron.greens...@plainsite.org> wrote:  
>  
> It didn’t say which field type. Buried in the logs I found a reference in  
> the Java stack trace—which *disappears* (and distorts the viewing window  
> horribly) after a few seconds when you try to view it in the web log UI—to  
> the string "units="degrees"".  
>

>

> This si a bug, and it is really annoying, not sure anyone already raised  
it, if not I suggest you to do that :)  
But you can use the logs themselves without any problem.

>

> >  
> Apparently there is some aspect of the Thai text field type that Solr  
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.  
>

>

> If you were not using the Thai text, why had you the Thai Text field type  
defined ?  
Keep It Simple Stupid is the way :)  
I find tons of Solr instances in production mith monster solrconfig.xml and  
schema.xml. basically the old default ones, without any particular reason.  
Don't do that !

>

> >  
> Now Solr was complaining about "Error loading class  
> 'solr.admin.AdminHandlers'". So I found the reference to  
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and  
> commented it out. Only then did Solr work again.  
>

>

> Seems to be you didn't take care of reading the update release notes, did  
you ?

>

>  
Cheers  
\--  
\--

>

> Benedetti Alessandro  
Visiting card : [http://about.me/alessandro_benedetti](http://about.me/alessan
dro_benedetti=c29sci11c2VyQGx1Y2VuZS5hcGFjaGUub3Jn)

>

> "Tyger, tyger burning bright  
In the forests of the night,  
What immortal hand or eye  
Could frame thy fearful symmetry?"

>

> William Blake - Songs of Experience -1794 England

Re: Miserable Experience Using Solr. Again.

[Alessandro Benedetti]

First of all I second Bram, I am sorry you had a bad experience with Solr,
but I think that:
-  without a minimum study and documentation
- without trying to follow the best practices
I think you are going to have a "miserable" experience with any software,
don't you ?

In addition to Bram :

On Mon, Sep 12, 2016 at 10:48 PM, Aaron Greenspan <
aaron.greens...@plainsite.org> wrote:
>
>  It didn’t say which field type. Buried in the logs I found a reference in
> the Java stack trace—which *disappears* (and distorts the viewing window
> horribly) after a few seconds when you try to view it in the web log UI—to
> the string "units="degrees"".
>

This si a bug, and it is really annoying, not sure anyone already raised
it, if not I suggest you to do that :)
But you can use the logs themselves without any problem.

>
> Apparently there is some aspect of the Thai text field type that Solr
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>

If you were not using the Thai text, why had you the Thai Text field type
defined ?
Keep It Simple Stupid is the way :)
I find tons of Solr instances in production mith monster solrconfig.xml and
schema.xml. basically the old default ones, without any particular reason.
Don't do that !

>
> Now Solr was complaining about "Error loading class
> 'solr.admin.AdminHandlers'". So I found the reference to
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> commented it out. Only then did Solr work again.
>

Seems to be you didn't take care of reading the update release notes, did
you ?


Cheers
-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Miserable Experience Using Solr. Again.

[Bram Van Dam]

I'm sorry you're having a "miserable" experience "again". That's
certainly not my experience with Solr. That being said:

> First I was getting errors about "Unsupported major.minor version 52.0", so I 
> needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6 with...
> yum install openjdk-1.8.0

This is not a Solr problem. Solr requires Java 8. Java 7 has been
officially end-of-lifed since april 2015. This means no more patches, no
more performance improvements and no more security updates (unless
you're paying Oracle). This is clearly stated in the (very decent) Solr
documentation. To use your own words: Java 7 is an antiquated nightmare
and the rest of the world has moved on to Java 8.

> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I 
> needed to do the not-exactly-intuitive…
> export 
> JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/

You don't need to set JAVA_HOME to run Solr. But if you do have a
JAVA_HOME environment variable, and it points to a wrong Java version,
you're going to have a bad time.

> Then after stopping the old process (with kill -9, since there seems to be no 
> graceful way to shut down Solr)

There's a stop command, which is documented. It's a non-surprising
location and has a non-surprising name. And even if there wasn't, "kill"
would have sufficed.

> There was some kind of problem with StopFilterFactory and the text_general 
> field type. Thanks to Stack Overflow I was able to determine that the 
> apparent problem was that there was a parameter, previously fine, which was 
> no longer fine. So I removed all instances of 
> enablePositionIncrements="true". That helped, but then I ran into a broader 
> error: "Plugin Initializing failure for [schema.xml] fieldType". It didn’t 
> say which field type. Buried in the logs I found a reference in the Java 
> stack trace—which *disappears* (and distorts the viewing window horribly) 
> after a few seconds when you try to view it in the web log UI—to the string 
> "units="degrees"". Sure enough, this string appeared in my schema.xml for a 
> class called "solr.SpatialRecursivePrefixTreeFieldType" that I’m pretty sure 
> I never use. I removed that parameter, and moved on to the next set of errors.

Releases come with release notes and -- when required -- upgrade
instructions and admonitions. It's certainly possible that there's been
an oversight here or there and you're more than welcome to point those out.

> The user interface is still as buggy as an early alpha of most
products, the errors are difficult to understand when they don’t
actually specify what’s wrong (and they almost never do), and there
should have been an automatic process to highlight and fix problems in
old (pre-6) configuration files.

What user interface? Are you talking about the Admin UI? That's a
convenience feature which helps you manage Solr. It makes life a lot
easier, even if it's not perfect. The logs are generally quite good at
explaining what's wrong.

> Never mind the fact that the XML-based configuration process is an
antiquated nightmare when the rest of the world has long since moved
onto databases.

An antiquated nightmare? The rest of the world? How would this work?
What benefit would it possibly have?

You're more than welcome to report any bugs you find
(https://issues.apache.org/jira/browse/SOLR). But I feel like general
ranting on the mailing list isn't very productive. Well, I suppose
venting feels good, so there's that.

Things that would be more productive:

1. Reading the documentation.
2. Taking a basic system administration class or two.
3. Pointing out -- or contributing to -- parts of the documentation that
aren't up to par. Either on the mailing list, or on Jira. Preferably in
a constructive way instead of a "miserable experience"-way.

I feel like you're missing the part where most open source development,
documentation, release management etc is done by volunteers. Volunteers
who tend to scratch their own itch first, and are then kind enough to
donate the fruit of their labour to the rest of the world. You can
certainly make requests, and you can certainly hope for things to improve.

If you're having a "miserable" time "again", then you can always hire a
Solr consultant to do the work for you. You can't demand free stuff to
scratch your every itch. You can either invest your time and figure out
how to do things yourself, or your money and have things done for you.
But there's no such thing as a free lunch.

 - Bram

Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

Sure - ping me off the list and I'll send my text file docs.

They're rough and (of course) focused on what I'm doing, but they just
might relieve some of the pain.

Caveat - all on Linux and command line - no Admin UI api's -- I like the
feel of the command line so I use it.

On Mon, Sep 12, 2016 at 8:41 PM,  wrote:

> Interested for sure
>
> Bill Bell
> Sent from mobile
>
>
> > On Sep 12, 2016, at 4:05 PM, John Bickerstaff 
> wrote:
> >
> > For what it's worth - I found enough frustration upgrading that I decided
> > to "upgrade by replacement"
> >
> > Now, I suppose if you've got a huge dataset to re-index that could be a
> > problem, but just in case an option like that helps you, I'll suggest
> this.
> >
> > 1. Install 6.x on a new machine using the "install for production"
> > instructions
> > 2. Use the configs from one of the sample projects to create an
> > appropriately-named collection
> > 3. Use the ability to "include" your configs into the other configs (they
> > live in separate files)
> >  I can provide more help here if you're interested
> > 4. Re-index all your data into the new version of SOLR...
> >
> > I have rough, but useable docs on this if you are interested in
> attempting
> > this approach.
> >
> > On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> > aaron.greens...@plainsite.org> wrote:
> >
> >> Hi,
> >>
> >> I have been on this list for some time because I know that any time I
> try
> >> to do anything related to Solr I’m going to have to spend hours on it,
> >> wondering why everything has to be so awful, and I just want somewhere
> to
> >> provide feedback with the dim hope that the product might improve one
> day.
> >> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely
> hate
> >> using Solr, and I have more feedback.
> >>
> >> I started with a confusing error on the web console, which I still can’t
> >> figure out how to password protect without going through an insanely
> >> process involving "ZooKeeper," which I don’t know anything about, or
> have,
> >> to the best of my knowledge:
> >>
> >> Problem accessing /solr/. Reason:
> >>
> >>Forbidden
> >>
> >> According to logs, this apparently meant that a MySQL query had failed
> due
> >> to a field name change. Since I would have to change my XML
> configuration
> >> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
> >> 6.2.0. It broke everything.
> >>
> >> First I was getting errors about "Unsupported major.minor version 52.0",
> >> so I needed to install the Linux x64 JRE 1.8.0, which I managed on
> CentOS 6
> >> with...
> >>
> >> yum install openjdk-1.8.0
> >>
> >> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
> >> then running...
> >>
> >> yum localinstall jre-8u101-linux-x64.rpm
> >>
> >> So far so good. But I didn’t have JAVA_HOME set properly apparently, so
> I
> >> needed to do the not-exactly-intuitive…
> >>
> >> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
> >> el6_8.x86_64/jre/
> >>
> >> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
> >> file from the dist/ folder in the old version to the new one. Then after
> >> stopping the old process (with kill -9, since there seems to be no
> graceful
> >> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
> >> over my two core folders from the old server/solr/ folder. I tried to
> start
> >> it up with bin/solr start, and watched the errors roll in.
> >>
> >> There was some kind of problem with StopFilterFactory and the
> text_general
> >> field type. Thanks to Stack Overflow I was able to determine that the
> >> apparent problem was that there was a parameter, previously fine, which
> was
> >> no longer fine. So I removed all instances of enablePositionIncrements="
> true".
> >> That helped, but then I ran into a broader error: "Plugin Initializing
> >> failure for [schema.xml] fieldType". It didn’t say which field type.
> Buried
> >> in the logs I found a reference in the Java stack trace—which
> *disappears*
> >> (and distorts the viewing window horribly) after a few seconds when you
> try
> >> to view it in the web log UI—to the string "units="degrees"". Sure
> enough,
> >> this string appeared in my schema.xml for a class called "solr.
> >> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use.
> I
> >> removed that parameter, and moved on to the next set of errors.
> >>
> >> Apparently there is some aspect of the Thai text field type that Solr
> >> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
> >>
> >> Now Solr was complaining about "Error loading class
> >> 'solr.admin.AdminHandlers'". So I found the reference to
> >> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> >> commented it out. Only then did Solr work again.
> >>
> >> This was not a smooth process. It took about two hours. The user
> interface
> >> is still as 

Re: Miserable Experience Using Solr. Again.

[billnbell]

Interested for sure

Bill Bell
Sent from mobile


> On Sep 12, 2016, at 4:05 PM, John Bickerstaff  
> wrote:
> 
> For what it's worth - I found enough frustration upgrading that I decided
> to "upgrade by replacement"
> 
> Now, I suppose if you've got a huge dataset to re-index that could be a
> problem, but just in case an option like that helps you, I'll suggest this.
> 
> 1. Install 6.x on a new machine using the "install for production"
> instructions
> 2. Use the configs from one of the sample projects to create an
> appropriately-named collection
> 3. Use the ability to "include" your configs into the other configs (they
> live in separate files)
>  I can provide more help here if you're interested
> 4. Re-index all your data into the new version of SOLR...
> 
> I have rough, but useable docs on this if you are interested in attempting
> this approach.
> 
> On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> aaron.greens...@plainsite.org> wrote:
> 
>> Hi,
>> 
>> I have been on this list for some time because I know that any time I try
>> to do anything related to Solr I’m going to have to spend hours on it,
>> wondering why everything has to be so awful, and I just want somewhere to
>> provide feedback with the dim hope that the product might improve one day.
>> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
>> using Solr, and I have more feedback.
>> 
>> I started with a confusing error on the web console, which I still can’t
>> figure out how to password protect without going through an insanely
>> process involving "ZooKeeper," which I don’t know anything about, or have,
>> to the best of my knowledge:
>> 
>> Problem accessing /solr/. Reason:
>> 
>>Forbidden
>> 
>> According to logs, this apparently meant that a MySQL query had failed due
>> to a field name change. Since I would have to change my XML configuration
>> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
>> 6.2.0. It broke everything.
>> 
>> First I was getting errors about "Unsupported major.minor version 52.0",
>> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
>> with...
>> 
>> yum install openjdk-1.8.0
>> 
>> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
>> then running...
>> 
>> yum localinstall jre-8u101-linux-x64.rpm
>> 
>> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
>> needed to do the not-exactly-intuitive…
>> 
>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
>> el6_8.x86_64/jre/
>> 
>> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
>> file from the dist/ folder in the old version to the new one. Then after
>> stopping the old process (with kill -9, since there seems to be no graceful
>> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
>> over my two core folders from the old server/solr/ folder. I tried to start
>> it up with bin/solr start, and watched the errors roll in.
>> 
>> There was some kind of problem with StopFilterFactory and the text_general
>> field type. Thanks to Stack Overflow I was able to determine that the
>> apparent problem was that there was a parameter, previously fine, which was
>> no longer fine. So I removed all instances of 
>> enablePositionIncrements="true".
>> That helped, but then I ran into a broader error: "Plugin Initializing
>> failure for [schema.xml] fieldType". It didn’t say which field type. Buried
>> in the logs I found a reference in the Java stack trace—which *disappears*
>> (and distorts the viewing window horribly) after a few seconds when you try
>> to view it in the web log UI—to the string "units="degrees"". Sure enough,
>> this string appeared in my schema.xml for a class called "solr.
>> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I
>> removed that parameter, and moved on to the next set of errors.
>> 
>> Apparently there is some aspect of the Thai text field type that Solr
>> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>> 
>> Now Solr was complaining about "Error loading class
>> 'solr.admin.AdminHandlers'". So I found the reference to
>> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
>> commented it out. Only then did Solr work again.
>> 
>> This was not a smooth process. It took about two hours. The user interface
>> is still as buggy as an early alpha of most products, the errors are
>> difficult to understand when they don’t actually specify what’s wrong (and
>> they almost never do), and there should have been an automatic process to
>> highlight and fix problems in old (pre-6) configuration files. Never mind
>> the fact that the XML-based configuration process is an antiquated
>> nightmare when the rest of the world has long since moved onto databases.
>> 
>> Maybe this will help someone else out there.
>> 
>> Aaron
>> 
>> PlainSite | http://www.plainsite.org

Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

I would also add that dealing with Java versions has always been a pain
until you get used to the whole "JAVA HOME" thing, but that this isn't
anything to do with SOLR per se - it's just part and parcel of dealing with
open source software that uses Java...

Big changes between major versions of any software are common - there is
some documentation here that may help...

https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6

It was reading this that made me decide to try "upgrade by replace" to
avoid the whole "update" issue entirely - although I also had to upgrade
Java on my VMs...

On Mon, Sep 12, 2016 at 4:05 PM, John Bickerstaff 
wrote:

> For what it's worth - I found enough frustration upgrading that I decided
> to "upgrade by replacement"
>
> Now, I suppose if you've got a huge dataset to re-index that could be a
> problem, but just in case an option like that helps you, I'll suggest this.
>
> 1. Install 6.x on a new machine using the "install for production"
> instructions
> 2. Use the configs from one of the sample projects to create an
> appropriately-named collection
> 3. Use the ability to "include" your configs into the other configs (they
> live in separate files)
>   I can provide more help here if you're interested
> 4. Re-index all your data into the new version of SOLR...
>
> I have rough, but useable docs on this if you are interested in attempting
> this approach.
>
> On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> aaron.greens...@plainsite.org> wrote:
>
>> Hi,
>>
>> I have been on this list for some time because I know that any time I try
>> to do anything related to Solr I’m going to have to spend hours on it,
>> wondering why everything has to be so awful, and I just want somewhere to
>> provide feedback with the dim hope that the product might improve one day.
>> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
>> using Solr, and I have more feedback.
>>
>> I started with a confusing error on the web console, which I still can’t
>> figure out how to password protect without going through an insanely
>> process involving "ZooKeeper," which I don’t know anything about, or have,
>> to the best of my knowledge:
>>
>> Problem accessing /solr/. Reason:
>>
>> Forbidden
>>
>> According to logs, this apparently meant that a MySQL query had failed
>> due to a field name change. Since I would have to change my XML
>> configuration files, I decided to use the opportunity to upgrade from Solr
>> 5.1.4 to 6.2.0. It broke everything.
>>
>> First I was getting errors about "Unsupported major.minor version 52.0",
>> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
>> with...
>>
>> yum install openjdk-1.8.0
>>
>> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
>> then running...
>>
>> yum localinstall jre-8u101-linux-x64.rpm
>>
>> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
>> needed to do the not-exactly-intuitive…
>>
>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el
>> 6_8.x86_64/jre/
>>
>> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
>> file from the dist/ folder in the old version to the new one. Then after
>> stopping the old process (with kill -9, since there seems to be no graceful
>> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
>> over my two core folders from the old server/solr/ folder. I tried to start
>> it up with bin/solr start, and watched the errors roll in.
>>
>> There was some kind of problem with StopFilterFactory and the
>> text_general field type. Thanks to Stack Overflow I was able to determine
>> that the apparent problem was that there was a parameter, previously fine,
>> which was no longer fine. So I removed all instances of
>> enablePositionIncrements="true". That helped, but then I ran into a
>> broader error: "Plugin Initializing failure for [schema.xml] fieldType". It
>> didn’t say which field type. Buried in the logs I found a reference in the
>> Java stack trace—which *disappears* (and distorts the viewing window
>> horribly) after a few seconds when you try to view it in the web log UI—to
>> the string "units="degrees"". Sure enough, this string appeared in my
>> schema.xml for a class called "solr.SpatialRecursivePrefixTreeFieldType"
>> that I’m pretty sure I never use. I removed that parameter, and moved on to
>> the next set of errors.
>>
>> Apparently there is some aspect of the Thai text field type that Solr
>> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>>
>> Now Solr was complaining about "Error loading class
>> 'solr.admin.AdminHandlers'". So I found the reference to
>> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
>> commented it out. Only then did Solr work again.
>>
>> This was not a smooth process. It took about two hours. The user
>> interface is still as buggy as 

Re: Miserable Experience Using Solr. Again.

[John Bickerstaff]

For what it's worth - I found enough frustration upgrading that I decided
to "upgrade by replacement"

Now, I suppose if you've got a huge dataset to re-index that could be a
problem, but just in case an option like that helps you, I'll suggest this.

1. Install 6.x on a new machine using the "install for production"
instructions
2. Use the configs from one of the sample projects to create an
appropriately-named collection
3. Use the ability to "include" your configs into the other configs (they
live in separate files)
  I can provide more help here if you're interested
4. Re-index all your data into the new version of SOLR...

I have rough, but useable docs on this if you are interested in attempting
this approach.

On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
aaron.greens...@plainsite.org> wrote:

> Hi,
>
> I have been on this list for some time because I know that any time I try
> to do anything related to Solr I’m going to have to spend hours on it,
> wondering why everything has to be so awful, and I just want somewhere to
> provide feedback with the dim hope that the product might improve one day.
> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
> using Solr, and I have more feedback.
>
> I started with a confusing error on the web console, which I still can’t
> figure out how to password protect without going through an insanely
> process involving "ZooKeeper," which I don’t know anything about, or have,
> to the best of my knowledge:
>
> Problem accessing /solr/. Reason:
>
> Forbidden
>
> According to logs, this apparently meant that a MySQL query had failed due
> to a field name change. Since I would have to change my XML configuration
> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
> 6.2.0. It broke everything.
>
> First I was getting errors about "Unsupported major.minor version 52.0",
> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
> with...
>
> yum install openjdk-1.8.0
>
> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
> then running...
>
> yum localinstall jre-8u101-linux-x64.rpm
>
> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
> needed to do the not-exactly-intuitive…
>
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
> el6_8.x86_64/jre/
>
> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
> file from the dist/ folder in the old version to the new one. Then after
> stopping the old process (with kill -9, since there seems to be no graceful
> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
> over my two core folders from the old server/solr/ folder. I tried to start
> it up with bin/solr start, and watched the errors roll in.
>
> There was some kind of problem with StopFilterFactory and the text_general
> field type. Thanks to Stack Overflow I was able to determine that the
> apparent problem was that there was a parameter, previously fine, which was
> no longer fine. So I removed all instances of enablePositionIncrements="true".
> That helped, but then I ran into a broader error: "Plugin Initializing
> failure for [schema.xml] fieldType". It didn’t say which field type. Buried
> in the logs I found a reference in the Java stack trace—which *disappears*
> (and distorts the viewing window horribly) after a few seconds when you try
> to view it in the web log UI—to the string "units="degrees"". Sure enough,
> this string appeared in my schema.xml for a class called "solr.
> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I
> removed that parameter, and moved on to the next set of errors.
>
> Apparently there is some aspect of the Thai text field type that Solr
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>
> Now Solr was complaining about "Error loading class
> 'solr.admin.AdminHandlers'". So I found the reference to
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> commented it out. Only then did Solr work again.
>
> This was not a smooth process. It took about two hours. The user interface
> is still as buggy as an early alpha of most products, the errors are
> difficult to understand when they don’t actually specify what’s wrong (and
> they almost never do), and there should have been an automatic process to
> highlight and fix problems in old (pre-6) configuration files. Never mind
> the fact that the XML-based configuration process is an antiquated
> nightmare when the rest of the world has long since moved onto databases.
>
> Maybe this will help someone else out there.
>
> Aaron
>
> PlainSite | http://www.plainsite.org