There is some black magic in schemaless for sure. An interesting thing about it though ... it's not just doing field guessing and dynamic schema mutation, it's also doing some field name normalization (removing whitespace), ID injection (if needed), and locale-aware parsing of incoming data (which of course it needs to do to be effective at guessing). I've had to grapple with this in the Schema Designer backend in that I want the field name normalization and locale-aware parsing, but I don't need the last stage in the URP Chain (mutating the schema) since I have sample docs and a chance to review the suggested fields based on those docs before creating the ConfigSet.
For 9.0, maybe the default config has three options: 1) no transformations on the input data at all; the user must provide input data that matches the schema w/o any transformations, 2) basic transformations and locale-aware parsing, but no schema mutations, and 3) schema mutations (aka schemaless). I'd argue that option #1 is too restrictive and make Solr harder to get started with; I think option #2 is useful but could be trappy in that it's not clear (esp. to beginners) that these simple transformations are happening. I'm all for making option #3 (mutations) an opt-in vs. opt-out as it is now esp. now that we'll soon have a Schema Designer in the UI. On Wed, Apr 28, 2021 at 4:18 PM Timothy Potter <[email protected]> wrote: > > I agree with Alex here. We can't overload beginners with a bunch of > jargon and complexity just because experts understand how to use curl > effectively. I also don't think we should remove a feature b/c one > instance of misuse is found in the wild, sounds like Gus' client was > being lazy. Better docs are welcome of course. > > I actually want to integrate the PostTool with the Schema Designer, so > new users (or whoever really) can post a bunch of docs into the temp > Schema Designer staging area and then tune their schema in the UI. > Makes for a nice getting started experience. > > Tim > > On Wed, Apr 28, 2021 at 3:26 PM Alexandre Rafalovitch > <[email protected]> wrote: > > > > "Good enough/Recommended" for what? Serious question. > > > > Because it may be - more than - good enough to "send files to the > > server", but the post tool is also doing a lot of Solr business logic > > that beginner users may not have understood yet. Like automatic > > commit. Like choosing endpoint and content type based on the file > > extension. Like actually saying what it is doing. Beginners may not > > have the bandwidth to understand all those elements in order to index > > their second document (first document being the tutorial one > > copy/paste here). > > > > Removing a post tool because curl is good enough - in my personal view > > - is abandoning beginners. Unless, that "for what" is clear and the > > gap between curl and post tool is filled in some other ways, through > > better documentation or improved API or whatever. > > > > On the original question, I think the post tool is like DIH and like > > the default schema, people stick to them and push their boundaries > > because our beginner->production story is full of gaps. What to do > > about it though, I am not sure. A suggested warning seems like a > > reasonable non-harmful suggestion, though. > > > > Regards, > > Alex. > > > > On Wed, 28 Apr 2021 at 17:04, Ishan Chattopadhyaya > > <[email protected]> wrote: > > > > > > We should remove the post tool > > > Altogether. Curl is good enough and recommended. > > > > > > On Thu, 29 Apr, 2021, 2:15 am Gus Heck, <[email protected]> wrote: > > >> > > >> I've generally been of the impression/opinion that the Post Tool is > > >> really just a convenience for folks testing out solr to see what it can > > >> do, and not really meant as a production ingestion solution. > > >> > > >> A little while back I had a client that had a third party tool that > > >> "integrated with solr" by invoking post.jar on documents with a script > > >> to loop through all the files in a directory and post them (the third > > >> party software's direct example of how to integrate, not the client's > > >> idea at all). Needless to say this caused difficulties with the > > >> gigabytes of data the third party tool had stored in many directories. > > >> Of course I don't know, but I'd guess that someone with little > > >> experience was tasked with the integration with solr at the third party > > >> software company and they followed some examples... then turned them > > >> into an "integration" blissfully unaware of the limitations of what they > > >> had done. > > >> > > >> I just re-read the ref guide page on post tool, and there's nothing > > >> there to indicate to the reader that this might not be a good production > > >> level solution. Also I notice a couple of recent Jira issues regarding > > >> handling of corner cases of strange (broken) behavior or content in a > > >> web site's response, giving the impression that that user (who reported > > >> both issues) might be treading a path that will stretch the bounds of > > >> what the post tool can/should be relied upon for. > > >> > > >> https://issues.apache.org/jira/browse/SOLR-15381 > > >> https://issues.apache.org/jira/browse/SOLR-15370 > > >> > > >> How do folks feel about adding a warning or info box at the top of post > > >> tool docs indicating that it is not meant as a production solution, only > > >> as a quick way to test out documents. We might also say something more > > >> concrete like "virtually any use for a corpus containing over a few > > >> thousand documents is a bad idea"? ... or something like that, > > >> suggestions welcome... > > >> > > >> If folks agree then it seems that these two issues are likely to be > > >> WONTFIX. > > >> > > >> -Gus > > >> > > >> -- > > >> http://www.needhamsoftware.com (work) > > >> http://www.the111shift.com (play) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
