[
https://issues.apache.org/jira/browse/SOLR-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235576#comment-17235576
]
Mark Robert Miller commented on SOLR-14788:
-------------------------------------------
{quote}Mark, I have something of extremely narrow focus that I would like to
bring up.
{quote}
Hey Shawn, sorry, I only recently saw this.
So the real deal with the Overseer is that it ended up a disaster.
I won't get into it again, but when I agreed we needed a node(s) with these
types of capabilities, it was to fulfill a part of the design that I intended
to work on reaching. Essentially, I changed jobs as SolrCloud was released and
my new job was hdfs and related, not solrcloud design and impl and finishing,
etc. Sure, I took time to try and make the thing float, but it was not my work
load or direct task(s).
So the Overseer just doesn't work at all like I envisioned, it doesn't solve
the problem I envisioned, in fact, it made things worse in pretty much every
regard compared to what we had. It was intended to be an optimization and
coordination point that enhanced the system vs the naive path.
That panned out pretty much 0. So when you talk about all these state updates
and ZooKeeper queues, and slow restarts, and lost overseers, and scalability
and all that, it really hardly applies. We hired the Overseer to be a farmer
and instead he was a tractor.
Trying to solve for those silly looping threads and crazy number of state
updates and blocking/locking/slow behavior has been 100% the wrong approach.
Instead, we hire a farmer and this time make sure he is a farmer first.
> Solr: The Next Big Thing
> ------------------------
>
> Key: SOLR-14788
> URL: https://issues.apache.org/jira/browse/SOLR-14788
> Project: Solr
> Issue Type: Task
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Critical
>
> h3.
> [!https://www.unicode.org/consortium/aacimg/1F46E.png!|https://www.unicode.org/consortium/adopted-characters.html#b1F46E]{color:#00875a}*The
> Policeman is on duty!*{color}
> {quote}_{color:#de350b}*When The Policeman is on duty, sit back, relax, and
> have some fun. Try to make some progress. Don't stress too much about the
> impact of your changes or maintaining stability and performance and
> correctness so much. Until the end of phase 1, I've got your back. I have a
> variety of tools and contraptions I have been building over the years and I
> will continue training them on this branch. I will review your changes and
> peer out across the land and course correct where needed. As Mike D will be
> thinking, "Sounds like a bottleneck Mark." And indeed it will be to some
> extent. Which is why once stage one is completed, I will flip The Policeman
> to off duty. When off duty, I'm always* {color:#de350b}*occasionally*{color}
> *down for some vigilante justice, but I won't be walking the beat, all that
> stuff about sit back and relax goes out the window.*{color}_
> {quote}
>
> I have stolen this title from Ishan or Noble and Ishan.
> This issue is meant to capture the work of a small team that is forming to
> push Solr and SolrCloud to the next phase.
> I have kicked off the work with an effort to create a very fast and solid
> base. That work is not 100% done, but it's ready to join the fight.
> Tim Potter has started giving me a tremendous hand in finishing up. Ishan and
> Noble have already contributed support and testing and have plans for
> additional work to shore up some of our current shortcomings.
> Others have expressed an interest in helping and hopefully they will pop up
> here as well.
> Let's organize and discuss our efforts here and in various sub issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
