Re: Overseer documentation

2020-04-24 Thread Cassandra Targett 
Agreed this is great and difficult work, Ilan. Thanks so much for taking it on.

I wonder if some of this maybe should end up in Javadocs? If those classes 
don’t say what they’re doing or seem incomplete (and I have no idea what they 
say about themselves) then we should consider maybe using some of this to 
improve that?

I don’t have much to say about the content - I might understand this all better 
if I read a doc like this though! - except to reply to David’s question to me:
On Apr 23, 2020, 2:36 PM -0500, David Smiley , wrote:


>
> This is "developer documentation".  Cassandra:  I see you created 
> solr/dev-docs/ and I suppose this would best belong there?  Mark Miller had 
> tried Confluence.  Pros/cons there.  I want to ensure readers of the code in 
> Overseer (and maybe other key class or two) notice this dev documentation.  
> Should I add a http link to the GitHub location of the dev doc markdown, or 
> do you recommend something else?
>

Well, I added the directories but I didn’t just make it up - we had a thread in 
the Dev list about doing it and since no one else moved to make it happen and I 
had some internal docs to write when I handed off PMC chair to Anshum, I did 
it. But besides being what we already said we wanted to do, I think it should 
go in our source code for at least all the same reasons we put the Ref Guide in 
the source - that’s where we already are, that’s where contributors need to be, 
and we can ensure the docs are properly relevant to the versions via our 
branching approach.

I have a Google Docs add-on which can convert Google docs to Asciidoc format, 
which for a document this complex I would recommend as a format. The add-on is 
not going to do this doc perfectly, but will get 50% or more the way there. I’d 
be willing to give it a try, but if people want a mix of Asciidoc + Markdown 
formatted files and want to convert this to Markdown, by all means go for it.

Re: Overseer documentation

2020-04-24 Thread Mike Drob 
#2 and #8 were the ones that immediately jumped out at me.

On 2020/04/24 10:33:55, Ilan Ginzburg  wrote: 
> Thanks David, Mike and Shalin. Glad you find this useful!
> 
> Mike, do you mind pointing out here or in the doc which of the bugs
> you saw in production? That way I can start filing Jiras for actual
> issues.
> 
> Ilan
> 
> On Fri, Apr 24, 2020 at 4:01 AM Shalin Shekhar Mangar
>  wrote:
> >
> > This is good stuff Ilan. Thank you for writing and sharing with us. I 
> > intend to take a deeper look at this next week.
> >
> > On Wed, Apr 22, 2020 at 2:36 AM Ilan Ginzburg  wrote:
> >>
> >> Hello Solr devs,
> >>
> >> This is my first post here. I work at Salesforce in France, we're
> >> adopting SolrCloud and we need it to scale more than it currently
> >> does.
> >>
> >> I've looked at Overseer and documented my understanding. I'm sharing
> >> the result, it might help others and is a way to get feedback (I might
> >> have misunderstood some things) and/or collaboration on continuing
> >> documenting the implementation. Basically I started writing the doc I
> >> wanted to find.
> >>
> >> In the process, I believe I've identified what may be a few bugs
> >> (there's a section listing them at the beginning). I've found these by
> >> reading code (not running code), so take with a grain of salt.
> >> I plan to file Jiras for those bugs that do seem real and are
> >> important enough, and then also start working on some to help
> >> fix/improve.
> >>
> >> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/
> >>
> >> This is WIP. Please do not hesitate to provide feedback/leave comments.
> >>
> >> Thanks,
> >> Ilan
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Overseer documentation

2020-04-24 Thread Ilan Ginzburg 
Thanks David, Mike and Shalin. Glad you find this useful!

Mike, do you mind pointing out here or in the doc which of the bugs
you saw in production? That way I can start filing Jiras for actual
issues.

Ilan

On Fri, Apr 24, 2020 at 4:01 AM Shalin Shekhar Mangar
 wrote:
>
> This is good stuff Ilan. Thank you for writing and sharing with us. I intend 
> to take a deeper look at this next week.
>
> On Wed, Apr 22, 2020 at 2:36 AM Ilan Ginzburg  wrote:
>>
>> Hello Solr devs,
>>
>> This is my first post here. I work at Salesforce in France, we're
>> adopting SolrCloud and we need it to scale more than it currently
>> does.
>>
>> I've looked at Overseer and documented my understanding. I'm sharing
>> the result, it might help others and is a way to get feedback (I might
>> have misunderstood some things) and/or collaboration on continuing
>> documenting the implementation. Basically I started writing the doc I
>> wanted to find.
>>
>> In the process, I believe I've identified what may be a few bugs
>> (there's a section listing them at the beginning). I've found these by
>> reading code (not running code), so take with a grain of salt.
>> I plan to file Jiras for those bugs that do seem real and are
>> important enough, and then also start working on some to help
>> fix/improve.
>>
>> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/
>>
>> This is WIP. Please do not hesitate to provide feedback/leave comments.
>>
>> Thanks,
>> Ilan
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Overseer documentation

2020-04-23 Thread Shalin Shekhar Mangar 
This is good stuff Ilan. Thank you for writing and sharing with us. I
intend to take a deeper look at this next week.

On Wed, Apr 22, 2020 at 2:36 AM Ilan Ginzburg  wrote:

> Hello Solr devs,
>
> This is my first post here. I work at Salesforce in France, we're
> adopting SolrCloud and we need it to scale more than it currently
> does.
>
> I've looked at Overseer and documented my understanding. I'm sharing
> the result, it might help others and is a way to get feedback (I might
> have misunderstood some things) and/or collaboration on continuing
> documenting the implementation. Basically I started writing the doc I
> wanted to find.
>
> In the process, I believe I've identified what may be a few bugs
> (there's a section listing them at the beginning). I've found these by
> reading code (not running code), so take with a grain of salt.
> I plan to file Jiras for those bugs that do seem real and are
> important enough, and then also start working on some to help
> fix/improve.
>
>
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/
>
> This is WIP. Please do not hesitate to provide feedback/leave comments.
>
> Thanks,
> Ilan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
Regards,
Shalin Shekhar Mangar.

Re: Overseer documentation

2020-04-23 Thread Mike Drob 
Wow Ilan! This is very comprehensive!

Regarding your list of potential bugs, I can confirm that at least two of
those are accurate and I've seen them in production. We found work-arounds
and got distracted before creating JIRAs, and this was a great reminder of
those war stories.

I think this would be a great addition to dev-docs/ and a link from the
Overseer Javadoc to this file would be reasonable.

I hope to have time to dive into this deeper next week.


Mike

On Thu, Apr 23, 2020 at 2:36 PM David Smiley  wrote:

> Thanks Ilan!
>
> I especially love the lead section "Overseer: from queues to state" with
> the diagram.  Indeed, this is the documentation I (we?) wished already
> existed.  I'd like to try to ensure this part of the document is more
> tightly associated with our project for others to see.
>
> This is "developer documentation".  Cassandra:  I see you created
> solr/dev-docs/ and I suppose this would best belong there?  Mark Miller had
> tried Confluence.  Pros/cons there.  I want to ensure readers of the code
> in Overseer (and maybe other key class or two) notice this dev
> documentation.  Should I add a http link to the GitHub location of the dev
> doc markdown, or do you recommend something else?
>
> For the rest of the doc, there are problem call-outs (picture of a
> triangle hazard with an exclamation point) -- readers can't miss them.   I
> hope those of us that know SolrCloud internals best can look at those
> points closer and maybe file JIRA issues.  That isn't me, honestly.
>
> I've been thinking that some of these problems might best be fixed by
> larger architectural changes rather than incrementally fixing a design with
> substantial weaknesses (and tech-debt complexities).  Noble/Ishan's
> SOLR-13951  issue
> "Avoid replica state updates to state.json" will help a lot but there will
> be much more to be done to address Solr's over-reliance on the Overseer to
> accomplish collection/cluster management. This is something I want to
> contemplate more.  I hope Curator recipes may be a source of inspiration,
> which I plan to review this weekend.
>
> (note: I work with Ilan)
> ~ David
>
> On Tue, Apr 21, 2020 at 5:06 PM Ilan Ginzburg  wrote:
>
>> Hello Solr devs,
>>
>> This is my first post here. I work at Salesforce in France, we're
>> adopting SolrCloud and we need it to scale more than it currently
>> does.
>>
>> I've looked at Overseer and documented my understanding. I'm sharing
>> the result, it might help others and is a way to get feedback (I might
>> have misunderstood some things) and/or collaboration on continuing
>> documenting the implementation. Basically I started writing the doc I
>> wanted to find.
>>
>> In the process, I believe I've identified what may be a few bugs
>> (there's a section listing them at the beginning). I've found these by
>> reading code (not running code), so take with a grain of salt.
>> I plan to file Jiras for those bugs that do seem real and are
>> important enough, and then also start working on some to help
>> fix/improve.
>>
>>
>> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/
>>
>> This is WIP. Please do not hesitate to provide feedback/leave comments.
>>
>> Thanks,
>> Ilan
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Overseer documentation

2020-04-23 Thread David Smiley 
Thanks Ilan!

I especially love the lead section "Overseer: from queues to state" with
the diagram.  Indeed, this is the documentation I (we?) wished already
existed.  I'd like to try to ensure this part of the document is more
tightly associated with our project for others to see.

This is "developer documentation".  Cassandra:  I see you created
solr/dev-docs/ and I suppose this would best belong there?  Mark Miller had
tried Confluence.  Pros/cons there.  I want to ensure readers of the code
in Overseer (and maybe other key class or two) notice this dev
documentation.  Should I add a http link to the GitHub location of the dev
doc markdown, or do you recommend something else?

For the rest of the doc, there are problem call-outs (picture of a triangle
hazard with an exclamation point) -- readers can't miss them.   I hope
those of us that know SolrCloud internals best can look at those points
closer and maybe file JIRA issues.  That isn't me, honestly.

I've been thinking that some of these problems might best be fixed by
larger architectural changes rather than incrementally fixing a design with
substantial weaknesses (and tech-debt complexities).  Noble/Ishan's
SOLR-13951  issue "Avoid
replica state updates to state.json" will help a lot but there will be much
more to be done to address Solr's over-reliance on the Overseer to
accomplish collection/cluster management. This is something I want to
contemplate more.  I hope Curator recipes may be a source of inspiration,
which I plan to review this weekend.

(note: I work with Ilan)
~ David

On Tue, Apr 21, 2020 at 5:06 PM Ilan Ginzburg  wrote:

> Hello Solr devs,
>
> This is my first post here. I work at Salesforce in France, we're
> adopting SolrCloud and we need it to scale more than it currently
> does.
>
> I've looked at Overseer and documented my understanding. I'm sharing
> the result, it might help others and is a way to get feedback (I might
> have misunderstood some things) and/or collaboration on continuing
> documenting the implementation. Basically I started writing the doc I
> wanted to find.
>
> In the process, I believe I've identified what may be a few bugs
> (there's a section listing them at the beginning). I've found these by
> reading code (not running code), so take with a grain of salt.
> I plan to file Jiras for those bugs that do seem real and are
> important enough, and then also start working on some to help
> fix/improve.
>
>
> https://docs.google.com/document/d/1KTHq3noZBVUQ7QNuBGEhujZ_duwTVpAsvN3Nz5anQUY/
>
> This is WIP. Please do not hesitate to provide feedback/leave comments.
>
> Thanks,
> Ilan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>