Re: Solr Alpha (EA) release of Reference Branch
On Tue, Oct 6, 2020 at 10:45 PM Anshum Gupta wrote: > > I haven’t looked at the current ref branch recently, but the folks who > have looked at it, if you think that this code can be merged into master > even as big chunks, that’d be the most confidence building way forward. > > +1 for considering this approach. merge up master into the branch, and make a big-ass PR to merge back. let people help review (maybe improve) the change as a whole. It's just a big PR, some huge ones like this have been done in lucene before too, unofficially called "unfuck" branches (sorry if you are offended at my terminology). Sometimes you just fix, refactor, cleanup, and keep iterating and see where it can lead. sometimes you revert a bunch of commits because you followed the wrong rabbit-hole, etc. Sometimes it may seem inconvenient, but I think we can all agree It's important to have folks that want to not just take the small fix, but see where it can go and make the whole thing better. Remember Mike's flexible indexing branch? So why not try this way, look at actual code changes and try to get into the master branch? Of course Uwe is willing to point build resources at it either way, but if you want to maximize testing, start with the devs and everyone's jenkins first before throwing at users. Master branch will get you more testing, for sure.
Re: Solr Alpha (EA) release of Reference Branch
On Tue, Oct 6, 2020 at 7:45 PM Anshum Gupta wrote: > Thanks for initiating this discussion, Ishan. > > For the sake of making sure that we are all on the same page, let me > summarize my understanding and take on this thread. > > The current situation > Mark has a reference branch, which the folks who have looked at the > branch, feel that it’s a much better, improved, reliable, and sustainable > version of the current master i.e. take the same baseline and make it > better. We would like to get those changes to the project, but aren’t sure > about how to do so. Releasing the branch when it’s ready to go, as an alpha > release will allow users to test it. > > 1. Is releasing the branch officially going to help us achieve the goal of > having a well tested branch ? > 2. Assuming #1 is true, do we as a community want to release the branch > officially and assume responsibility? > I think so! We should all try to help out to the best we can. 3. What is our path forward after the release I.e. do we merge the branch > into master or swap out current master. > > What do we plan to do (options). > I feel there is a consensus on everyone wanting the best for the project > and wanting Marks’ changes released. > > #1 - There are differing opinions, and I personally think we can have our > test harnesses test the new branch, but I think most companies running Solr > at scale would have concerns with taking up an alpha release and deploying > it in production. The various tests that a bunch of folks are working on is > our best bet at testing out the branch, in which case I’m not sure if we > want an official release. > > #2 - I feel that having an official release and having artifacts show up > in maven central will confuse people. The 4.0 alpha release was very > different in the sense that it was the same branch, the code wasn’t > replacing anything existing but introducing a completely new feature i.e. > SolrCloud. > > #3 - I’m still unclear on how these changes will be released in terms of > the community consensus. I’ve tried to merge parts of Marks’ effort from > another time into master, but it’s very difficult, almost impossible to > isolate and extract commits on the basis of coverage/features/etc. This is > a lot of really great effort and after having spoken with Mark multiple > times, I really feel we should figure out a way to absorb this but I do > have concerns around replacing the master branch completely. > > While I do like the idea that Tomás proposed, I also feel that maintaining > and managing cherry-picking across 9x, master, and ref branch will only > make it difficult for people to work though the duration of 9x. > > I haven’t looked at the current ref branch recently, but the folks who > have looked at it, if you think that this code can be merged into master > even as big chunks, that’d be the most confidence building way forward. > > > > > On Tue, Oct 6, 2020 at 11:37 AM Ilan Ginzburg wrote: > >> Copying below Mark's posts from ASF Slack #solr-next-big-thing channel. >> >> The Solr Reference Branch. >> Document 1, a quick intro. >> You can think of the Solr Reference Branch as a remaster of Solr. It >> is not an attempt to redesign Solr or make it more fancy. The goal of >> the Solr Reference Branch is to be a better incarnation of the current >> Apache Solr, which will provide a base for future development and >> design. >> There are a variety of problems with Solr today that make it difficult >> to adopt and run. This is me being as honest and objective as I can >> be, though no doubt, many will see it as an exaggeration or negative >> focus. I just see it as the way it is and has been, it's just taken me >> a real long time to actually get all the way under the rug to find the >> really hardened nasty cockroaches burrowed in there. >> 1. Resource usage and management is wasteful, inefficient, buggy, and >> haphazard. >> 2. SolrCloud is not long term reliable. Exceptional cases will >> frequently flummox the system, and exceptional cases are supposed to >> be our wheelhouse and primary focus. Leaders will be lost and not >> recover, the Overseer will go away, GC storms will hit, tight loops in >> a bad case will crank up resources, and retries will be abundant and >> overaggressive. >> 3. Our blocking and locking is generally not efficient, especially in key >> paths. >> 4. We get thread safety wrong (too often) in some important spots. >> 5. Distributed updates have to be added locally before they are >> distributed, and then that distribution is generally inefficient, >> prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need >> for packing updates into a single request to achieve any kind of >> performance, losing proper error handling and eating the many rough >> edges of the ConcurrentUpdateSolrClient. >> 6. Our Zookeeper foundation code is often inefficient, buggy, >> unreliable, and improperly used (we don’t always use async or multi >> where we should, we
Re: Solr Alpha (EA) release of Reference Branch
Thanks for initiating this discussion, Ishan. For the sake of making sure that we are all on the same page, let me summarize my understanding and take on this thread. The current situation Mark has a reference branch, which the folks who have looked at the branch, feel that it’s a much better, improved, reliable, and sustainable version of the current master i.e. take the same baseline and make it better. We would like to get those changes to the project, but aren’t sure about how to do so. Releasing the branch when it’s ready to go, as an alpha release will allow users to test it. 1. Is releasing the branch officially going to help us achieve the goal of having a well tested branch ? 2. Assuming #1 is true, do we as a community want to release the branch officially and assume responsibility? 3. What is our path forward after the release I.e. do we merge the branch into master or swap out current master. What do we plan to do (options). I feel there is a consensus on everyone wanting the best for the project and wanting Marks’ changes released. #1 - There are differing opinions, and I personally think we can have our test harnesses test the new branch, but I think most companies running Solr at scale would have concerns with taking up an alpha release and deploying it in production. The various tests that a bunch of folks are working on is our best bet at testing out the branch, in which case I’m not sure if we want an official release. #2 - I feel that having an official release and having artifacts show up in maven central will confuse people. The 4.0 alpha release was very different in the sense that it was the same branch, the code wasn’t replacing anything existing but introducing a completely new feature i.e. SolrCloud. #3 - I’m still unclear on how these changes will be released in terms of the community consensus. I’ve tried to merge parts of Marks’ effort from another time into master, but it’s very difficult, almost impossible to isolate and extract commits on the basis of coverage/features/etc. This is a lot of really great effort and after having spoken with Mark multiple times, I really feel we should figure out a way to absorb this but I do have concerns around replacing the master branch completely. While I do like the idea that Tomás proposed, I also feel that maintaining and managing cherry-picking across 9x, master, and ref branch will only make it difficult for people to work though the duration of 9x. I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. On Tue, Oct 6, 2020 at 11:37 AM Ilan Ginzburg wrote: > Copying below Mark's posts from ASF Slack #solr-next-big-thing channel. > > The Solr Reference Branch. > Document 1, a quick intro. > You can think of the Solr Reference Branch as a remaster of Solr. It > is not an attempt to redesign Solr or make it more fancy. The goal of > the Solr Reference Branch is to be a better incarnation of the current > Apache Solr, which will provide a base for future development and > design. > There are a variety of problems with Solr today that make it difficult > to adopt and run. This is me being as honest and objective as I can > be, though no doubt, many will see it as an exaggeration or negative > focus. I just see it as the way it is and has been, it's just taken me > a real long time to actually get all the way under the rug to find the > really hardened nasty cockroaches burrowed in there. > 1. Resource usage and management is wasteful, inefficient, buggy, and > haphazard. > 2. SolrCloud is not long term reliable. Exceptional cases will > frequently flummox the system, and exceptional cases are supposed to > be our wheelhouse and primary focus. Leaders will be lost and not > recover, the Overseer will go away, GC storms will hit, tight loops in > a bad case will crank up resources, and retries will be abundant and > overaggressive. > 3. Our blocking and locking is generally not efficient, especially in key > paths. > 4. We get thread safety wrong (too often) in some important spots. > 5. Distributed updates have to be added locally before they are > distributed, and then that distribution is generally inefficient, > prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need > for packing updates into a single request to achieve any kind of > performance, losing proper error handling and eating the many rough > edges of the ConcurrentUpdateSolrClient. > 6. Our Zookeeper foundation code is often inefficient, buggy, > unreliable, and improperly used (we don’t always use async or multi > where we should, we force updates from zk instead of being notified, > we don’t handle session expiration as well as we should, our > algorithms are slow and buggy, we make a multitude more calls than we > should (especially on cluster startup), etc, etc) > 7. We have circular
Re: restlet dependencies
No strong opinion from me. I think the back-compat concern is minor. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Tue, Oct 6, 2020 at 5:42 PM Noble Paul wrote: > I think we should call that out in the changes.txt and make the changes > right away. > > On Wed, Oct 7, 2020, 8:20 AM Timothy Potter wrote: > >> Just want to close the loop on this restlet issue. I've removed the >> restlet dependency in master (e879a52291ef7dcd0514e7419d811b6ff800bcce) but >> have not backported that to 8.x yet. >> >> I'm hesitant to backport because I had to change public function >> signatures on ManagedResource, e.g. >> https://github.com/apache/lucene-solr/commit/e879a52291ef7dcd0514e7419d811b6ff800bcce#diff-faaf5cd2bfe038f81ca4c0cb1986b42dR355 >> ... >> >> So technically, this could break upgrades for environments with >> extensions to those classes; practically speaking, I think the risk of that >> is low, but not sure it is worth it? From what I understand, the restlet >> maven issue was mostly affecting master / gradle builds and the ant/ivy >> builds in 8.x weren't affected as much? >> >> It's an easy back-port from master to 8.x if that's what we want to do, >> but wanted to raise awareness that it will break public function signatures >> on classes that may have extensions in the wild ;-) >> >> ~ Tim >> >> On Thu, Oct 1, 2020 at 8:36 AM Timothy Potter >> wrote: >> >>> Awesome guys, thanks for the pointers ... am cooking up a PR (for >>> master) for this today >>> >>> On Thu, Oct 1, 2020 at 2:22 AM Noble Paul wrote: >>> The annotation ( https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/api/EndPoint.java ) supports wild cards and templated paths On Thu, Oct 1, 2020 at 5:28 PM Ishan Chattopadhyaya wrote: > > > But when I suggested porting the code that uses restlet to JAX-RS / Jersey, Ishan said > > that wasn't necessary and is already supported with some Annotations ... I have no idea > > what that means and need more info about what is already in place. > > I was mainly referring to the @Endpoint annotations. It is available for V2 APIs today (and I think it should be fine to use for anything we build now onwards, including managed resources V2). > It is possible to make it work with V1, but that will require some work. > > On Thu, Oct 1, 2020 at 12:50 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: >> >> @Tim >> >> Please check ClusterAPI or ZookeeperReadAPI etc. >> Recently used it in Yasa: https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java >> >> On Thu, Oct 1, 2020 at 10:46 AM Noble Paul wrote: >>> >>> @Tim Potter >>> >>> I tried several times to get rid of the restlet dependency & keep the >>> functionality as is. I failed miserably. I'm not saying this to >>> discourage anyone who wants to give a try. Just letting you know that >>> it is not as easy as it may sound >>> >>> On Thu, Oct 1, 2020 at 2:42 AM Houston Putman < houstonput...@gmail.com> wrote: >>> > >>> > +1 to Tomas' proposal. Created SOLR-14907 to track the effort. >>> > >>> > - Houston >>> > >>> > On Wed, Sep 30, 2020 at 12:26 PM Tomás Fernández Löbbe < tomasflo...@gmail.com> wrote: >>> >> >>> >> > Let's support the single file upload feature >>> >> +1, but let this behave exactly as a zip file with a single file in it (regarding trusted/untrusted). We just need to change the configset handler to be able to handle non-zip files, and have a way to "locate" that file inside the configset (in case it needs to go somewhere other than the root). >>> >> >>> >> On Wed, Sep 30, 2020 at 8:45 AM Eric Pugh < ep...@opensourceconnections.com> wrote: >>> >>> >>> >>> I think that me in “violent agreement” with you. Let’s understand the Annotations approach that we have, or pick something that is commonly used like JAX-RS / Jersey. >>> >>> >>> >>> >>> >>> >>> >>> On Sep 30, 2020, at 11:41 AM, Timothy Potter < thelabd...@gmail.com> wrote: >>> >>> >>> >>> I'm sorry, I don't understand what you mean by "make it a single pattern (the annotations?)" Eric? >>> >>> >>> >>> To me, the pattern is well established in the Java world: JAX-RS (with Jersey as the underlying impl. which has nice integration with Jetty). But when I suggested porting the code that uses restlet to JAX-RS / Jersey, Ishan said that wasn't necessary and is already supported with some Annotations ... I have no idea what that means and need more info about what is already in place. Short of that, replacing restlet with JAX-RS / Jersey looks like
Re: restlet dependencies
I think we should call that out in the changes.txt and make the changes right away. On Wed, Oct 7, 2020, 8:20 AM Timothy Potter wrote: > Just want to close the loop on this restlet issue. I've removed the > restlet dependency in master (e879a52291ef7dcd0514e7419d811b6ff800bcce) but > have not backported that to 8.x yet. > > I'm hesitant to backport because I had to change public function > signatures on ManagedResource, e.g. > https://github.com/apache/lucene-solr/commit/e879a52291ef7dcd0514e7419d811b6ff800bcce#diff-faaf5cd2bfe038f81ca4c0cb1986b42dR355 > ... > > So technically, this could break upgrades for environments with extensions > to those classes; practically speaking, I think the risk of that is low, > but not sure it is worth it? From what I understand, the restlet maven > issue was mostly affecting master / gradle builds and the ant/ivy builds in > 8.x weren't affected as much? > > It's an easy back-port from master to 8.x if that's what we want to do, > but wanted to raise awareness that it will break public function signatures > on classes that may have extensions in the wild ;-) > > ~ Tim > > On Thu, Oct 1, 2020 at 8:36 AM Timothy Potter > wrote: > >> Awesome guys, thanks for the pointers ... am cooking up a PR (for master) >> for this today >> >> On Thu, Oct 1, 2020 at 2:22 AM Noble Paul wrote: >> >>> The annotation ( >>> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/api/EndPoint.java >>> ) >>> supports wild cards and templated paths >>> >>> On Thu, Oct 1, 2020 at 5:28 PM Ishan Chattopadhyaya >>> wrote: >>> > >>> > > But when I suggested porting the code that uses restlet to JAX-RS / >>> Jersey, Ishan said >>> > > that wasn't necessary and is already supported with some Annotations >>> ... I have no idea >>> > > what that means and need more info about what is already in place. >>> > >>> > I was mainly referring to the @Endpoint annotations. It is available >>> for V2 APIs today (and I think it should be fine to use for anything we >>> build now onwards, including managed resources V2). >>> > It is possible to make it work with V1, but that will require some >>> work. >>> > >>> > On Thu, Oct 1, 2020 at 12:50 PM Ishan Chattopadhyaya < >>> ichattopadhy...@gmail.com> wrote: >>> >> >>> >> @Tim >>> >> >>> >> Please check ClusterAPI or ZookeeperReadAPI etc. >>> >> Recently used it in Yasa: >>> https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java >>> >> >>> >> On Thu, Oct 1, 2020 at 10:46 AM Noble Paul >>> wrote: >>> >>> >>> >>> @Tim Potter >>> >>> >>> >>> I tried several times to get rid of the restlet dependency & keep the >>> >>> functionality as is. I failed miserably. I'm not saying this to >>> >>> discourage anyone who wants to give a try. Just letting you know that >>> >>> it is not as easy as it may sound >>> >>> >>> >>> On Thu, Oct 1, 2020 at 2:42 AM Houston Putman < >>> houstonput...@gmail.com> wrote: >>> >>> > >>> >>> > +1 to Tomas' proposal. Created SOLR-14907 to track the effort. >>> >>> > >>> >>> > - Houston >>> >>> > >>> >>> > On Wed, Sep 30, 2020 at 12:26 PM Tomás Fernández Löbbe < >>> tomasflo...@gmail.com> wrote: >>> >>> >> >>> >>> >> > Let's support the single file upload feature >>> >>> >> +1, but let this behave exactly as a zip file with a single file >>> in it (regarding trusted/untrusted). We just need to change the configset >>> handler to be able to handle non-zip files, and have a way to "locate" that >>> file inside the configset (in case it needs to go somewhere other than the >>> root). >>> >>> >> >>> >>> >> On Wed, Sep 30, 2020 at 8:45 AM Eric Pugh < >>> ep...@opensourceconnections.com> wrote: >>> >>> >>> >>> >>> >>> I think that me in “violent agreement” with you. Let’s >>> understand the Annotations approach that we have, or pick something that is >>> commonly used like JAX-RS / Jersey. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Sep 30, 2020, at 11:41 AM, Timothy Potter < >>> thelabd...@gmail.com> wrote: >>> >>> >>> >>> >>> >>> I'm sorry, I don't understand what you mean by "make it a single >>> pattern (the annotations?)" Eric? >>> >>> >>> >>> >>> >>> To me, the pattern is well established in the Java world: JAX-RS >>> (with Jersey as the underlying impl. which has nice integration with >>> Jetty). But when I suggested porting the code that uses restlet to JAX-RS / >>> Jersey, Ishan said that wasn't necessary and is already supported with some >>> Annotations ... I have no idea what that means and need more info about >>> what is already in place. Short of that, replacing restlet with JAX-RS / >>> Jersey looks like a trivial amount of work to me (and I'm happy to take it >>> on). >>> >>> >>> >>> >>> >>> Tim >>> >>> >>> >>> >>> >>> On Wed, Sep 30, 2020 at 9:36 AM Eric Pugh < >>> ep...@opensourceconnections.com> wrote: >>> >>> >>> >>> The use case of “I want to update something via a API” is I >>> think pretty common, and it would be nice
Re: restlet dependencies
Just want to close the loop on this restlet issue. I've removed the restlet dependency in master (e879a52291ef7dcd0514e7419d811b6ff800bcce) but have not backported that to 8.x yet. I'm hesitant to backport because I had to change public function signatures on ManagedResource, e.g. https://github.com/apache/lucene-solr/commit/e879a52291ef7dcd0514e7419d811b6ff800bcce#diff-faaf5cd2bfe038f81ca4c0cb1986b42dR355 ... So technically, this could break upgrades for environments with extensions to those classes; practically speaking, I think the risk of that is low, but not sure it is worth it? From what I understand, the restlet maven issue was mostly affecting master / gradle builds and the ant/ivy builds in 8.x weren't affected as much? It's an easy back-port from master to 8.x if that's what we want to do, but wanted to raise awareness that it will break public function signatures on classes that may have extensions in the wild ;-) ~ Tim On Thu, Oct 1, 2020 at 8:36 AM Timothy Potter wrote: > Awesome guys, thanks for the pointers ... am cooking up a PR (for master) > for this today > > On Thu, Oct 1, 2020 at 2:22 AM Noble Paul wrote: > >> The annotation ( >> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/api/EndPoint.java >> ) >> supports wild cards and templated paths >> >> On Thu, Oct 1, 2020 at 5:28 PM Ishan Chattopadhyaya >> wrote: >> > >> > > But when I suggested porting the code that uses restlet to JAX-RS / >> Jersey, Ishan said >> > > that wasn't necessary and is already supported with some Annotations >> ... I have no idea >> > > what that means and need more info about what is already in place. >> > >> > I was mainly referring to the @Endpoint annotations. It is available >> for V2 APIs today (and I think it should be fine to use for anything we >> build now onwards, including managed resources V2). >> > It is possible to make it work with V1, but that will require some work. >> > >> > On Thu, Oct 1, 2020 at 12:50 PM Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> wrote: >> >> >> >> @Tim >> >> >> >> Please check ClusterAPI or ZookeeperReadAPI etc. >> >> Recently used it in Yasa: >> https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java >> >> >> >> On Thu, Oct 1, 2020 at 10:46 AM Noble Paul >> wrote: >> >>> >> >>> @Tim Potter >> >>> >> >>> I tried several times to get rid of the restlet dependency & keep the >> >>> functionality as is. I failed miserably. I'm not saying this to >> >>> discourage anyone who wants to give a try. Just letting you know that >> >>> it is not as easy as it may sound >> >>> >> >>> On Thu, Oct 1, 2020 at 2:42 AM Houston Putman < >> houstonput...@gmail.com> wrote: >> >>> > >> >>> > +1 to Tomas' proposal. Created SOLR-14907 to track the effort. >> >>> > >> >>> > - Houston >> >>> > >> >>> > On Wed, Sep 30, 2020 at 12:26 PM Tomás Fernández Löbbe < >> tomasflo...@gmail.com> wrote: >> >>> >> >> >>> >> > Let's support the single file upload feature >> >>> >> +1, but let this behave exactly as a zip file with a single file >> in it (regarding trusted/untrusted). We just need to change the configset >> handler to be able to handle non-zip files, and have a way to "locate" that >> file inside the configset (in case it needs to go somewhere other than the >> root). >> >>> >> >> >>> >> On Wed, Sep 30, 2020 at 8:45 AM Eric Pugh < >> ep...@opensourceconnections.com> wrote: >> >>> >>> >> >>> >>> I think that me in “violent agreement” with you. Let’s >> understand the Annotations approach that we have, or pick something that is >> commonly used like JAX-RS / Jersey. >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> On Sep 30, 2020, at 11:41 AM, Timothy Potter < >> thelabd...@gmail.com> wrote: >> >>> >>> >> >>> >>> I'm sorry, I don't understand what you mean by "make it a single >> pattern (the annotations?)" Eric? >> >>> >>> >> >>> >>> To me, the pattern is well established in the Java world: JAX-RS >> (with Jersey as the underlying impl. which has nice integration with >> Jetty). But when I suggested porting the code that uses restlet to JAX-RS / >> Jersey, Ishan said that wasn't necessary and is already supported with some >> Annotations ... I have no idea what that means and need more info about >> what is already in place. Short of that, replacing restlet with JAX-RS / >> Jersey looks like a trivial amount of work to me (and I'm happy to take it >> on). >> >>> >>> >> >>> >>> Tim >> >>> >>> >> >>> >>> On Wed, Sep 30, 2020 at 9:36 AM Eric Pugh < >> ep...@opensourceconnections.com> wrote: >> >>> >> >>> The use case of “I want to update something via a API” is I >> think pretty common, and it would be nice to make it a single pattern (the >> annotations?) with lots of examples/developer docs for the next person. >> >>> >> >>> >> >>> >> >>> On Sep 30, 2020, at 11:04 AM, Timothy Potter < >> thelabd...@gmail.com> wrote: >> >>> >> >>> I started looking
Re: Solr Alpha (EA) release of Reference Branch
Copying below Mark's posts from ASF Slack #solr-next-big-thing channel. The Solr Reference Branch. Document 1, a quick intro. You can think of the Solr Reference Branch as a remaster of Solr. It is not an attempt to redesign Solr or make it more fancy. The goal of the Solr Reference Branch is to be a better incarnation of the current Apache Solr, which will provide a base for future development and design. There are a variety of problems with Solr today that make it difficult to adopt and run. This is me being as honest and objective as I can be, though no doubt, many will see it as an exaggeration or negative focus. I just see it as the way it is and has been, it's just taken me a real long time to actually get all the way under the rug to find the really hardened nasty cockroaches burrowed in there. 1. Resource usage and management is wasteful, inefficient, buggy, and haphazard. 2. SolrCloud is not long term reliable. Exceptional cases will frequently flummox the system, and exceptional cases are supposed to be our wheelhouse and primary focus. Leaders will be lost and not recover, the Overseer will go away, GC storms will hit, tight loops in a bad case will crank up resources, and retries will be abundant and overaggressive. 3. Our blocking and locking is generally not efficient, especially in key paths. 4. We get thread safety wrong (too often) in some important spots. 5. Distributed updates have to be added locally before they are distributed, and then that distribution is generally inefficient, prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need for packing updates into a single request to achieve any kind of performance, losing proper error handling and eating the many rough edges of the ConcurrentUpdateSolrClient. 6. Our Zookeeper foundation code is often inefficient, buggy, unreliable, and improperly used (we don’t always use async or multi where we should, we force updates from zk instead of being notified, we don’t handle session expiration as well as we should, our algorithms are slow and buggy, we make a multitude more calls than we should (especially on cluster startup), etc, etc) 7. We have circular dependencies between major classes that can start threads in their constructors that start interacting with the other classes before construction is complete. 8. Our XML handling is abysmally outdated and slow for multiple reasons. Our heavy Xpath usage is incredibly wasteful and expensive. 9. Our thread management is not understandable, not properly tunable, not efficient, sometimes buggy, not always consistent, and difficult to understand fundamentally. 10. Our Jetty configuration is lacking in a variety of ways, especially around shutdown and http2. 11. The dynamic schema feature can be very expensive and not fully thread safe. 12. The Overseer is extremely inefficient, can be extremely slow to stop, had a buggy leader election algorithm, doesn’t handle session expiration as well as it should, can keep trying to come back from the dead, and the list goes on. 13. Our connection resuse is often very poor or non existent, when it’s improved, it always reverts back to bad or worse. 14. HTTP1.1 is not great for our type of application in a variety of ways that HTTP2 solves – but we still use a lot of HTTP1.1 and HTTP2 is not configured well and the client needs some work. 15. Lifecycle of important objects is often off, most things can and will leak (SolrCores, SolrIndexSearchers, Directory’s, Solr clients), some things will close objects more than once or that don’t belong to them, or close things in a bad order. 16. There is often sleeps and/or pulling that is a magnitude slower than proper event driven waits. 17. Our tests are actually pretty unstable and making them stable is way, way more difficult than most people realize. I’m quite sure I’ve spent much, much more time on this than anyone out there, and I can tell you, the tests are not stable in a 1,000 shifting ways that have and will continue to cause lots of damage. 18. We don’t have good async update/search support for scaling and better resource usage. 19. We often duplicate resources or create new pools instead of sharing. 20. We don’t do tons of parallelizable stuff in parallel, when we do it’s inconsistent. 21. Our Collections API can often not wait correctly for the proper state for what it did to be ready before returning. Even if it gets it right, a cloud client that made the request won’t necessarily have the updated state local when the request returns. Things often still work, but with a variety of interesting and slow results possible. 22. We don’t often holistically look at what we have built and how it fits together and so often there are silly things, bad fits, one off bad patterns, lazy attempts at something, etc. 24. Close and shutdown are inefficient and slow across a huge swatch of our object tree. These issues tend to be growy and breed less concern over time. 25. There are a variety of ways and
Re: Solr Alpha (EA) release of Reference Branch
> Let's say we cut 9x and now there is a new master taken from the reference branch. I never said “make a new master”, I said merge changes in ref branch into master. If things are broken into pieces like Ishan is suggesting, those changes can be merged into 9.x too. I only suggested this because you felt unsure about merging to master now and I guess this is due to fear of introducing bugs so close to a potential 9.0 release, is that not right? > We will never be able to reconcile these 2 branches Sorry, but how is that different if we do an alpha release from the branch now? What would be the process after that? Let's say people don't find issues and we want to merge those changes, what’s the plan then? > Choice 1: I’m fine with choice 1 if that’s what you want, as long as it’s not an official release for the reasons stated above. > I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release What do you mean? I thought this is what you were suggesting, make an official release from the reference_impl branch? I think Ilan’s last email is on spot, and I agree 100% with what he can express much better than I can :) > Mark's descriptions in Slack go in the right way but are still too high level Can someone share those here? or in Jira? On Tue, Oct 6, 2020 at 5:09 AM Noble Paul wrote: > > I think the danger is high to treat this branch as a black box (or an > "all or nothing"). > > True Ilan. Ideally, I would like a few of us to study the code & > start pulling in changes we are confident of (even to 8x branch, why > not). We cannot burden a single developer to do everything. > > This cannot be a task just for one or 2 devs. We all will have to work > together to decompose the changes and digest them into master. I can > do my bit. > > But, I'm sure we may hit a point where certain changes cannot be > isolated and absorbed. We will have to collectively make a call, how > to absorb them > > On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya > wrote: > > > > > > I'm willing to help and I believe others will too if the amount of work > for contributing is reasonable (i.e. not a three months effort). > > > > I looked into the possibility of doing so. To me, it seemed to be that > it is very hard to do so: possibly 1 year project for me. Problem is that > it is hard to pull out a particular class of improvements (say thread > management improvement) and have all tests pass with it (because tests have > gotten extensive improvements of their own) and also observe the effect of > the improvement. IIUC, every improvement to Solr seemed to require many > iterations to get the tests happy. I remember Mark telling me that it may > not even be possible for him to do something like that (i.e. bring all > changes into master as tiny pieces). > > > > What I volunteered to do, however, is to decompose roughly all the > general improvements into smaller, manageable commits. However, making sure > all tests pass at every commit point is beyond my capability. > > > > On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, wrote: > >> > >> Another option to integrate this work into the main code line would be > to understand what changes have been made and where (Mark's descriptions in > Slack go in the right way but are still too high level), and then port or > even redo them in main, one by one. > >> > >> I think the danger is high to treat this branch as a black box (or an > "all or nothing"). Using the merging itself to change our understanding and > increase our knowledge of what was done can greatly reduce the risk. > >> > >> We do develop new features in Solr 9 without beta releasing them, so if > we port Mark's improvements by small chunks (and maybe in the process > decide that some should not be ported or not now) I don't see why this > can't integrate to become like other improvements done to the code. If > specific changes do require a beta release, do that release from master and > pick the right moment. > >> > >> I'm willing to help and I believe others will too if the amount of work > for contributing is reasonable (i.e. not a three months effort). This > requires documenting the changes done in that branch, pointing to where > these changes happened and then picking them up one by one and porting them > more or less independently of each other. We might only port a subset of > changes by the time 9.0 is released, that's fine we can continue in > following releases. > >> > >> My 2 cents... > >> Ilan > >> > >> Le mar. 6 oct. 2020 à 09:56, Noble Paul a écrit > : > >>> > >>> Yes, A docker image will definitely help. I wasn't trying to downplay > that > >>> > >>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya > >>> wrote: > >>> > > >>> > > >>> > > Docker is not a big requirement for large scale installations. > Most of them already have their own install scripts. Availability of docker > is not important for them. If a user is only
Re: 8.6.3 Release
That is correct. 8.x docker builds have not been affected in any way. On Tue, Oct 6, 2020 at 11:30 AM Cassandra Targett wrote: > I wanted to ask now that the 8.6.3 vote is underway - for the docker-solr > image, are the update instructions in the docker-solr repo still the same > for 8.x even though the build process has been moved to the main project > for 9.0? Meaning, to release the 8.6.3 image there’s no change from before, > right? > > I’m asking specifically about these instructions: > > https://github.com/docker-solr/docker-solr/blob/master/update.md > On Oct 1, 2020, 9:28 AM -0500, Jason Gerlowski , > wrote: > > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can > someone please sanity check the summaries there when they get a > chance? Would appreciate the review. > > 8.6.3 is a bit interesting in that Lucene has no changes in this > bugfix release. As a result I had to omit the standard phrase in the > Solr release notes about there being additional changes at the Lucene > level, and change some of the wording in the Lucene announcement to > indicate the lack of changes. So that's something to pay particular > attention to, if someone can check my wording there. > > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863 > [2] > https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863 > > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski > wrote: > > > The only one that was previously mentioned as a blocker was > SOLR-14835, but from the comments on the ticket it looks like it ended > up being purely a cosmetic issue. Andrzej left a comment there > suggesting that we "address" this with documentation for 8.6.3 but > otherwise leave it as-is. > > So it looks like we're unblocked on starting the release process. > Will begin the preliminary steps this afternoon. > > On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett > wrote: > > > It looks to me like everything for 8.6.3 is resolved now ( > https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a > Jetty upgrade less compelling to try. > > Are there any other issues not currently marked for 8.6.3 we’re waiting > for before starting the RC? > On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , > wrote: > > That said, if someone can use 8.6.3, what’s stopping them from going to > 8.7 when it’e released? > > > The same things that always stop users from going directly to the > latest-and-greatest: fear of instability from new minor-release > features, reliance on behavior changed across minor versions, breaking > changes on Lucene elements that don't guarantee backcompat (e.g. > SOLR-14254), security issues in later versions (new libraries pulled > in with vulns), etc. There's lots of reasons a given user might want > to stick on 8.6.x rather than 8.7 (in the short/medium term). > > I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above > the worst of the Jetty issue should be mitigated by work on our end - > but I think there's a lot of reasons users might not upgrade as far as > we'd expect/like. > > > On Mon, Sep 28, 2020 at 2:05 PM Erick Erickson > wrote: > > > For me, there’s a sharp distinction between changing a dependency in a > point release just because there’s a new version, and changing the > dependency because there’s a bug in it. That said, if someone can use > 8.6.3, what’s stopping them from going to 8.7 when it’e released? Would it > make more sense to do the upgrades for 8.7 and get that out the door rather > than backport? > > FWIW, > Erick > > On Sep 28, 2020, at 1:45 PM, Jason Gerlowski > wrote: > > Hey all, > > I wanted to add 2 more blocker tickets to the list: SOLR-14897 and > SOLR-14898. These tickets (while bad bugs in their own right) are > especially necessary because they work around a Jetty buffer-reuse bug > (see SOLR-14896) that causes sporadic request failures once triggered. > > So that brings the list of 8.6.3 blockers up to: SOLR-14850, > SOLR-14835, SOLR-14897, and SOLR-14898. (Thanks David for the quick > work on SOLR-14768!) > > Additionally, should we also consider a Jetty upgrade for 8.6.3 in > light of the issue mentioned above? I know it's atypical for bug-fix > releases to change deps, but here the bug is serious and tied directly > to the dep. SOLR-14897 and SOLR-14898 help greatly here, but the > Jetty bug is likely still a problem for users making requests that > match a specific (albeit rare) profile. Anyone have thoughts? > > Best, > > Jason > > On Fri, Sep 25, 2020 at 12:28 AM Houston Putman > wrote: > > > If I recall correctly, thats a step in the release wizard. > > After checking, I think this fits the bill: > > https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/releaseWizard.yaml#L1435 > > - Houston > > On Fri, Sep 25, 2020 at 12:06 AM David Smiley wrote: > > > When moving changes from 8.7 to 8.6.3, must we (the mover of an
Re: 8.6.3 Release
I wanted to ask now that the 8.6.3 vote is underway - for the docker-solr image, are the update instructions in the docker-solr repo still the same for 8.x even though the build process has been moved to the main project for 9.0? Meaning, to release the 8.6.3 image there’s no change from before, right? I’m asking specifically about these instructions: https://github.com/docker-solr/docker-solr/blob/master/update.md On Oct 1, 2020, 9:28 AM -0500, Jason Gerlowski , wrote: > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can > someone please sanity check the summaries there when they get a > chance? Would appreciate the review. > > 8.6.3 is a bit interesting in that Lucene has no changes in this > bugfix release. As a result I had to omit the standard phrase in the > Solr release notes about there being additional changes at the Lucene > level, and change some of the wording in the Lucene announcement to > indicate the lack of changes. So that's something to pay particular > attention to, if someone can check my wording there. > > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863 > [2] https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863 > > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski > wrote: > > > > The only one that was previously mentioned as a blocker was > > SOLR-14835, but from the comments on the ticket it looks like it ended > > up being purely a cosmetic issue. Andrzej left a comment there > > suggesting that we "address" this with documentation for 8.6.3 but > > otherwise leave it as-is. > > > > So it looks like we're unblocked on starting the release process. > > Will begin the preliminary steps this afternoon. > > > > On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett > > wrote: > > > > > > It looks to me like everything for 8.6.3 is resolved now > > > (https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it > > > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a > > > Jetty upgrade less compelling to try. > > > > > > Are there any other issues not currently marked for 8.6.3 we’re waiting > > > for before starting the RC? > > > On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , > > > wrote: > > > > > > That said, if someone can use 8.6.3, what’s stopping them from going to > > > 8.7 when it’e released? > > > > > > > > > The same things that always stop users from going directly to the > > > latest-and-greatest: fear of instability from new minor-release > > > features, reliance on behavior changed across minor versions, breaking > > > changes on Lucene elements that don't guarantee backcompat (e.g. > > > SOLR-14254), security issues in later versions (new libraries pulled > > > in with vulns), etc. There's lots of reasons a given user might want > > > to stick on 8.6.x rather than 8.7 (in the short/medium term). > > > > > > I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above > > > the worst of the Jetty issue should be mitigated by work on our end - > > > but I think there's a lot of reasons users might not upgrade as far as > > > we'd expect/like. > > > > > > > > > On Mon, Sep 28, 2020 at 2:05 PM Erick Erickson > > > wrote: > > > > > > > > > For me, there’s a sharp distinction between changing a dependency in a > > > point release just because there’s a new version, and changing the > > > dependency because there’s a bug in it. That said, if someone can use > > > 8.6.3, what’s stopping them from going to 8.7 when it’e released? Would > > > it make more sense to do the upgrades for 8.7 and get that out the door > > > rather than backport? > > > > > > FWIW, > > > Erick > > > > > > On Sep 28, 2020, at 1:45 PM, Jason Gerlowski > > > wrote: > > > > > > Hey all, > > > > > > I wanted to add 2 more blocker tickets to the list: SOLR-14897 and > > > SOLR-14898. These tickets (while bad bugs in their own right) are > > > especially necessary because they work around a Jetty buffer-reuse bug > > > (see SOLR-14896) that causes sporadic request failures once triggered. > > > > > > So that brings the list of 8.6.3 blockers up to: SOLR-14850, > > > SOLR-14835, SOLR-14897, and SOLR-14898. (Thanks David for the quick > > > work on SOLR-14768!) > > > > > > Additionally, should we also consider a Jetty upgrade for 8.6.3 in > > > light of the issue mentioned above? I know it's atypical for bug-fix > > > releases to change deps, but here the bug is serious and tied directly > > > to the dep. SOLR-14897 and SOLR-14898 help greatly here, but the > > > Jetty bug is likely still a problem for users making requests that > > > match a specific (albeit rare) profile. Anyone have thoughts? > > > > > > Best, > > > > > > Jason > > > > > > On Fri, Sep 25, 2020 at 12:28 AM Houston Putman > > > wrote: > > > > > > > > > If I recall correctly, thats a step in the release wizard. > > > > > > After checking, I think this fits the bill: > > >
Re: CWiki and IDE instructions
Gone. > On Oct 4, 2020, at 3:39 PM, Erick Erickson wrote: > > The "How to contribute" page here: > > https://cwiki.apache.org/confluence/display/solr/HowToContribute > > contains links to two obsolete pages for configuring Eclipse and IntelliJ: > > https://cwiki.apache.org/confluence/display/solr/HowToConfigureEclipse > > https://cwiki.apache.org/confluence/display/lucene/HowtoConfigureIntelliJ > > As well as a link to configuring NetBeans. > > I don't have the bandwidth to rewrite these pages. Do we really even want > them? Or should we just replace the links on the "How To Contribute" page > with a short message giving the "ant idea", "ant eclipse", "gradlew idea or > "gradlew eclipse" commands and a message about "contact the Solr user's list > or dev list for assistance with setting up your IDE if you have problems". > > Then delete the linked pages? - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr Alpha (EA) release of Reference Branch
> I think the danger is high to treat this branch as a black box (or an "all or > nothing"). True Ilan. Ideally, I would like a few of us to study the code & start pulling in changes we are confident of (even to 8x branch, why not). We cannot burden a single developer to do everything. This cannot be a task just for one or 2 devs. We all will have to work together to decompose the changes and digest them into master. I can do my bit. But, I'm sure we may hit a point where certain changes cannot be isolated and absorbed. We will have to collectively make a call, how to absorb them On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya wrote: > > > I'm willing to help and I believe others will too if the amount of work for > contributing is reasonable (i.e. not a three months effort). > > I looked into the possibility of doing so. To me, it seemed to be that it is > very hard to do so: possibly 1 year project for me. Problem is that it is > hard to pull out a particular class of improvements (say thread management > improvement) and have all tests pass with it (because tests have gotten > extensive improvements of their own) and also observe the effect of the > improvement. IIUC, every improvement to Solr seemed to require many > iterations to get the tests happy. I remember Mark telling me that it may not > even be possible for him to do something like that (i.e. bring all changes > into master as tiny pieces). > > What I volunteered to do, however, is to decompose roughly all the general > improvements into smaller, manageable commits. However, making sure all tests > pass at every commit point is beyond my capability. > > On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, wrote: >> >> Another option to integrate this work into the main code line would be to >> understand what changes have been made and where (Mark's descriptions in >> Slack go in the right way but are still too high level), and then port or >> even redo them in main, one by one. >> >> I think the danger is high to treat this branch as a black box (or an "all >> or nothing"). Using the merging itself to change our understanding and >> increase our knowledge of what was done can greatly reduce the risk. >> >> We do develop new features in Solr 9 without beta releasing them, so if we >> port Mark's improvements by small chunks (and maybe in the process decide >> that some should not be ported or not now) I don't see why this can't >> integrate to become like other improvements done to the code. If specific >> changes do require a beta release, do that release from master and pick the >> right moment. >> >> I'm willing to help and I believe others will too if the amount of work for >> contributing is reasonable (i.e. not a three months effort). This requires >> documenting the changes done in that branch, pointing to where these changes >> happened and then picking them up one by one and porting them more or less >> independently of each other. We might only port a subset of changes by the >> time 9.0 is released, that's fine we can continue in following releases. >> >> My 2 cents... >> Ilan >> >> Le mar. 6 oct. 2020 à 09:56, Noble Paul a écrit : >>> >>> Yes, A docker image will definitely help. I wasn't trying to downplay that >>> >>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya >>> wrote: >>> > >>> > >>> > > Docker is not a big requirement for large scale installations. Most of >>> > > them already have their own install scripts. Availability of docker is >>> > > not important for them. If a user is only encouraged to install Solr >>> > > because of a docker image , most likely they are not running a large >>> > > enough cluster >>> > >>> > I disagree, Noble. Having a docker image us going to be useful to some >>> > clients, with complex usecases. Great point, David! >>> > >>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, >>> > wrote: >>> >> >>> >> As I said, I'm *personally* not confident in putting such a big >>> >> changeset into master that wasn't vetted in a real user environment >>> >> widely. I have, in the past, done enough bad things to Solr (directly or >>> >> indirectly), and I don't want to repeat the same. Also, I'll be very >>> >> uncomfortable if someone else did so. >>> >> >>> >> Having said this, if someone else wants to port the changes over to >>> >> master *without first getting enough real world testing*, feel free to >>> >> do so, and I can focus my efforts elsewhere. >>> >> >>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, >>> >> wrote: >>> >>> >>> >>> I was thinking (and I haven’t flushed it out completely but will throw >>> >>> the idea) that an alternative approach with this timeline could be to >>> >>> cut 9x branch around November/December? And then you could merge into >>> >>> master, it would have the latest changes from master plus the ref >>> >>> branch changes. From there any nightly build could be use to help >>> >>> test/debug. >>> >>>
Re: Solr Alpha (EA) release of Reference Branch
I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces). What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability. On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, wrote: > Another option to integrate this work into the main code line would be to > understand what changes have been made and where (Mark's descriptions in > Slack go in the right way but are still too high level), and then port or > even redo them in main, one by one. > > I think the danger is high to treat this branch as a black box (or an "all > or nothing"). Using the merging itself to change our understanding and > increase our knowledge of what was done can greatly reduce the risk. > > We do develop new features in Solr 9 without beta releasing them, so if we > port Mark's improvements by small chunks (and maybe in the process decide > that some should not be ported or not now) I don't see why this can't > integrate to become like other improvements done to the code. If specific > changes do require a beta release, do that release from master and pick the > right moment. > > I'm willing to help and I believe others will too if the amount of work > for contributing is reasonable (i.e. not a three months effort). This > requires documenting the changes done in that branch, pointing to where > these changes happened and then picking them up one by one and porting them > more or less independently of each other. We might only port a subset of > changes by the time 9.0 is released, that's fine we can continue in > following releases. > > My 2 cents... > Ilan > > Le mar. 6 oct. 2020 à 09:56, Noble Paul a écrit : > >> Yes, A docker image will definitely help. I wasn't trying to downplay that >> >> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya >> wrote: >> > >> > >> > > Docker is not a big requirement for large scale installations. Most >> of them already have their own install scripts. Availability of docker is >> not important for them. If a user is only encouraged to install Solr >> because of a docker image , most likely they are not running a large enough >> cluster >> > >> > I disagree, Noble. Having a docker image us going to be useful to some >> clients, with complex usecases. Great point, David! >> > >> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, < >> ichattopadhy...@gmail.com> wrote: >> >> >> >> As I said, I'm *personally* not confident in putting such a big >> changeset into master that wasn't vetted in a real user environment widely. >> I have, in the past, done enough bad things to Solr (directly or >> indirectly), and I don't want to repeat the same. Also, I'll be very >> uncomfortable if someone else did so. >> >> >> >> Having said this, if someone else wants to port the changes over to >> master *without first getting enough real world testing*, feel free to do >> so, and I can focus my efforts elsewhere. >> >> >> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, < >> tomasflo...@gmail.com> wrote: >> >>> >> >>> I was thinking (and I haven’t flushed it out completely but will >> throw the idea) that an alternative approach with this timeline could be to >> cut 9x branch around November/December? And then you could merge into >> master, it would have the latest changes from master plus the ref branch >> changes. From there any nightly build could be use to help test/debug. >> >>> >> >>> That said I don’t know for sure what are the changes in the branch >> that do not belong in 9. The problem with them being 10x only is that >> backports would potentially be more difficult for all the life of 9. >> >>> >> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul >> wrote: >> >> >I don't think it can be said what committers do and don't do with >> regards to running Solr. All of us would answer this differently and at >> different points in time. >> >> " I have run it in one large cluster, so it is certified to be bug >> free/stable" I don't think it's a reasonable approach. We need as much >> feedback from our users because each of them stress Solr in a different >> way. This is not to suggest that committers are not doing testing or their >> tests are not valid. When
Re: Solr Alpha (EA) release of Reference Branch
Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one. I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk. We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment. I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases. My 2 cents... Ilan Le mar. 6 oct. 2020 à 09:56, Noble Paul a écrit : > Yes, A docker image will definitely help. I wasn't trying to downplay that > > On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya > wrote: > > > > > > > Docker is not a big requirement for large scale installations. Most of > them already have their own install scripts. Availability of docker is not > important for them. If a user is only encouraged to install Solr because of > a docker image , most likely they are not running a large enough cluster > > > > I disagree, Noble. Having a docker image us going to be useful to some > clients, with complex usecases. Great point, David! > > > > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, < > ichattopadhy...@gmail.com> wrote: > >> > >> As I said, I'm *personally* not confident in putting such a big > changeset into master that wasn't vetted in a real user environment widely. > I have, in the past, done enough bad things to Solr (directly or > indirectly), and I don't want to repeat the same. Also, I'll be very > uncomfortable if someone else did so. > >> > >> Having said this, if someone else wants to port the changes over to > master *without first getting enough real world testing*, feel free to do > so, and I can focus my efforts elsewhere. > >> > >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, < > tomasflo...@gmail.com> wrote: > >>> > >>> I was thinking (and I haven’t flushed it out completely but will throw > the idea) that an alternative approach with this timeline could be to cut > 9x branch around November/December? And then you could merge into master, > it would have the latest changes from master plus the ref branch changes. > From there any nightly build could be use to help test/debug. > >>> > >>> That said I don’t know for sure what are the changes in the branch > that do not belong in 9. The problem with them being 10x only is that > backports would potentially be more difficult for all the life of 9. > >>> > >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul > wrote: > > >I don't think it can be said what committers do and don't do with > regards to running Solr. All of us would answer this differently and at > different points in time. > > " I have run it in one large cluster, so it is certified to be bug > free/stable" I don't think it's a reasonable approach. We need as much > feedback from our users because each of them stress Solr in a different > way. This is not to suggest that committers are not doing testing or their > tests are not valid. When I talk to the committers out here they say they > do not see any performance stability issues at all. But, my client reports > issues on a day to day basis. > > > > > Definitely publish a Docker image BTW -- it's the best way to try > out any software. > > Docker is not a big requirement for large scale installations. Most > of them already have their own install scripts. Availability of docker is > not important for them. If a user is only encouraged to install Solr > because of a docker image , most likely they are not running a large enough > cluster > > > > On Tue, Oct 6, 2020, 6:30 AM David Smiley wrote: > > > > Thanks so much for your responses Ishan... I'm getting much more > information in this thread than my attempts to get questions answered on > the JIRA issue months ago. And especially, thank you for volunteering for > the difficult porting efforts! > > > > Tomas said: > >> > >> I do agree with the previous comments that calling it "Solr 10" > (even with the "-alpha") would confuse users, maybe use "reference"? or >
Re: Solr Alpha (EA) release of Reference Branch
> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David! On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, < ichattopadhy...@gmail.com> wrote: > As I said, I'm *personally* not confident in putting such a big changeset > into master that wasn't vetted in a real user environment widely. I have, > in the past, done enough bad things to Solr (directly or indirectly), and I > don't want to repeat the same. Also, I'll be very uncomfortable if someone > else did so. > > Having said this, if someone else wants to port the changes over to master > *without first getting enough real world testing*, feel free to do so, and > I can focus my efforts elsewhere. > > On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, > wrote: > >> I was thinking (and I haven’t flushed it out completely but will throw >> the idea) that an alternative approach with this timeline could be to cut >> 9x branch around November/December? And then you could merge into master, >> it would have the latest changes from master plus the ref branch changes. >> From there any nightly build could be use to help test/debug. >> >> That said I don’t know for sure what are the changes in the branch that >> do not belong in 9. The problem with them being 10x only is that backports >> would potentially be more difficult for all the life of 9. >> >> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul wrote: >> >>> >I don't think it can be said what committers do and don't do with >>> regards to running Solr. All of us would answer this differently and at >>> different points in time. >>> >>> " I have run it in one large cluster, so it is certified to be bug >>> free/stable" I don't think it's a reasonable approach. We need as much >>> feedback from our users because each of them stress Solr in a >>> different way. This is not to suggest that committers are not doing testing >>> or their tests are not valid. When I talk to the committers out here they >>> say they do not see any performance stability issues at all. But, my client >>> reports issues on a day to day basis. >>> >>> >>> >>> > Definitely publish a Docker image BTW -- it's the best way to try out >>> any software. >>> >>> Docker is not a big requirement for large scale installations. Most of >>> them already have their own install scripts. Availability of docker is not >>> important for them. If a user is only encouraged to install Solr because of >>> a docker image , most likely they are not running a large enough cluster >>> >>> >>> >>> On Tue, Oct 6, 2020, 6:30 AM David Smiley wrote: >>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago. And especially, thank you for volunteering for the difficult porting efforts! Tomas said: > I do agree with the previous comments that calling it "Solr 10" (even > with the "-alpha") would confuse users, maybe use "reference"? or maybe > something in reference to SOLR-14788? > I have the opposite opinion. This word "reference" is baffling to me despite whatever Mark's explanation is. I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better. *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on. Alex then Ishan said: > > Maybe we could release it to > > committers community first and dogfood it "internally"? > > Alex: It is meaningless. Committers don't run large scale > installations. We barely even have time to take care of running unit tests > before destabilizing our builds. We are not the right audience. However, > we > all can anyway check out the branch and start playing with it, even > without > a release. There are orgs that don't want to install any code that wasn't > officially released; this release is geared towards them (to help us test > this at their scale). > I don't think it can be said what committers do and don't do with regards to running Solr. All of us would answer this differently and at different points in time. From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes. (Putting on my Salesforce
Re: Solr Alpha (EA) release of Reference Branch
Yes, A docker image will definitely help. I wasn't trying to downplay that On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya wrote: > > > > Docker is not a big requirement for large scale installations. Most of them > > already have their own install scripts. Availability of docker is not > > important for them. If a user is only encouraged to install Solr because of > > a docker image , most likely they are not running a large enough cluster > > I disagree, Noble. Having a docker image us going to be useful to some > clients, with complex usecases. Great point, David! > > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, > wrote: >> >> As I said, I'm *personally* not confident in putting such a big changeset >> into master that wasn't vetted in a real user environment widely. I have, in >> the past, done enough bad things to Solr (directly or indirectly), and I >> don't want to repeat the same. Also, I'll be very uncomfortable if someone >> else did so. >> >> Having said this, if someone else wants to port the changes over to master >> *without first getting enough real world testing*, feel free to do so, and I >> can focus my efforts elsewhere. >> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, >> wrote: >>> >>> I was thinking (and I haven’t flushed it out completely but will throw the >>> idea) that an alternative approach with this timeline could be to cut 9x >>> branch around November/December? And then you could merge into master, it >>> would have the latest changes from master plus the ref branch changes. >>> From there any nightly build could be use to help test/debug. >>> >>> That said I don’t know for sure what are the changes in the branch that do >>> not belong in 9. The problem with them being 10x only is that backports >>> would potentially be more difficult for all the life of 9. >>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul wrote: >I don't think it can be said what committers do and don't do with regards >to running Solr. All of us would answer this differently and at >different points in time. " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis. > Definitely publish a Docker image BTW -- it's the best way to try out > any software. Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster On Tue, Oct 6, 2020, 6:30 AM David Smiley wrote: > > Thanks so much for your responses Ishan... I'm getting much more > information in this thread than my attempts to get questions answered on > the JIRA issue months ago. And especially, thank you for volunteering > for the difficult porting efforts! > > Tomas said: >> >> I do agree with the previous comments that calling it "Solr 10" (even >> with the "-alpha") would confuse users, maybe use "reference"? or maybe >> something in reference to SOLR-14788? > > > I have the opposite opinion. This word "reference" is baffling to me > despite whatever Mark's explanation is. I like the justification Ishan > gave for 10-alpha and I don't think I could re-phrase his justification > any better. *If* the release was _not_ official (thus wouldn't show up > in the usual places anyone would look for a release), I think it would > alleviate that confusion concern even more, although I think "alpha" > ought to be enough of a signal not to use it without digging deeper on > what's going on. > > Alex then Ishan said: >> >> > Maybe we could release it to >> > committers community first and dogfood it "internally"? >> >> Alex: It is meaningless. Committers don't run large scale installations. >> We barely even have time to take care of running unit tests before >> destabilizing our builds. We are not the right audience. However, we all >> can anyway check out the branch and start playing with it, even without >> a release. There are orgs that don't want to install any code that >> wasn't officially released; this release is geared towards them (to help >> us test this at their scale). > > > I don't think it can be said what committers do and don't do with regards > to running Solr.
Re: Solr Alpha (EA) release of Reference Branch
@Tomas Fernandez Lobbe This is a very risky proposition. Let's say we cut 9x and now there is a new master taken from the reference branch. That master is now going to be 10.0. * We never get it to the hands of the users anytime soon. We will never be able to reconcile these 2 branches * The master & 9x branches will diverge so much that we cannot make any meaningful commits to any of these branches Reference branch will have all the features of Solr 9.0 because it is forked from master (some latest changes are yet to be merged). So, reference branch is 100% compatible with master. A user should be able to do a drop in replacement between 9.0 & reference branch. It is also possible to have a cluster with nodes running 9x and reference together( A rolling restart should be possible). Let's take stock of what we have today. Choice 1: * Mark has made some significant changes to improve performance stability * Mark has promised to make all tests pass over the next 2-3 weeks * Ishan has promised to do the work to make a release possible * Ishan has promised to port changes to master as soon as we get enough user feedback. He has also expressed his reservations on doing this before it is tested in the wild * I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release Choice: 2 * Tomas promise to port changes from the branch to master * Tomas make a release Choice 3: * Status quo * We throw away all the good work that is done & demotivate the developers * Deprive users of all the promised goodness I'm not sure what else is a choice and who is volunteering to work on that. We need people who are willing to offer to get their hands dirty. Personally, I'm totally against choice #3 and I'm willing to collaborate with anyone who is has a plan to make all the goodness in the branch available to our users On Tue, Oct 6, 2020 at 6:39 PM Ishan Chattopadhyaya wrote: > > As I said, I'm *personally* not confident in putting such a big changeset > into master that wasn't vetted in a real user environment widely. I have, in > the past, done enough bad things to Solr (directly or indirectly), and I > don't want to repeat the same. Also, I'll be very uncomfortable if someone > else did so. > > Having said this, if someone else wants to port the changes over to master > *without first getting enough real world testing*, feel free to do so, and I > can focus my efforts elsewhere. > > On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, > wrote: >> >> I was thinking (and I haven’t flushed it out completely but will throw the >> idea) that an alternative approach with this timeline could be to cut 9x >> branch around November/December? And then you could merge into master, it >> would have the latest changes from master plus the ref branch changes. From >> there any nightly build could be use to help test/debug. >> >> That said I don’t know for sure what are the changes in the branch that do >> not belong in 9. The problem with them being 10x only is that backports >> would potentially be more difficult for all the life of 9. >> >> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul wrote: >>> >>> >I don't think it can be said what committers do and don't do with regards >>> >to running Solr. All of us would answer this differently and at different >>> >points in time. >>> >>> " I have run it in one large cluster, so it is certified to be bug >>> free/stable" I don't think it's a reasonable approach. We need as much >>> feedback from our users because each of them stress Solr in a different >>> way. This is not to suggest that committers are not doing testing or their >>> tests are not valid. When I talk to the committers out here they say they >>> do not see any performance stability issues at all. But, my client reports >>> issues on a day to day basis. >>> >>> >>> >>> > Definitely publish a Docker image BTW -- it's the best way to try out any >>> > software. >>> >>> Docker is not a big requirement for large scale installations. Most of them >>> already have their own install scripts. Availability of docker is not >>> important for them. If a user is only encouraged to install Solr because of >>> a docker image , most likely they are not running a large enough cluster >>> >>> >>> >>> On Tue, Oct 6, 2020, 6:30 AM David Smiley wrote: Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago. And especially, thank you for volunteering for the difficult porting efforts! Tomas said: > > I do agree with the previous comments that calling it "Solr 10" (even > with the "-alpha") would confuse users, maybe use "reference"? or maybe > something in reference to SOLR-14788? I have the opposite opinion. This word "reference" is baffling to me despite
Re: Solr Alpha (EA) release of Reference Branch
As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so. Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere. On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, wrote: > I was thinking (and I haven’t flushed it out completely but will throw the > idea) that an alternative approach with this timeline could be to cut 9x > branch around November/December? And then you could merge into master, it > would have the latest changes from master plus the ref branch changes. > From there any nightly build could be use to help test/debug. > > That said I don’t know for sure what are the changes in the branch that do > not belong in 9. The problem with them being 10x only is that backports > would potentially be more difficult for all the life of 9. > > On Mon, Oct 5, 2020 at 4:54 PM Noble Paul wrote: > >> >I don't think it can be said what committers do and don't do with >> regards to running Solr. All of us would answer this differently and at >> different points in time. >> >> " I have run it in one large cluster, so it is certified to be bug >> free/stable" I don't think it's a reasonable approach. We need as much >> feedback from our users because each of them stress Solr in a >> different way. This is not to suggest that committers are not doing testing >> or their tests are not valid. When I talk to the committers out here they >> say they do not see any performance stability issues at all. But, my client >> reports issues on a day to day basis. >> >> >> >> > Definitely publish a Docker image BTW -- it's the best way to try out >> any software. >> >> Docker is not a big requirement for large scale installations. Most of >> them already have their own install scripts. Availability of docker is not >> important for them. If a user is only encouraged to install Solr because of >> a docker image , most likely they are not running a large enough cluster >> >> >> >> On Tue, Oct 6, 2020, 6:30 AM David Smiley wrote: >> >>> Thanks so much for your responses Ishan... I'm getting much more >>> information in this thread than my attempts to get questions answered on >>> the JIRA issue months ago. And especially, thank you for volunteering for >>> the difficult porting efforts! >>> >>> Tomas said: >>> I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788? >>> >>> I have the opposite opinion. This word "reference" is baffling to me >>> despite whatever Mark's explanation is. I like the justification Ishan >>> gave for 10-alpha and I don't think I could re-phrase his justification any >>> better. *If* the release was _not_ official (thus wouldn't show up in the >>> usual places anyone would look for a release), I think it would alleviate >>> that confusion concern even more, although I think "alpha" ought to be >>> enough of a signal not to use it without digging deeper on what's going on. >>> >>> Alex then Ishan said: >>> > Maybe we could release it to > committers community first and dogfood it "internally"? Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale). >>> >>> I don't think it can be said what committers do and don't do with >>> regards to running Solr. All of us would answer this differently and at >>> different points in time. From time to time, though not at present, I've >>> been well positioned to try out a new version of Solr in a stage/test >>> environment to see how it goes. (Putting on my Salesforce metaphorical >>> hat...) Even though I'm not able to deploy it in a realistic way today, I'm >>> able to run a battery of tests to see if one of the features we depend on >>> have changed or is broken. That's useful feedback to an alpha release! >>> And even though I'm saying I'm not well positioned to try out some new Solr >>> release in a production-ish setting now, it's something I could make a good >>> case for internally since upgrades take a lot of effort where I work. It's >>> in our interest for SolrCloud to be very stable (of course). >>> >>> Regardless, I think what you're driving at Ishan is that you want an >>>
Re: [VOTE] Release Lucene/Solr 8.6.3 RC1
+1 (binding) SUCCESS! [1:05:14.591357] On Mon, Oct 5, 2020 at 1:13 PM Anshum Gupta wrote: > +1 (binding) > > SUCCESS! [1:00:37.423566] > > Tried basic indexing and search and ran the smoke tester. > > On Sat, Oct 3, 2020 at 6:53 PM Jason Gerlowski > wrote: > >> Please vote for release candidate 1 for Lucene/Solr 8.6.3 >> >> >> The artifacts can be downloaded from: >> >> >> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4 >> >> >> You can run the smoke tester directly with this command: >> >> >> python3 -u dev-tools/scripts/smokeTestRelease.py \ >> >> >> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4 >> >> >> The vote will be open for at least 72 hours i.e. until 2020-10-07 02:00 >> UTC. >> >> >> [ ] +1 approve >> >> [ ] +0 no opinion >> >> [ ] -1 disapprove (and reason why) >> >> >> Here is my +1 >> > > > -- > Anshum Gupta >
