Re: solr 5 leaving tomcat, will I be the only one fearing about this?
> For cleaner version control on my system, I create $SOLR_HOME/lib as a > symlink to another directory that I create myself -- > $INSTALL_DIR/solrlib ... that way, all binary stuff like jars is in the > *program* version control location, not the *data* location. That’s right. It is possible to change the location of sharedLib in solr.xml, but perhaps we should add another location as default too, such as $INSTALL_DIR/server/solr-lib. And perhaps it should be possible to add lib dirs with a startup option? -Dsolr.lib.dir=/path/to/libs Most libs are version sensitive and something you’d want to consider for upgrade when you upgrade Solr, so they fit better in INSTALL_DIR than in SOLR_HOME. But there may be other libs that provide some business logic and fit well in SOLR_HOME/lib. Jan
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Thanks John... yes that was the first idea came to our mind, but it will require doubling our servers (in replica data centers as well etc), definitely we can't afford the cost. We have thought of first establishing a small pool of 'hot' servers and use them to take incoming new index data using upgraded solr version (a relative much smaller resources pool), meanwhile take one exist server (and its replicas as well) at one time to upgrade one by one. Although most (99%) index will happen at the small hot servers pool, but there are still some of the updates to the 'cold' servers at all time. We will also need to introduce a write lock down on the impacted servers... with one server at a time, the scope of impact will be reduced to its minimum... I am pretty sure we must not be the only one that has to face the re-index issue with large data set... am I correct? If there is a better approach, please share... thanks a lot! Renee -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300550.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Renee - you have probably already thought of this, but just in case it helps... (It helped me a lot several years ago and I hadn't thought of it at the time...) If you end up needing to do a big re-index, Production doesn't have to be affected (assuming you have the hardware/cloud resources). You can reindex into a non-production environment, test, and when all is well, just point load balancers at the new Solr environment. In this scenario your users would probably not even notice the change unless you tell them... Of course this does NOT in any way lessen the other stressors you've been discussing - and - as I said, you probably already thought of it... On Mon, Oct 10, 2016 at 10:18 AM, Renee Sun <renee_...@mcafee.com> wrote: > Shawn and Ari, > the 3rd party jars are exactly just one of the concerns I have. > We had more than just a multi-lingual integration, we have to integrate > with > many other 3rd party tools. We basically deploy all those jars into an > 'external' lib extension path in production, then for each 3rd party tools > involved, we have to follow their instruction to integrate with solr/tomcat > by symlink them into either tomcat/lib or ourwebapps/WEB-INF/lib etc. We > will need to rewrite these integration in the build process I imagine. > > I am sure there will be a lot of other work if we go for this 'upgrade' ... > I bet we will need to re-index the data ... for each major solr version > upgrade (like from 3.0 to 4.0) the data needs to be re-indexed and this is > another huge concern. > > Our data is not that BIG considering what others have nowadays, but still > few hundreds of terabytes are a pain in the neck to go through re-index > process, resource wise and time wise. Almost a road block. > > I have tested that solr shard query works with querying data from solr > servers with different versions, so we could upgrade our production solr > servers and reindex data on them one by one... but still, it is almost a > non-practical thing to do, similar to how Ari feels, I also would rather > not > upgrade ... > > I guess we fortunately started to use Solr long time ago (which benefited > our system of its key feature to search for email content in our SAAS > services), but on the other side we become so depending on the old version > of solr, as a reality with how Solr evolves, it is so hard for us to keep > up > with these upgrades... > > few years ago, the scalability become a major issue in our system, I did a > lot of experiments with the Solr 4.0, unfortunately by then the lack of > supporting of multi-tenancy as well as other fundamental flaws drove it out > of our choices, so our team ended up with developing a light weighted > scalalible layer wrapping up on top of solr, which worked very well but > here > we are... from all elements: build process, architect, data migration etc > it is scary. > > Good discussion and it is great help ... :-) > Renee > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- > fearing-about-this-tp4300065p4300523.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Shawn and Ari, the 3rd party jars are exactly just one of the concerns I have. We had more than just a multi-lingual integration, we have to integrate with many other 3rd party tools. We basically deploy all those jars into an 'external' lib extension path in production, then for each 3rd party tools involved, we have to follow their instruction to integrate with solr/tomcat by symlink them into either tomcat/lib or ourwebapps/WEB-INF/lib etc. We will need to rewrite these integration in the build process I imagine. I am sure there will be a lot of other work if we go for this 'upgrade' ... I bet we will need to re-index the data ... for each major solr version upgrade (like from 3.0 to 4.0) the data needs to be re-indexed and this is another huge concern. Our data is not that BIG considering what others have nowadays, but still few hundreds of terabytes are a pain in the neck to go through re-index process, resource wise and time wise. Almost a road block. I have tested that solr shard query works with querying data from solr servers with different versions, so we could upgrade our production solr servers and reindex data on them one by one... but still, it is almost a non-practical thing to do, similar to how Ari feels, I also would rather not upgrade ... I guess we fortunately started to use Solr long time ago (which benefited our system of its key feature to search for email content in our SAAS services), but on the other side we become so depending on the old version of solr, as a reality with how Solr evolves, it is so hard for us to keep up with these upgrades... few years ago, the scalability become a major issue in our system, I did a lot of experiments with the Solr 4.0, unfortunately by then the lack of supporting of multi-tenancy as well as other fundamental flaws drove it out of our choices, so our team ended up with developing a light weighted scalalible layer wrapping up on top of solr, which worked very well but here we are... from all elements: build process, architect, data migration etc it is scary. Good discussion and it is great help ... :-) Renee -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300523.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
For what it's worth / in case it's helpful... I haven't dealt with JDBC in this context so I can't offer anything useful there... You can reference the data in Zookeeper when creating a new collection - so you don't need to put the configs anywhere on the Solr boxes themselves. It's not automatic (as far as I know) but the following command line or HTTP line will do it... (In my case "expath" was the name of the collection AND the name of the collection's configuration in Zookeeper.) Note: in my case the solr user owned the files and so I had to use the following syntax to "run as" solr user... The command inside the quotes is the important part. The -n is what allows you to reference an already-existing configuration setting "node" in Zookeeper. sudo su - solr -c "/opt/solr/bin/solr create -c expath -n expath -shards 1 -replicationFactor 2" For the HTTP line, replace localhost with the ip or hostname of your running Solr instance http://localhost:8983/solr/admin/collections?action=CREATE=expath=1=3=1=expath On Mon, Oct 10, 2016 at 5:59 AM, Aristedes Maniatiswrote: > Hi Shalin > > Thanks for your reply. > > On 10/10/16 3:28pm, Shalin Shekhar Mangar wrote: > > As far as configuration is concerned -- everything Solr specific goes to > > zookeeper. The solr.xml can also be put into zookeeper. JNDI is not > > supported -- can you explain why you need it? > > Where do I put the JDBC connection data for fetching my source documents? > Do I have to write my passwords to the zookeeper cluster also? > > > We also use com.vividsolutions.jts.geom so we need to add this jar > somewhere. The only places that seem to work are inside the Jetty > installation because that's where the classloader will find it. > ${solr.installation.dir}/server/lib or into ${solr.installation.dir}/ > server/solr-webapp/webapp/WEB-INF/lib/ > > In the longer term I'll get rid of that jar since equivalent functionality > is now available in Solr. But I'm not there yet. > > > > Can cluster properties solve > > the problem? or replica properties? Both of those can go into zookeeper. > > Can I create a collection by just adding configuration to zookeeper? > > > > Integration testing is possible using MiniSolrCloudCluster which is built > > to be used programatically. It starts a real zookeeper instance along > with > > real solr instances on jetties. > > MiniSolrCloudCluster might just be the solution I need for this thing. > Thanks. > > > > Most other things are by design e.g. whole cluster changes on > configuration > > change otherwise sneaky bugs where one node misses a config change are > > easily possible and very very difficult to debug. But you can e.g. > create a > > new collection with just one node, stage your changes, run health checks > > and update the configuration of the main collection. > > > > The old master/slave setup was very simple. The cloud is a whole new ball > > game. Having control of jetty has given Solr a lot of flexibility. At > this > > time I discourage anyone from changing anything inside Jetty's config. If > > there are certain things that are not possible without then please let us > > know so figure out how to build such things in Solr itself. We want jetty > > to be a implementation detail and no more. > > > > If you have suggestions on how we can fix some of these problems, please > > speak up. > > Mostly it would be nice to have more hooks in the startup scripts. So I'd > like to pass additional folders to add to the classpath. Or JNDI settings. > > Perhaps my problems are related to the fact that I expect to store my > configuration in version control. Zookeeper then has to be driven by some > release process (in my case Jenkins with gradle scripts to push the data > into production). But that means I'm now storing my database passwords in > version control alongside configuration. Mixing up passwords and > configuration is bad, which is why JNDI was invented and is useful. I'm not > sure how Solr/Zookeeper intends us to solve this. > > Next, let's say our data centre burns to the ground and I need to > provision everything from scratch. > > 1. I want to be able to install Solr/Zookeeper. I've scripted this in > saltstack so we are all good. > 2. Then I press "release" in Jenkins. This pushes my configuration into > Zookeeper. > 3. But Solr now has no collections, even though they are defined in > zookeeper, so now I need to run another script to create them and start the > population of the data. > 4. Do I now need some other command or process to push the JDBC passwords > into Zookeeper from my secure store? > > Could Solr automatically provision new collections purely from the data > stored in Zookeeper? > > > Cheers > Ari > > > > > On Sun, Oct 9, 2016 at 5:41 AM, Aristedes Maniatis > wrote: > > > >> On 9/10/16 2:09am, Shawn Heisey wrote: > >>> One of the historical challenges on this mailing list is that we were > >>> rarely aware of what steps the
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
On 10/10/2016 6:59 AM, Aristedes Maniatis wrote: > We also use com.vividsolutions.jts.geom so we need to add this jar > somewhere. The only places that seem to work are inside the Jetty > installation because that's where the classloader will find it. > ${solr.installation.dir}/server/lib or into > ${solr.installation.dir}/server/solr-webapp/webapp/WEB-INF/lib/ JTS is different than other jars. If it gets loaded by the Solr classloader, it doesn't seem to actually work. Installation of THAT particular jar is recommended in WEB-INF/lib, so it loads at the same time as Solr itself. Pretty much any other extra jar (analysis components, DIH, JDBC driver, etc) can be placed in $SOLR_HOME/lib and this location will be processed exactly once during Solr startup, with the jars available to every core. No solrconfig.xml configuration is necessary for jars in this location to load. This is the way that I recommend dealing with jars other than JTS. For cleaner version control on my system, I create $SOLR_HOME/lib as a symlink to another directory that I create myself -- $INSTALL_DIR/solrlib ... that way, all binary stuff like jars is in the *program* version control location, not the *data* location. Thanks, Shawn
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Hi Shalin Thanks for your reply. On 10/10/16 3:28pm, Shalin Shekhar Mangar wrote: > As far as configuration is concerned -- everything Solr specific goes to > zookeeper. The solr.xml can also be put into zookeeper. JNDI is not > supported -- can you explain why you need it? Where do I put the JDBC connection data for fetching my source documents? Do I have to write my passwords to the zookeeper cluster also? We also use com.vividsolutions.jts.geom so we need to add this jar somewhere. The only places that seem to work are inside the Jetty installation because that's where the classloader will find it. ${solr.installation.dir}/server/lib or into ${solr.installation.dir}/server/solr-webapp/webapp/WEB-INF/lib/ In the longer term I'll get rid of that jar since equivalent functionality is now available in Solr. But I'm not there yet. > Can cluster properties solve > the problem? or replica properties? Both of those can go into zookeeper. Can I create a collection by just adding configuration to zookeeper? > Integration testing is possible using MiniSolrCloudCluster which is built > to be used programatically. It starts a real zookeeper instance along with > real solr instances on jetties. MiniSolrCloudCluster might just be the solution I need for this thing. Thanks. > Most other things are by design e.g. whole cluster changes on configuration > change otherwise sneaky bugs where one node misses a config change are > easily possible and very very difficult to debug. But you can e.g. create a > new collection with just one node, stage your changes, run health checks > and update the configuration of the main collection. > > The old master/slave setup was very simple. The cloud is a whole new ball > game. Having control of jetty has given Solr a lot of flexibility. At this > time I discourage anyone from changing anything inside Jetty's config. If > there are certain things that are not possible without then please let us > know so figure out how to build such things in Solr itself. We want jetty > to be a implementation detail and no more. > > If you have suggestions on how we can fix some of these problems, please > speak up. Mostly it would be nice to have more hooks in the startup scripts. So I'd like to pass additional folders to add to the classpath. Or JNDI settings. Perhaps my problems are related to the fact that I expect to store my configuration in version control. Zookeeper then has to be driven by some release process (in my case Jenkins with gradle scripts to push the data into production). But that means I'm now storing my database passwords in version control alongside configuration. Mixing up passwords and configuration is bad, which is why JNDI was invented and is useful. I'm not sure how Solr/Zookeeper intends us to solve this. Next, let's say our data centre burns to the ground and I need to provision everything from scratch. 1. I want to be able to install Solr/Zookeeper. I've scripted this in saltstack so we are all good. 2. Then I press "release" in Jenkins. This pushes my configuration into Zookeeper. 3. But Solr now has no collections, even though they are defined in zookeeper, so now I need to run another script to create them and start the population of the data. 4. Do I now need some other command or process to push the JDBC passwords into Zookeeper from my secure store? Could Solr automatically provision new collections purely from the data stored in Zookeeper? Cheers Ari > On Sun, Oct 9, 2016 at 5:41 AM, Aristedes Maniatiswrote: > >> On 9/10/16 2:09am, Shawn Heisey wrote: >>> One of the historical challenges on this mailing list is that we were >>> rarely aware of what steps the user had taken to install or start Solr, >>> and we had to support pretty much any scenario. Since 5.0, the number >>> of supported ways to deploy and start Solr is greatly reduced, and those >>> ways were written by the project, so we tend to have a better >>> understanding of what is happening when a user starts Solr. We also >>> usually know the relative location of the logfiles and Solr's data. >> >> >> This migration is causing a lot of grief for us as well, and we are still >> struggling to get all the bits in place. Before: >> >> * gradle build script >> * gradle project includes our own unit tests, run in jenkins >> * generates war file >> * relevant configuration is embedded into the build >> * deployment specific variables (db uris, passwords, ip addresses) >> conveniently contained in one context.xml file >> >> >> Now: >> >> * Solr version is no longer bound to our tests or configuration >> >> * configuration is now scattered in three places: >> - zookeeper >> - solr.xml in the data directory >> - jetty files as part of the solr install that you need to replace (for >> example to set JNDI properties) >> >> * deployment is also scattered: >> - Solr platform specific package manager (pkg in FreeBSD in my case, >> which I've had
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
As far as configuration is concerned -- everything Solr specific goes to zookeeper. The solr.xml can also be put into zookeeper. JNDI is not supported -- can you explain why you need it? Can cluster properties solve the problem? or replica properties? Both of those can go into zookeeper. Integration testing is possible using MiniSolrCloudCluster which is built to be used programatically. It starts a real zookeeper instance along with real solr instances on jetties. Most other things are by design e.g. whole cluster changes on configuration change otherwise sneaky bugs where one node misses a config change are easily possible and very very difficult to debug. But you can e.g. create a new collection with just one node, stage your changes, run health checks and update the configuration of the main collection. The old master/slave setup was very simple. The cloud is a whole new ball game. Having control of jetty has given Solr a lot of flexibility. At this time I discourage anyone from changing anything inside Jetty's config. If there are certain things that are not possible without then please let us know so figure out how to build such things in Solr itself. We want jetty to be a implementation detail and no more. If you have suggestions on how we can fix some of these problems, please speak up. On Sun, Oct 9, 2016 at 5:41 AM, Aristedes Maniatiswrote: > On 9/10/16 2:09am, Shawn Heisey wrote: > > One of the historical challenges on this mailing list is that we were > > rarely aware of what steps the user had taken to install or start Solr, > > and we had to support pretty much any scenario. Since 5.0, the number > > of supported ways to deploy and start Solr is greatly reduced, and those > > ways were written by the project, so we tend to have a better > > understanding of what is happening when a user starts Solr. We also > > usually know the relative location of the logfiles and Solr's data. > > > This migration is causing a lot of grief for us as well, and we are still > struggling to get all the bits in place. Before: > > * gradle build script > * gradle project includes our own unit tests, run in jenkins > * generates war file > * relevant configuration is embedded into the build > * deployment specific variables (db uris, passwords, ip addresses) > conveniently contained in one context.xml file > > > Now: > > * Solr version is no longer bound to our tests or configuration > > * configuration is now scattered in three places: > - zookeeper > - solr.xml in the data directory > - jetty files as part of the solr install that you need to replace (for > example to set JNDI properties) > > * deployment is also scattered: > - Solr platform specific package manager (pkg in FreeBSD in my case, > which I've had to write myself since it didn't exist) > - updating config files above > - writing custom scripts to push Zookeeper configuration into production > - creating collections/cores using the API rather than in a config file > > * unit testing no longer possible since you can't run a mock zookeeper > instance > > * zookeeper is very hard to integrate with deployment processes (salt, > puppet, etc) since configuration is no longer a set of version controlled > files > > * you can't change the configuration of one node as a 'soft deployment': > the whole cluster needs to be changed at once > > If we didn't need a less broken replication solution, I'd stay on Solr4 > forever. > > > I really liked the old war deployment. It bound the solr version and > configuration management into our version controlled source repository > except for one context.xml file that contained server specific deployment > options. Nice. > > The new arrangement is a mess. > > > Ari > > > > -- > --> > Aristedes Maniatis > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A > -- Regards, Shalin Shekhar Mangar.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Although I don't like docker, I do make heavy use of saltstack with jails plus ZFS snapshots to shipping virtual machines around. Docker is mostly just a reinvention of BSD/Solaris jails/zones. You might prefer docker's own custom dockerfile language and I prefer python with salt. But ultimately it is a lot of workaround for the stupid decisions Solr made by scattering configuration everywhere and forcing us to customise the application itself. But thanks for the confirmation that "this is just the way it is"... I'm really not missing some easy way to do all this. Ari On 10/10/16 6:59am, Georg Sorst wrote: > If you can, switch to Docker (https://hub.docker.com/_/solr/). It's a pain > to get everything going the right way, but once it's running you get a lot > of stuff for free: > > * Deployment, scaling etc. is all taken care of by the Docker ecosystem > * Testing is a breeze. Need a clean Solr instance to run your application > against? It's just one command line away > * You can version the Dockerfile (Docker build instructions), so you can > version your whole setup. For example we add our own web app to the Docker > image (we shouldn't be doing that, I know) and put the resulting images > into our private Docker repository > > Aristedes Maniatisschrieb am So., 9. Okt. 2016 um > 02:14 Uhr: > >> On 9/10/16 11:11am, Aristedes Maniatis wrote: >>> * deployment is also scattered: >>> - Solr platform specific package manager (pkg in FreeBSD in my case, >> which I've had to write myself since it didn't exist) >>> - updating config files above >>> - writing custom scripts to push Zookeeper configuration into production >>> - creating collections/cores using the API rather than in a config file >> >> Oh, and pushing additional jars (like a JDBC adapter) into a special >> folder. Again, not easily testable or version controlled. >> >> >> Ari >> >> >> >> -- >> --> >> Aristedes Maniatis >> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >> > -- --> Aristedes Maniatis GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
If you can, switch to Docker (https://hub.docker.com/_/solr/). It's a pain to get everything going the right way, but once it's running you get a lot of stuff for free: * Deployment, scaling etc. is all taken care of by the Docker ecosystem * Testing is a breeze. Need a clean Solr instance to run your application against? It's just one command line away * You can version the Dockerfile (Docker build instructions), so you can version your whole setup. For example we add our own web app to the Docker image (we shouldn't be doing that, I know) and put the resulting images into our private Docker repository Aristedes Maniatisschrieb am So., 9. Okt. 2016 um 02:14 Uhr: > On 9/10/16 11:11am, Aristedes Maniatis wrote: > > * deployment is also scattered: > > - Solr platform specific package manager (pkg in FreeBSD in my case, > which I've had to write myself since it didn't exist) > > - updating config files above > > - writing custom scripts to push Zookeeper configuration into production > > - creating collections/cores using the API rather than in a config file > > Oh, and pushing additional jars (like a JDBC adapter) into a special > folder. Again, not easily testable or version controlled. > > > Ari > > > > -- > --> > Aristedes Maniatis > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
On 9/10/16 11:11am, Aristedes Maniatis wrote: > * deployment is also scattered: > - Solr platform specific package manager (pkg in FreeBSD in my case, which > I've had to write myself since it didn't exist) > - updating config files above > - writing custom scripts to push Zookeeper configuration into production > - creating collections/cores using the API rather than in a config file Oh, and pushing additional jars (like a JDBC adapter) into a special folder. Again, not easily testable or version controlled. Ari -- --> Aristedes Maniatis GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
On 9/10/16 2:09am, Shawn Heisey wrote: > One of the historical challenges on this mailing list is that we were > rarely aware of what steps the user had taken to install or start Solr, > and we had to support pretty much any scenario. Since 5.0, the number > of supported ways to deploy and start Solr is greatly reduced, and those > ways were written by the project, so we tend to have a better > understanding of what is happening when a user starts Solr. We also > usually know the relative location of the logfiles and Solr's data. This migration is causing a lot of grief for us as well, and we are still struggling to get all the bits in place. Before: * gradle build script * gradle project includes our own unit tests, run in jenkins * generates war file * relevant configuration is embedded into the build * deployment specific variables (db uris, passwords, ip addresses) conveniently contained in one context.xml file Now: * Solr version is no longer bound to our tests or configuration * configuration is now scattered in three places: - zookeeper - solr.xml in the data directory - jetty files as part of the solr install that you need to replace (for example to set JNDI properties) * deployment is also scattered: - Solr platform specific package manager (pkg in FreeBSD in my case, which I've had to write myself since it didn't exist) - updating config files above - writing custom scripts to push Zookeeper configuration into production - creating collections/cores using the API rather than in a config file * unit testing no longer possible since you can't run a mock zookeeper instance * zookeeper is very hard to integrate with deployment processes (salt, puppet, etc) since configuration is no longer a set of version controlled files * you can't change the configuration of one node as a 'soft deployment': the whole cluster needs to be changed at once If we didn't need a less broken replication solution, I'd stay on Solr4 forever. I really liked the old war deployment. It bound the solr version and configuration management into our version controlled source repository except for one context.xml file that contained server specific deployment options. Nice. The new arrangement is a mess. Ari -- --> Aristedes Maniatis GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
On 10/7/2016 5:13 PM, Renee Sun wrote: > I just read through the following link Shawn shared in his reply: > https://wiki.apache.org/solr/WhyNoWar > > While the following statement is true: > > "Supporting a single set of binary bits is FAR easier than worrying > about what kind of customized environment the user has chosen for their > deployment. " One of the historical challenges on this mailing list is that we were rarely aware of what steps the user had taken to install or start Solr, and we had to support pretty much any scenario. Since 5.0, the number of supported ways to deploy and start Solr is greatly reduced, and those ways were written by the project, so we tend to have a better understanding of what is happening when a user starts Solr. We also usually know the relative location of the logfiles and Solr's data. > But it also probably will reduce the flexibility... for example, we tune for > Scalability at tomcat level, such as its thread pool etc. I assume the > standalone Solr (which is still using Jetty underlying) would expose > sufficient configurable 'knobs' that allow me to turn 'Solr' to meet our > data work load. Yes, Jetty's full configuration would be at your disposal. The configuration which tends to matter the most -- maxThreads -- has been pre-tuned to a very large value. The other settings have been set to values that work well for the vast majority of Solr installs. > If we want to minimize the migration work, our existing business logic > component will remain in tomcat, then the fact that we will have co-exist > jetty and tomcat deployed in production system is a bit strange... or is it? > > Even if I could port our webapps to use Jetty, I assume the way solr is > embedding Jetty I would be able to integrate at that level, I probably end > up with 2 Jetty container instances running on same server, correct? It is > still too early for me to be sure how this will impact our system but I am a > little worried. You can take the server/solr-webapp/webapp directory from the download in 5.3 and later, which contains the same information that used to be in the .war file, and install it as a webapp into Tomcat. If you use something other than "/solr" as the context URL, then the admin UI in 6.0 and later won't work right, but Solr itself (the HTTP API) should work just fine no matter what the context URL is. I don't have precise information about how to do this kind of install, but I know it CAN be done. The tomcat documentation refers to this as an "exploded web application" and I think you can deploy such an application either directly by copying it into the correct directory, or with a context file. In the Jetty that comes with Solr, the exploded webapp is deployed with a context file. Unless they use tomcat-specific code, you might also be able to take your other webapps and install them into the Jetty that comes with Solr. The Jetty bits that are included with Solr are identical to a Jetty package downloaded from eclipse.org, with a few components removed that Solr doesn't use. I wrote the "WhyNoWar" wiki page, so any errors that it contains are most likely mine. I'm not aware of any errors, but it's possible that some are there. Thanks, Shawn
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Chegg uses a servlet filter to collect metrics on each request and forward them to Graphite and New Relic. We can configure that because we have hooks into the webapp config. This works with 4.10.4. We haven’t tried it with a later version, but it could be a blocker for upgrading. Our SLA commitments to internal clients are based on 95th percentile response times. We need to track those. We could (should) contribute this, but it would still be an “edit the code” integration for different metrics systems. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 7, 2016, at 4:13 PM, Renee Sun <renee_...@mcafee.com> wrote: > > I just read through the following link Shawn shared in his reply: > https://wiki.apache.org/solr/WhyNoWar > > While the following statement is true: > > "Supporting a single set of binary bits is FAR easier than worrying > about what kind of customized environment the user has chosen for their > deployment. " > > But it also probably will reduce the flexibility... for example, we tune for > Scalability at tomcat level, such as its thread pool etc. I assume the > standalone Solr (which is still using Jetty underlying) would expose > sufficient configurable 'knobs' that allow me to turn 'Solr' to meet our > data work load. > > If we want to minimize the migration work, our existing business logic > component will remain in tomcat, then the fact that we will have co-exist > jetty and tomcat deployed in production system is a bit strange... or is it? > > Even if I could port our webapps to use Jetty, I assume the way solr is > embedding Jetty I would be able to integrate at that level, I probably end > up with 2 Jetty container instances running on same server, correct? It is > still too early for me to be sure how this will impact our system but I am a > little worried. > > Renee > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300259.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
I think it just means they won't officially support deploying war to tomcat or other container. make sense to me if I was in charge of solr, I would just support jetty, predictable with a single configuration. I wouldn't want to spent countless hrs supporting various configurations. Instead use those hrs to further solr development. I'm sure someone that has enough familiarity with tomcat and Java and solr shouldn't have any issue, after all solr is free but you need to pay for support. On Fri, Oct 7, 2016, 7:13 PM Renee Sun <renee_...@mcafee.com> wrote: > I just read through the following link Shawn shared in his reply: > https://wiki.apache.org/solr/WhyNoWar > > While the following statement is true: > > "Supporting a single set of binary bits is FAR easier than worrying > about what kind of customized environment the user has chosen for their > deployment. " > > But it also probably will reduce the flexibility... for example, we tune > for > Scalability at tomcat level, such as its thread pool etc. I assume the > standalone Solr (which is still using Jetty underlying) would expose > sufficient configurable 'knobs' that allow me to turn 'Solr' to meet our > data work load. > > If we want to minimize the migration work, our existing business logic > component will remain in tomcat, then the fact that we will have co-exist > jetty and tomcat deployed in production system is a bit strange... or is > it? > > Even if I could port our webapps to use Jetty, I assume the way solr is > embedding Jetty I would be able to integrate at that level, I probably end > up with 2 Jetty container instances running on same server, correct? It is > still too early for me to be sure how this will impact our system but I am > a > little worried. > > Renee > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300259.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
I just read through the following link Shawn shared in his reply: https://wiki.apache.org/solr/WhyNoWar While the following statement is true: "Supporting a single set of binary bits is FAR easier than worrying about what kind of customized environment the user has chosen for their deployment. " But it also probably will reduce the flexibility... for example, we tune for Scalability at tomcat level, such as its thread pool etc. I assume the standalone Solr (which is still using Jetty underlying) would expose sufficient configurable 'knobs' that allow me to turn 'Solr' to meet our data work load. If we want to minimize the migration work, our existing business logic component will remain in tomcat, then the fact that we will have co-exist jetty and tomcat deployed in production system is a bit strange... or is it? Even if I could port our webapps to use Jetty, I assume the way solr is embedding Jetty I would be able to integrate at that level, I probably end up with 2 Jetty container instances running on same server, correct? It is still too early for me to be sure how this will impact our system but I am a little worried. Renee -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300259.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Rajesh was right on Renee -- the only big concern might be if that other code is tightly-coupled to Tomcat or to other things which *must* have Tomcat. But it sounds to me as if your multi-lingual processors - if they just work with Solr/Tomcat - out to "just work" with Solr/Jetty - or work with minimal tweaking... Good luck! Interesting problem! On Fri, Oct 7, 2016 at 3:47 PM, Renee Sun <renee_...@mcafee.com> wrote: > Thanks everyone, I think this is very helpful... I will post more specific > questions once we start to get more familiar with solr 6. > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- > fearing-about-this-tp4300065p4300253.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Thanks everyone, I think this is very helpful... I will post more specific questions once we start to get more familiar with solr 6. -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300253.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
On 10/7/2016 10:33 AM, Renee Sun wrote: > In our production, we have cloud based big data indexing using Solr for many > years. We have developed lots business related logic/component deployed as > webapps working seamlessly with solr. > > I will give you a simple example, we purchased multi-lingual processors (and > many other 3rd parties) which we integrated with solr by carefully deploy > the libraries (e.g.) in the tomcat container so they work together. This > basically means we have to rewrite all those components to make it work with > solr 5 or 6. > > In my opinion, for those solr users like our company, it will really be > beneficial if Solr could keep supporting deploying a war and maintain > parallel support with its new standalone release, although this might be too > much work? For right now, Solr is still a webapp. It is not packaged as a .war file, but the information that would have been extracted from a .war file is still there, in exactly the same layout. It is already exploded, and the included Jetty accesses it as a webapp directly. You should be able to add the exploded webapp to another container like you would a .war file ... although you are on your own to make it work if you choose this path. https://wiki.apache.org/solr/WhyNoWar Eventually, no idea when, Solr will become a true standalone application, not a webapp. It is likely that this will be initially accomplished by embedding Jetty directly into the code, so internally Solr will remain much the same ... but after we pass that point, Solr will be free to evolve considerably. Thanks, Shawn
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Renee, I dont understand what would be the difficulties of putting your 3rd party distributions in some contrib folders and import them in your solr configs and your processors should get loaded using solr class loader. we have used our custom based synonym processor putting the jar in contrib folder and imported in our solrconfig.xml for ex: placed our custom jar file in ${solr_home}/contrib/analysis-extras/lucene-libs/ folder. with solr 4.9 version using jetty. Before we upgraded from tomcat based solr deployment we used to have our custom jar file in solr.war/WEB-INF/lib. I'm not sure if this answers your question, this to give you comfortability of solr with jetty, which is more preferred deployment. *Rajesh**.* On Fri, Oct 7, 2016 at 3:09 PM, John Bickerstaff <j...@johnbickerstaff.com> wrote: > I won't speak for the committers, but I'm guessing you won't find a lot of > support for the idea of continuing to provide a WAR file with the standard > SOLR releases... > > I feel for you and your situation however - I've had to wrestle with a > number of situations where a somewhat monolithic architecture was disturbed > by newer ways of doing things... > > That leaves 2 options... > > A self-maintained build of the latest SOLR into a WAR file (challenging in > it's own way) or the very least amount of code necessary to allow your > Tomcat-based tools to talk to a 6.x Solr server running in Jetty... > > I imagine something better than re-writing all your code can be done, > although I also don't think you can get away with no new code either... At > a high level, some kind of API/message bus/interface comes to mind, but I > don't know enough about your situation to be able to guess what might be a > good approach. > > If you're interested in a discussion about how to approach the change, I'd > be happy to offer ideas, but I'd need to know how your other tools > currently talk to Solr... Of course, you may not want to even have that > discussion if the task is just to big... > > On Fri, Oct 7, 2016 at 9:33 AM, Renee Sun <renee_...@mcafee.com> wrote: > > > Thanks ... but that is an extremely simplified situation. > > > > We are not just looking for Solr as a new tool to start using it. > > > > In our production, we have cloud based big data indexing using Solr for > > many > > years. We have developed lots business related logic/component deployed > as > > webapps working seamlessly with solr. > > > > I will give you a simple example, we purchased multi-lingual processors > > (and > > many other 3rd parties) which we integrated with solr by carefully deploy > > the libraries (e.g.) in the tomcat container so they work together. This > > basically means we have to rewrite all those components to make it work > > with > > solr 5 or 6. > > > > In my opinion, for those solr users like our company, it will really be > > beneficial if Solr could keep supporting deploying a war and maintain > > parallel support with its new standalone release, although this might be > > too > > much work? > > > > Thanks > > Renee > > > > > > > > -- > > View this message in context: http://lucene.472066.n3. > > nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- > > fearing-about-this-tp4300065p4300202.html > > Sent from the Solr - User mailing list archive at Nabble.com. > > >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
I won't speak for the committers, but I'm guessing you won't find a lot of support for the idea of continuing to provide a WAR file with the standard SOLR releases... I feel for you and your situation however - I've had to wrestle with a number of situations where a somewhat monolithic architecture was disturbed by newer ways of doing things... That leaves 2 options... A self-maintained build of the latest SOLR into a WAR file (challenging in it's own way) or the very least amount of code necessary to allow your Tomcat-based tools to talk to a 6.x Solr server running in Jetty... I imagine something better than re-writing all your code can be done, although I also don't think you can get away with no new code either... At a high level, some kind of API/message bus/interface comes to mind, but I don't know enough about your situation to be able to guess what might be a good approach. If you're interested in a discussion about how to approach the change, I'd be happy to offer ideas, but I'd need to know how your other tools currently talk to Solr... Of course, you may not want to even have that discussion if the task is just to big... On Fri, Oct 7, 2016 at 9:33 AM, Renee Sun <renee_...@mcafee.com> wrote: > Thanks ... but that is an extremely simplified situation. > > We are not just looking for Solr as a new tool to start using it. > > In our production, we have cloud based big data indexing using Solr for > many > years. We have developed lots business related logic/component deployed as > webapps working seamlessly with solr. > > I will give you a simple example, we purchased multi-lingual processors > (and > many other 3rd parties) which we integrated with solr by carefully deploy > the libraries (e.g.) in the tomcat container so they work together. This > basically means we have to rewrite all those components to make it work > with > solr 5 or 6. > > In my opinion, for those solr users like our company, it will really be > beneficial if Solr could keep supporting deploying a war and maintain > parallel support with its new standalone release, although this might be > too > much work? > > Thanks > Renee > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- > fearing-about-this-tp4300065p4300202.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Thanks ... but that is an extremely simplified situation. We are not just looking for Solr as a new tool to start using it. In our production, we have cloud based big data indexing using Solr for many years. We have developed lots business related logic/component deployed as webapps working seamlessly with solr. I will give you a simple example, we purchased multi-lingual processors (and many other 3rd parties) which we integrated with solr by carefully deploy the libraries (e.g.) in the tomcat container so they work together. This basically means we have to rewrite all those components to make it work with solr 5 or 6. In my opinion, for those solr users like our company, it will really be beneficial if Solr could keep supporting deploying a war and maintain parallel support with its new standalone release, although this might be too much work? Thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065p4300202.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
To give an exanmple: While you earlier would do some thing like Download and install Tomcat (or some other servlet container) Setup Tomcat as a service Download and unpack Solr Create a SOLR_HOME folder with correct content copy solr.war into tomcat/webapps set CATALINA_OPTS=“-Dsolr.solr.home=/path/to/home -Dsolr.x.y=z…. GC-flags etc” Setup Tomcat as a service service tomcat start You would with 6.x do: Download Solr and unpack the install-script solr/bin/install_solr_service solr-6.2.0.tgz # Install Tune /etc/default/solr.in.sh to your likings (mem, port, solr-home, Zk etc) service solr start (or bin/solr start [options]) Your client would talk to Solr on typically http://host.name:8983/solr/ as a standalone server, not as one out of many webapps on 8080. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 7. okt. 2016 kl. 02.32 skrev Alexandre Rafalovitch <arafa...@gmail.com>: > > Treat Solr as a blackbox standalone database. Your MySQL is running > standalone, right? > > And try to go to Solr 6, if you can. 5 is not latest anymore and there had > been lots of scaling improvements in 6. > > Regards, >Alex > > On 7 Oct 2016 5:02 AM, "Renee Sun" <renee_...@mcafee.com> wrote: > >> need some general advises please... >> >> our infra is built with multiple webapps with tomcat ... the scale layer is >> archived on top of those webapps which work hand-in-hand with solr admin >> APIs / shard queries / commit or optimize / core management etc etc. >> >> While I have not get a chance to actually play with solr 5 yet, just by >> imagination, we will be facing some huge changes in our infra to be able to >> upgrade to solr 5, yes? >> >> Thanks >> Renee >> >> >> >> -- >> View this message in context: http://lucene.472066.n3. >> nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- >> fearing-about-this-tp4300065.html >> Sent from the Solr - User mailing list archive at Nabble.com. >>
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
Treat Solr as a blackbox standalone database. Your MySQL is running standalone, right? And try to go to Solr 6, if you can. 5 is not latest anymore and there had been lots of scaling improvements in 6. Regards, Alex On 7 Oct 2016 5:02 AM, "Renee Sun" <renee_...@mcafee.com> wrote: > need some general advises please... > > our infra is built with multiple webapps with tomcat ... the scale layer is > archived on top of those webapps which work hand-in-hand with solr admin > APIs / shard queries / commit or optimize / core management etc etc. > > While I have not get a chance to actually play with solr 5 yet, just by > imagination, we will be facing some huge changes in our infra to be able to > upgrade to solr 5, yes? > > Thanks > Renee > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one- > fearing-about-this-tp4300065.html > Sent from the Solr - User mailing list archive at Nabble.com. >
solr 5 leaving tomcat, will I be the only one fearing about this?
need some general advises please... our infra is built with multiple webapps with tomcat ... the scale layer is archived on top of those webapps which work hand-in-hand with solr admin APIs / shard queries / commit or optimize / core management etc etc. While I have not get a chance to actually play with solr 5 yet, just by imagination, we will be facing some huge changes in our infra to be able to upgrade to solr 5, yes? Thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/solr-5-leaving-tomcat-will-I-be-the-only-one-fearing-about-this-tp4300065.html Sent from the Solr - User mailing list archive at Nabble.com.