Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
Hi Davis, I wanted to thank you for suggesting Ansible as one of the automation tool as it has been working very well in automating the deployments of Zookeeper, Solr on our clusters. Thanks, Susheel On Wed, Oct 21, 2015 at 10:47 AM, Davis, Daniel (NIH/NLM) [C] < daniel.da...@nih.gov> wrote: > Susheel, > > Our puppet stuff is very close to our infrastructure, using specific > Netapp volumes and such, and assuming some files come from NFS. > It is also personally embarrassing to me that we still use NIS - doh! > > -Original Message- > From: Susheel Kumar [mailto:susheel2...@gmail.com] > Sent: Tuesday, October 20, 2015 8:34 PM > To: solr-user@lucene.apache.org > Subject: Re: DevOps question : auto deployment/setup of Solr & Zookeeper > on medium-large clusters > > Thanks, Davis, Jeff. > > We are not using AWS. Is there any scripts/framework already developed > using puppet available? > > On Tue, Oct 20, 2015 at 7:59 PM, Jeff Wartes <jwar...@whitepages.com> > wrote: > > > > > If you’re using AWS, there’s this: > > https://github.com/LucidWorks/solr-scale-tk > > If you’re using chef, there’s this: > > https://github.com/vkhatri/chef-solrcloud > > > > (There are several other chef cookbooks for Solr out there, but this > > is the only one I’m aware of that supports Solr 5.3.) > > > > For ZK, I’m less familiar, but if you’re using chef there’s this: > > https://github.com/SimpleFinance/chef-zookeeper > > And this might be handy to know about too: > > https://github.com/Netflix/exhibitor/wiki > > > > > > On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]" > > <daniel.da...@nih.gov> > > wrote: > > > > >Waste of money in my opinion. I would point you towards other tools - > > >bash scripts and free configuration managers such as puppet, chef, salt, > > >or ansible.Depending on what development you are doing, you may want > > >a continuous integration environment. For a small company starting > out, > > >using a free CI, maybe SaaS, is a good choice. A professional version > > >such as Bamboo, TeamCity, Jenkins are almost essential in a large > > >enterprise if you are doing diverse builds. > > > > > >When you create a VM, you can generally specify a script to run after > the > > >VM is mostly created. There is a protocol (PXE Boot) that enables this > > >- a PXE server listens and hears that a new server with such-and-such > > >Ethernet Address is starting. The PXE server makes it boot like a > > >CD-ROM/DVD install, booting from installation media on the network and > > >installing.Once that install is down, a custom script may be > invoked. > > > This script is typically a bash script, because you may not be able to > > >count on too much else being installed. However, python/perl are also > > >reasonable choices - just be careful that the modules/libraries you are > > >using for the script are present.The same PXE protocol is used in > > >large on-premises installations (vCenter) and in the cloud > > >(AWS/Digital Ocean). We don't care about the PXE server - the point > > >is that you can generally run a bash script after your install. > > > > > >The bash script can bootstrap other services such as puppet, chef, or > > >salt, and/or setup keys so that push configuration management tools such > > >as ansible can reach the server. The bash script may even be smart > > >enough to do all of the setup you need, depending on what other servers > > >you need to configure. Smart bash scripts are good for a small > company, > > >but for large setups, I'd use puppet, chef, salt, and/or ansible. > > > > > >What I tend to do is to deploy things in such a way that puppet > > >(because it is what we use here) can setup things so that a "solradm" > > >account can setup everything else, and solr and zookeeper are running > as a "solrapp" > > >user using puppet.Then, my continuous integration server, which is > > >Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, > > >BuildBot), installs solr as "solradm" and sets it up to run as > "solrapp". > > > > > >I am not a systems administrator, and I'm not really in "DevOps", my > > >job is to be above all of that and do "systems architecture" which I > > >am lucky still involves coding both in system administration and > applications > > >development. So, that's my 2 cents. > &g
RE: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
Susheel, Our puppet stuff is very close to our infrastructure, using specific Netapp volumes and such, and assuming some files come from NFS. It is also personally embarrassing to me that we still use NIS - doh! -Original Message- From: Susheel Kumar [mailto:susheel2...@gmail.com] Sent: Tuesday, October 20, 2015 8:34 PM To: solr-user@lucene.apache.org Subject: Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters Thanks, Davis, Jeff. We are not using AWS. Is there any scripts/framework already developed using puppet available? On Tue, Oct 20, 2015 at 7:59 PM, Jeff Wartes <jwar...@whitepages.com> wrote: > > If you’re using AWS, there’s this: > https://github.com/LucidWorks/solr-scale-tk > If you’re using chef, there’s this: > https://github.com/vkhatri/chef-solrcloud > > (There are several other chef cookbooks for Solr out there, but this > is the only one I’m aware of that supports Solr 5.3.) > > For ZK, I’m less familiar, but if you’re using chef there’s this: > https://github.com/SimpleFinance/chef-zookeeper > And this might be handy to know about too: > https://github.com/Netflix/exhibitor/wiki > > > On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]" > <daniel.da...@nih.gov> > wrote: > > >Waste of money in my opinion. I would point you towards other tools - > >bash scripts and free configuration managers such as puppet, chef, salt, > >or ansible.Depending on what development you are doing, you may want > >a continuous integration environment. For a small company starting out, > >using a free CI, maybe SaaS, is a good choice. A professional version > >such as Bamboo, TeamCity, Jenkins are almost essential in a large > >enterprise if you are doing diverse builds. > > > >When you create a VM, you can generally specify a script to run after the > >VM is mostly created. There is a protocol (PXE Boot) that enables this > >- a PXE server listens and hears that a new server with such-and-such > >Ethernet Address is starting. The PXE server makes it boot like a > >CD-ROM/DVD install, booting from installation media on the network and > >installing.Once that install is down, a custom script may be invoked. > > This script is typically a bash script, because you may not be able to > >count on too much else being installed. However, python/perl are also > >reasonable choices - just be careful that the modules/libraries you are > >using for the script are present.The same PXE protocol is used in > >large on-premises installations (vCenter) and in the cloud > >(AWS/Digital Ocean). We don't care about the PXE server - the point > >is that you can generally run a bash script after your install. > > > >The bash script can bootstrap other services such as puppet, chef, or > >salt, and/or setup keys so that push configuration management tools such > >as ansible can reach the server. The bash script may even be smart > >enough to do all of the setup you need, depending on what other servers > >you need to configure. Smart bash scripts are good for a small company, > >but for large setups, I'd use puppet, chef, salt, and/or ansible. > > > >What I tend to do is to deploy things in such a way that puppet > >(because it is what we use here) can setup things so that a "solradm" > >account can setup everything else, and solr and zookeeper are running as a > >"solrapp" > >user using puppet.Then, my continuous integration server, which is > >Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, > >BuildBot), installs solr as "solradm" and sets it up to run as "solrapp". > > > >I am not a systems administrator, and I'm not really in "DevOps", my > >job is to be above all of that and do "systems architecture" which I > >am lucky still involves coding both in system administration and applications > >development. So, that's my 2 cents. > > > >Dan Davis, Systems/Applications Architect (Contractor), Office of > >Computer and Communications Systems, National Library of Medicine, > >NIH > > > >-Original Message- > >From: Susheel Kumar [mailto:susheel2...@gmail.com] > >Sent: Tuesday, October 20, 2015 9:19 AM > >To: solr-user@lucene.apache.org > >Subject: DevOps question : auto deployment/setup of Solr & Zookeeper > >on medium-large clusters > > > >Hello, > > > >Resending to see opinion from Dev-Ops perspective on the tools for > >installing/deployment of Solr & ZK on large no of machines and > >maintaining them. I hav
Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
We are using aws, and standardized deployments using Chef. As Jeff points out below, Exhibitor is a good tool to deploy with Zookeeper. We’ve had very good luck with it. On 10/20/15, 7:59 PM, "Jeff Wartes" <jwar...@whitepages.com> wrote: > >If you’re using AWS, there’s this: >https://github.com/LucidWorks/solr-scale-tk >If you’re using chef, there’s this: >https://github.com/vkhatri/chef-solrcloud > >(There are several other chef cookbooks for Solr out there, but this is >the only one I’m aware of that supports Solr 5.3.) > >For ZK, I’m less familiar, but if you’re using chef there’s this: >https://github.com/SimpleFinance/chef-zookeeper >And this might be handy to know about too: >https://github.com/Netflix/exhibitor/wiki > > >On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov> >wrote: > >>Waste of money in my opinion. I would point you towards other tools - >>bash scripts and free configuration managers such as puppet, chef, salt, >>or ansible.Depending on what development you are doing, you may want >>a continuous integration environment. For a small company starting out, >>using a free CI, maybe SaaS, is a good choice. A professional version >>such as Bamboo, TeamCity, Jenkins are almost essential in a large >>enterprise if you are doing diverse builds. >> >>When you create a VM, you can generally specify a script to run after the >>VM is mostly created. There is a protocol (PXE Boot) that enables this >>- a PXE server listens and hears that a new server with such-and-such >>Ethernet Address is starting. The PXE server makes it boot like a >>CD-ROM/DVD install, booting from installation media on the network and >>installing.Once that install is down, a custom script may be invoked. >> This script is typically a bash script, because you may not be able to >>count on too much else being installed. However, python/perl are also >>reasonable choices - just be careful that the modules/libraries you are >>using for the script are present.The same PXE protocol is used in >>large on-premises installations (vCenter) and in the cloud (AWS/Digital >>Ocean). We don't care about the PXE server - the point is that you can >>generally run a bash script after your install. >> >>The bash script can bootstrap other services such as puppet, chef, or >>salt, and/or setup keys so that push configuration management tools such >>as ansible can reach the server. The bash script may even be smart >>enough to do all of the setup you need, depending on what other servers >>you need to configure. Smart bash scripts are good for a small company, >>but for large setups, I'd use puppet, chef, salt, and/or ansible. >> >>What I tend to do is to deploy things in such a way that puppet (because >>it is what we use here) can setup things so that a "solradm" account can >>setup everything else, and solr and zookeeper are running as a "solrapp" >>user using puppet.Then, my continuous integration server, which is >>Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, >>BuildBot), installs solr as "solradm" and sets it up to run as "solrapp". >> >>I am not a systems administrator, and I'm not really in "DevOps", my job >>is to be above all of that and do "systems architecture" which I am lucky >>still involves coding both in system administration and applications >>development. So, that's my 2 cents. >> >>Dan Davis, Systems/Applications Architect (Contractor), >>Office of Computer and Communications Systems, >>National Library of Medicine, NIH >> >>-Original Message- >>From: Susheel Kumar [mailto:susheel2...@gmail.com] >>Sent: Tuesday, October 20, 2015 9:19 AM >>To: solr-user@lucene.apache.org >>Subject: DevOps question : auto deployment/setup of Solr & Zookeeper on >>medium-large clusters >> >>Hello, >> >>Resending to see opinion from Dev-Ops perspective on the tools for >>installing/deployment of Solr & ZK on large no of machines and >>maintaining them. I have heard Bladelogic or HP OO (commercial tools) >>etc. being used. >>Please share your experience or pros / cons of such tools. >> >>Thanks, >>Susheel >> >>On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumar <susheel2...@gmail.com> >>wrote: >> >>> Hi, >>> >>> I am trying to find the best practises for setting up Solr on new 20+ >>> machines & ZK (5+) and repeating same on other environments. What's >>> the best way to download, extract, setup Solr & ZK in an automated way >>> along with other dependencies like java etc. Among shell scripts or >>> puppet or docker or imaged vm's what is being used & suggested from >>> Dev-Ops perspective. >>> >>> Thanks, >>> Susheel >>> >
Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
Thanks, Davis, Jeff. We are not using AWS. Is there any scripts/framework already developed using puppet available? On Tue, Oct 20, 2015 at 7:59 PM, Jeff Wartes <jwar...@whitepages.com> wrote: > > If you’re using AWS, there’s this: > https://github.com/LucidWorks/solr-scale-tk > If you’re using chef, there’s this: > https://github.com/vkhatri/chef-solrcloud > > (There are several other chef cookbooks for Solr out there, but this is > the only one I’m aware of that supports Solr 5.3.) > > For ZK, I’m less familiar, but if you’re using chef there’s this: > https://github.com/SimpleFinance/chef-zookeeper > And this might be handy to know about too: > https://github.com/Netflix/exhibitor/wiki > > > On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov> > wrote: > > >Waste of money in my opinion. I would point you towards other tools - > >bash scripts and free configuration managers such as puppet, chef, salt, > >or ansible.Depending on what development you are doing, you may want > >a continuous integration environment. For a small company starting out, > >using a free CI, maybe SaaS, is a good choice. A professional version > >such as Bamboo, TeamCity, Jenkins are almost essential in a large > >enterprise if you are doing diverse builds. > > > >When you create a VM, you can generally specify a script to run after the > >VM is mostly created. There is a protocol (PXE Boot) that enables this > >- a PXE server listens and hears that a new server with such-and-such > >Ethernet Address is starting. The PXE server makes it boot like a > >CD-ROM/DVD install, booting from installation media on the network and > >installing.Once that install is down, a custom script may be invoked. > > This script is typically a bash script, because you may not be able to > >count on too much else being installed. However, python/perl are also > >reasonable choices - just be careful that the modules/libraries you are > >using for the script are present.The same PXE protocol is used in > >large on-premises installations (vCenter) and in the cloud (AWS/Digital > >Ocean). We don't care about the PXE server - the point is that you can > >generally run a bash script after your install. > > > >The bash script can bootstrap other services such as puppet, chef, or > >salt, and/or setup keys so that push configuration management tools such > >as ansible can reach the server. The bash script may even be smart > >enough to do all of the setup you need, depending on what other servers > >you need to configure. Smart bash scripts are good for a small company, > >but for large setups, I'd use puppet, chef, salt, and/or ansible. > > > >What I tend to do is to deploy things in such a way that puppet (because > >it is what we use here) can setup things so that a "solradm" account can > >setup everything else, and solr and zookeeper are running as a "solrapp" > >user using puppet.Then, my continuous integration server, which is > >Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, > >BuildBot), installs solr as "solradm" and sets it up to run as "solrapp". > > > >I am not a systems administrator, and I'm not really in "DevOps", my job > >is to be above all of that and do "systems architecture" which I am lucky > >still involves coding both in system administration and applications > >development. So, that's my 2 cents. > > > >Dan Davis, Systems/Applications Architect (Contractor), > >Office of Computer and Communications Systems, > >National Library of Medicine, NIH > > > >-Original Message- > >From: Susheel Kumar [mailto:susheel2...@gmail.com] > >Sent: Tuesday, October 20, 2015 9:19 AM > >To: solr-user@lucene.apache.org > >Subject: DevOps question : auto deployment/setup of Solr & Zookeeper on > >medium-large clusters > > > >Hello, > > > >Resending to see opinion from Dev-Ops perspective on the tools for > >installing/deployment of Solr & ZK on large no of machines and > >maintaining them. I have heard Bladelogic or HP OO (commercial tools) > >etc. being used. > >Please share your experience or pros / cons of such tools. > > > >Thanks, > >Susheel > > > >On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumar <susheel2...@gmail.com> > >wrote: > > > >> Hi, > >> > >> I am trying to find the best practises for setting up Solr on new 20+ > >> machines & ZK (5+) and repeating same on other environments. What's > >> the best way to download, extract, setup Solr & ZK in an automated way > >> along with other dependencies like java etc. Among shell scripts or > >> puppet or docker or imaged vm's what is being used & suggested from > >> Dev-Ops perspective. > >> > >> Thanks, > >> Susheel > >> > >
Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
If you’re using AWS, there’s this: https://github.com/LucidWorks/solr-scale-tk If you’re using chef, there’s this: https://github.com/vkhatri/chef-solrcloud (There are several other chef cookbooks for Solr out there, but this is the only one I’m aware of that supports Solr 5.3.) For ZK, I’m less familiar, but if you’re using chef there’s this: https://github.com/SimpleFinance/chef-zookeeper And this might be handy to know about too: https://github.com/Netflix/exhibitor/wiki On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov> wrote: >Waste of money in my opinion. I would point you towards other tools - >bash scripts and free configuration managers such as puppet, chef, salt, >or ansible.Depending on what development you are doing, you may want >a continuous integration environment. For a small company starting out, >using a free CI, maybe SaaS, is a good choice. A professional version >such as Bamboo, TeamCity, Jenkins are almost essential in a large >enterprise if you are doing diverse builds. > >When you create a VM, you can generally specify a script to run after the >VM is mostly created. There is a protocol (PXE Boot) that enables this >- a PXE server listens and hears that a new server with such-and-such >Ethernet Address is starting. The PXE server makes it boot like a >CD-ROM/DVD install, booting from installation media on the network and >installing.Once that install is down, a custom script may be invoked. > This script is typically a bash script, because you may not be able to >count on too much else being installed. However, python/perl are also >reasonable choices - just be careful that the modules/libraries you are >using for the script are present.The same PXE protocol is used in >large on-premises installations (vCenter) and in the cloud (AWS/Digital >Ocean). We don't care about the PXE server - the point is that you can >generally run a bash script after your install. > >The bash script can bootstrap other services such as puppet, chef, or >salt, and/or setup keys so that push configuration management tools such >as ansible can reach the server. The bash script may even be smart >enough to do all of the setup you need, depending on what other servers >you need to configure. Smart bash scripts are good for a small company, >but for large setups, I'd use puppet, chef, salt, and/or ansible. > >What I tend to do is to deploy things in such a way that puppet (because >it is what we use here) can setup things so that a "solradm" account can >setup everything else, and solr and zookeeper are running as a "solrapp" >user using puppet.Then, my continuous integration server, which is >Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, >BuildBot), installs solr as "solradm" and sets it up to run as "solrapp". > >I am not a systems administrator, and I'm not really in "DevOps", my job >is to be above all of that and do "systems architecture" which I am lucky >still involves coding both in system administration and applications >development. So, that's my 2 cents. > >Dan Davis, Systems/Applications Architect (Contractor), >Office of Computer and Communications Systems, >National Library of Medicine, NIH > >-Original Message- >From: Susheel Kumar [mailto:susheel2...@gmail.com] >Sent: Tuesday, October 20, 2015 9:19 AM >To: solr-user@lucene.apache.org >Subject: DevOps question : auto deployment/setup of Solr & Zookeeper on >medium-large clusters > >Hello, > >Resending to see opinion from Dev-Ops perspective on the tools for >installing/deployment of Solr & ZK on large no of machines and >maintaining them. I have heard Bladelogic or HP OO (commercial tools) >etc. being used. >Please share your experience or pros / cons of such tools. > >Thanks, >Susheel > >On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumar <susheel2...@gmail.com> >wrote: > >> Hi, >> >> I am trying to find the best practises for setting up Solr on new 20+ >> machines & ZK (5+) and repeating same on other environments. What's >> the best way to download, extract, setup Solr & ZK in an automated way >> along with other dependencies like java etc. Among shell scripts or >> puppet or docker or imaged vm's what is being used & suggested from >> Dev-Ops perspective. >> >> Thanks, >> Susheel >>
DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
Hello, Resending to see opinion from Dev-Ops perspective on the tools for installing/deployment of Solr & ZK on large no of machines and maintaining them. I have heard Bladelogic or HP OO (commercial tools) etc. being used. Please share your experience or pros / cons of such tools. Thanks, Susheel On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumarwrote: > Hi, > > I am trying to find the best practises for setting up Solr on new 20+ > machines & ZK (5+) and repeating same on other environments. What's the > best way to download, extract, setup Solr & ZK in an automated way along > with other dependencies like java etc. Among shell scripts or puppet or > docker or imaged vm's what is being used & suggested from Dev-Ops > perspective. > > Thanks, > Susheel >
RE: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters
Waste of money in my opinion. I would point you towards other tools - bash scripts and free configuration managers such as puppet, chef, salt, or ansible. Depending on what development you are doing, you may want a continuous integration environment. For a small company starting out, using a free CI, maybe SaaS, is a good choice. A professional version such as Bamboo, TeamCity, Jenkins are almost essential in a large enterprise if you are doing diverse builds. When you create a VM, you can generally specify a script to run after the VM is mostly created. There is a protocol (PXE Boot) that enables this - a PXE server listens and hears that a new server with such-and-such Ethernet Address is starting. The PXE server makes it boot like a CD-ROM/DVD install, booting from installation media on the network and installing.Once that install is down, a custom script may be invoked. This script is typically a bash script, because you may not be able to count on too much else being installed. However, python/perl are also reasonable choices - just be careful that the modules/libraries you are using for the script are present.The same PXE protocol is used in large on-premises installations (vCenter) and in the cloud (AWS/Digital Ocean). We don't care about the PXE server - the point is that you can generally run a bash script after your install. The bash script can bootstrap other services such as puppet, chef, or salt, and/or setup keys so that push configuration management tools such as ansible can reach the server. The bash script may even be smart enough to do all of the setup you need, depending on what other servers you need to configure. Smart bash scripts are good for a small company, but for large setups, I'd use puppet, chef, salt, and/or ansible. What I tend to do is to deploy things in such a way that puppet (because it is what we use here) can setup things so that a "solradm" account can setup everything else, and solr and zookeeper are running as a "solrapp" user using puppet.Then, my continuous integration server, which is Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity, BuildBot), installs solr as "solradm" and sets it up to run as "solrapp". I am not a systems administrator, and I'm not really in "DevOps", my job is to be above all of that and do "systems architecture" which I am lucky still involves coding both in system administration and applications development. So, that's my 2 cents. Dan Davis, Systems/Applications Architect (Contractor), Office of Computer and Communications Systems, National Library of Medicine, NIH -Original Message- From: Susheel Kumar [mailto:susheel2...@gmail.com] Sent: Tuesday, October 20, 2015 9:19 AM To: solr-user@lucene.apache.org Subject: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters Hello, Resending to see opinion from Dev-Ops perspective on the tools for installing/deployment of Solr & ZK on large no of machines and maintaining them. I have heard Bladelogic or HP OO (commercial tools) etc. being used. Please share your experience or pros / cons of such tools. Thanks, Susheel On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumar <susheel2...@gmail.com> wrote: > Hi, > > I am trying to find the best practises for setting up Solr on new 20+ > machines & ZK (5+) and repeating same on other environments. What's > the best way to download, extract, setup Solr & ZK in an automated way > along with other dependencies like java etc. Among shell scripts or > puppet or docker or imaged vm's what is being used & suggested from > Dev-Ops perspective. > > Thanks, > Susheel >