Re: Building a Solr cluster with Maven

[Greg Pendlebury]

 the artifacts that
> are published already. Publishing the whole zip would be a start,
> downstream builds could add logic to resolve it, explode, tweak, and
> re-publish. The maintain the strict separation from the war, it might be
> helpful to have a lib or "plugin-ins" folder in the zip that is by default
> loaded to the classpath as an extension point for users who are re-building
> the package?
>
> -Tim
>
> From: dev@lucene.apache.org At: 10/18/16 09:52:42
> To: dev@lucene.apache.org
> Subject: Re: Building a Solr cluster with Maven
>
> My team has modified the ant scripts to publish all the jars/poms and the
> zip to our local artifactory when we run our build. We have another project
> which pulls down all of these dependencies including the zip to build our
> actual solr deploy and a maven assembly which unpacks the zip file and
> extracts all of the webapp for our real distribution.
>
> I haven't upstreamed the changes for the ant tasks thinking there wouldn't
> be too much interest in that, but I could put together a patch if there is.
>
> The changes do the following:
>
> - Packages the zip along with the parent pom if a flag is set
> - Allows changing group which the poms are published to. For example
> instead of org.apache you can push it as com.xxx to avoid shadowing
> conflicts in your local repository.
>
> On Tue, Oct 18, 2016 at 8:42 AM David Smiley <david.w.smi...@gmail.com>
> wrote:
>
>> Thanks for bringing this up, Greg.  I too have felt the pain of this in
>> the move away from a WAR file in a project or two.  In one of the projects
>> that comes to mind, we built scripts that re-constituted a Solr
>> distribution from artifacts in Maven. For anything that wasn't in Maven
>> (e.g. the admin UI pages, Jetty configs), we checked it into source
>> control.  In hind sight... the simplicity of what you list as (1) -- check
>> the distro zip into a Maven repo local to the organization sounds better...
>> but I may be forgetting requirements that led us not to do this.  I look
>> forward to that zip shrinking once the docs are gone.  Another option,
>> depending on one's needs, is to pursue Docker, which I've lately become a
>> huge fan of.  I think Docker is particularly great for integration tests.
>> Does the scenario you wish to use the assets for relate to testing or some
>> other use-case?
>>
>> ~ David
>>
>>
>>
>> On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury <
>> greg.pendleb...@gmail.com> wrote:
>>
>> Are there any developers with a current working maven build for a
>> downstream Solr installation? ie. Not a build for Solr itself, but a build
>> that brings in the core Solr server plus local plugins, third party plugins
>> etc?
>>
>> I am in the process of updating one of our old builds (it builds both the
>> application and various shard instances) and have hit a stumbling block in
>> sourcing the dashboard static assets (everything under /webapp/web in
>> Solr's source).
>>
>> Prior to the move away from being a webapp I could get them by exploding
>> the war from Maven Central.
>>
>> In our very first foray into 5.x we had a local custom build to patch
>> SOLR-2649. We avoided solving this problem then by pushing the webapp into
>> our local Nexus as part of that build... but that wasn't a very good long
>> term choice.
>>
>> So now I'm trying to work out the best long term approach to take here.
>> Ideas so far:
>>
>>1. Manually download the required zip and add it into our Nexus
>>repository as a 3rd party artifact. Maven can source and extract anything
>>it needs from here. This is where I'm currently leaning for simplicity, 
>> but
>>the manual step required is annoying. It does have the advantage of 
>> causing
>>a build failure straight away when a version upgrade occurs, prompting the
>>developer to look into why.
>>2. Move a copy of the static assets for the dashboard into our
>>project and deploy them ourselves. This has the advantage of aligning our
>>approach with the resources we already maintain in the project (like
>>core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
>>worried that it is really fragile and developers will miss it during a
>>version upgrade, resulting in the dashboard creeping out-of-date and
>>(worse) introducing subtle bugs because of a version mismatch between the
>>UI and the underlying server code.
>>3. I'd like to think a long term approach would be for the core Solr
>>build to ship a JAR (or any othe

Re: Building a Solr cluster with Maven

[Timothy Rodriguez (BLOOMBERG/ 120 PARK)]

That'd be a helpful step.  I think it'd be even better if there was a way to 
generate somewhat customized versions of solr from the artifacts that are 
published already.  Publishing the whole zip would be a start, downstream 
builds could add logic to resolve it, explode, tweak, and re-publish.  The 
maintain the strict separation from the war, it might be helpful to have a lib 
or "plugin-ins" folder in the zip that is by default loaded to the classpath as 
an extension point for users who are re-building the package?

-Tim

From: dev@lucene.apache.org At: 10/18/16 09:52:42
To: dev@lucene.apache.org
Subject: Re: Building a Solr cluster with Maven

My team has modified the ant scripts to publish all the jars/poms and the zip 
to our local artifactory when we run our build. We have another project which 
pulls down all of these dependencies including the zip to build our actual solr 
deploy and a maven assembly which unpacks the zip file and extracts all of the 
webapp for our real distribution. 

I haven't upstreamed the changes for the ant tasks thinking there wouldn't be 
too much interest in that, but I could put together a patch if there is.

The changes do the following:

- Packages the zip along with the parent pom if a flag is set
- Allows changing group which the poms are published to. For example instead of 
org.apache you can push it as com.xxx to avoid shadowing conflicts in your 
local repository.

On Tue, Oct 18, 2016 at 8:42 AM David Smiley <david.w.smi...@gmail.com> wrote:

Thanks for bringing this up, Greg.  I too have felt the pain of this in the 
move away from a WAR file in a project or two.  In one of the projects that 
comes to mind, we built scripts that re-constituted a Solr distribution from 
artifacts in Maven. For anything that wasn't in Maven (e.g. the admin UI pages, 
Jetty configs), we checked it into source control.  In hind sight... the 
simplicity of what you list as (1) -- check the distro zip into a Maven repo 
local to the organization sounds better... but I may be forgetting requirements 
that led us not to do this.  I look forward to that zip shrinking once the docs 
are gone.  Another option, depending on one's needs, is to pursue Docker, which 
I've lately become a huge fan of.  I think Docker is particularly great for 
integration tests.  Does the scenario you wish to use the assets for relate to 
testing or some other use-case?

~ David


On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury <greg.pendleb...@gmail.com> 
wrote:

Are there any developers with a current working maven build for a downstream 
Solr installation? ie. Not a build for Solr itself, but a build that brings in 
the core Solr server plus local plugins, third party plugins etc?

I am in the process of updating one of our old builds (it builds both the 
application and various shard instances) and have hit a stumbling block in 
sourcing the dashboard static assets (everything under /webapp/web in Solr's 
source).

Prior to the move away from being a webapp I could get them by exploding the 
war from Maven Central.

In our very first foray into 5.x we had a local custom build to patch 
SOLR-2649. We avoided solving this problem then by pushing the webapp into our 
local Nexus as part of that build... but that wasn't a very good long term 
choice.

So now I'm trying to work out the best long term approach to take here. Ideas 
so far:

  1)Manually download the required zip and add it into our Nexus repository as 
a 3rd party artifact. Maven can source and extract anything it needs from here. 
This is where I'm currently leaning for simplicity, but the manual step 
required is annoying. It does have the advantage of causing a build failure 
straight away when a version upgrade occurs, prompting the developer to look 
into why.

  2)Move a copy of the static assets for the dashboard into our project and 
deploy them ourselves. This has the advantage of aligning our approach with the 
resources we already maintain in the project (like core.properties, schema.xml, 
solrconfig.xml, logging etc.). But I am worried that it is really fragile and 
developers will miss it during a version upgrade, resulting in the dashboard 
creeping out-of-date and (worse) introducing subtle bugs because of a version 
mismatch between the UI and the underlying server code.
  3)I'd like to think a long term approach would be for the core Solr build to 
ship a JAR (or any other assembly) to Maven Central like 'solr-dashboard'... 
but I'm not sure how that aligns with the move away from Solr being considered 
a webapp. It seems a shame that all of the Java code ends up in Maven central, 
but the web layer dead-ends in the ant build.
I might be missing something really obvious and there is already a way to do 
this. Is there some other distribution of the dashboard statics? Other than the 
downloadable zip that is.

Ta,
Greg

-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedI

Re: Building a Solr cluster with Maven

[Keith L]

My team has modified the ant scripts to publish all the jars/poms and the
zip to our local artifactory when we run our build. We have another project
which pulls down all of these dependencies including the zip to build our
actual solr deploy and a maven assembly which unpacks the zip file and
extracts all of the webapp for our real distribution.

I haven't upstreamed the changes for the ant tasks thinking there wouldn't
be too much interest in that, but I could put together a patch if there is.

The changes do the following:

- Packages the zip along with the parent pom if a flag is set
- Allows changing group which the poms are published to. For example
instead of org.apache you can push it as com.xxx to avoid shadowing
conflicts in your local repository.

On Tue, Oct 18, 2016 at 8:42 AM David Smiley 
wrote:

> Thanks for bringing this up, Greg.  I too have felt the pain of this in
> the move away from a WAR file in a project or two.  In one of the projects
> that comes to mind, we built scripts that re-constituted a Solr
> distribution from artifacts in Maven. For anything that wasn't in Maven
> (e.g. the admin UI pages, Jetty configs), we checked it into source
> control.  In hind sight... the simplicity of what you list as (1) -- check
> the distro zip into a Maven repo local to the organization sounds better...
> but I may be forgetting requirements that led us not to do this.  I look
> forward to that zip shrinking once the docs are gone.  Another option,
> depending on one's needs, is to pursue Docker, which I've lately become a
> huge fan of.  I think Docker is particularly great for integration tests.
> Does the scenario you wish to use the assets for relate to testing or some
> other use-case?
>
> ~ David
>
>
>
> On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury 
> wrote:
>
> Are there any developers with a current working maven build for a
> downstream Solr installation? ie. Not a build for Solr itself, but a build
> that brings in the core Solr server plus local plugins, third party plugins
> etc?
>
> I am in the process of updating one of our old builds (it builds both the
> application and various shard instances) and have hit a stumbling block in
> sourcing the dashboard static assets (everything under /webapp/web in
> Solr's source).
>
> Prior to the move away from being a webapp I could get them by exploding
> the war from Maven Central.
>
> In our very first foray into 5.x we had a local custom build to patch
> SOLR-2649. We avoided solving this problem then by pushing the webapp into
> our local Nexus as part of that build... but that wasn't a very good long
> term choice.
>
> So now I'm trying to work out the best long term approach to take here.
> Ideas so far:
>
>1. Manually download the required zip and add it into our Nexus
>repository as a 3rd party artifact. Maven can source and extract anything
>it needs from here. This is where I'm currently leaning for simplicity, but
>the manual step required is annoying. It does have the advantage of causing
>a build failure straight away when a version upgrade occurs, prompting the
>developer to look into why.
>2. Move a copy of the static assets for the dashboard into our project
>and deploy them ourselves. This has the advantage of aligning our approach
>with the resources we already maintain in the project (like
>core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
>worried that it is really fragile and developers will miss it during a
>version upgrade, resulting in the dashboard creeping out-of-date and
>(worse) introducing subtle bugs because of a version mismatch between the
>UI and the underlying server code.
>3. I'd like to think a long term approach would be for the core Solr
>build to ship a JAR (or any other assembly) to Maven Central like
>'solr-dashboard'... but I'm not sure how that aligns with the move away
>from Solr being considered a webapp. It seems a shame that all of the Java
>code ends up in Maven central, but the web layer dead-ends in the ant 
> build.
>
> I might be missing something really obvious and there is already a way to
> do this. Is there some other distribution of the dashboard statics? Other
> than the downloadable zip that is.
>
> Ta,
> Greg
>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>

Re: Building a Solr cluster with Maven

[David Smiley]

Thanks for bringing this up, Greg.  I too have felt the pain of this in the
move away from a WAR file in a project or two.  In one of the projects that
comes to mind, we built scripts that re-constituted a Solr distribution
from artifacts in Maven. For anything that wasn't in Maven (e.g. the admin
UI pages, Jetty configs), we checked it into source control.  In hind
sight... the simplicity of what you list as (1) -- check the distro zip
into a Maven repo local to the organization sounds better... but I may be
forgetting requirements that led us not to do this.  I look forward to that
zip shrinking once the docs are gone.  Another option, depending on one's
needs, is to pursue Docker, which I've lately become a huge fan of.  I
think Docker is particularly great for integration tests.  Does the
scenario you wish to use the assets for relate to testing or some other
use-case?

~ David



On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury 
wrote:

> Are there any developers with a current working maven build for a
> downstream Solr installation? ie. Not a build for Solr itself, but a build
> that brings in the core Solr server plus local plugins, third party plugins
> etc?
>
> I am in the process of updating one of our old builds (it builds both the
> application and various shard instances) and have hit a stumbling block in
> sourcing the dashboard static assets (everything under /webapp/web in
> Solr's source).
>
> Prior to the move away from being a webapp I could get them by exploding
> the war from Maven Central.
>
> In our very first foray into 5.x we had a local custom build to patch
> SOLR-2649. We avoided solving this problem then by pushing the webapp into
> our local Nexus as part of that build... but that wasn't a very good long
> term choice.
>
> So now I'm trying to work out the best long term approach to take here.
> Ideas so far:
>
>1. Manually download the required zip and add it into our Nexus
>repository as a 3rd party artifact. Maven can source and extract anything
>it needs from here. This is where I'm currently leaning for simplicity, but
>the manual step required is annoying. It does have the advantage of causing
>a build failure straight away when a version upgrade occurs, prompting the
>developer to look into why.
>2. Move a copy of the static assets for the dashboard into our project
>and deploy them ourselves. This has the advantage of aligning our approach
>with the resources we already maintain in the project (like
>core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
>worried that it is really fragile and developers will miss it during a
>version upgrade, resulting in the dashboard creeping out-of-date and
>(worse) introducing subtle bugs because of a version mismatch between the
>UI and the underlying server code.
>3. I'd like to think a long term approach would be for the core Solr
>build to ship a JAR (or any other assembly) to Maven Central like
>'solr-dashboard'... but I'm not sure how that aligns with the move away
>from Solr being considered a webapp. It seems a shame that all of the Java
>code ends up in Maven central, but the web layer dead-ends in the ant 
> build.
>
> I might be missing something really obvious and there is already a way to
> do this. Is there some other distribution of the dashboard statics? Other
> than the downloadable zip that is.
>
> Ta,
> Greg
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Building a Solr cluster with Maven

[Greg Pendlebury]

Are there any developers with a current working maven build for a
downstream Solr installation? ie. Not a build for Solr itself, but a build
that brings in the core Solr server plus local plugins, third party plugins
etc?

I am in the process of updating one of our old builds (it builds both the
application and various shard instances) and have hit a stumbling block in
sourcing the dashboard static assets (everything under /webapp/web in
Solr's source).

Prior to the move away from being a webapp I could get them by exploding
the war from Maven Central.

In our very first foray into 5.x we had a local custom build to patch
SOLR-2649. We avoided solving this problem then by pushing the webapp into
our local Nexus as part of that build... but that wasn't a very good long
term choice.

So now I'm trying to work out the best long term approach to take here.
Ideas so far:

   1. Manually download the required zip and add it into our Nexus
   repository as a 3rd party artifact. Maven can source and extract anything
   it needs from here. This is where I'm currently leaning for simplicity, but
   the manual step required is annoying. It does have the advantage of causing
   a build failure straight away when a version upgrade occurs, prompting the
   developer to look into why.
   2. Move a copy of the static assets for the dashboard into our project
   and deploy them ourselves. This has the advantage of aligning our approach
   with the resources we already maintain in the project (like
   core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
   worried that it is really fragile and developers will miss it during a
   version upgrade, resulting in the dashboard creeping out-of-date and
   (worse) introducing subtle bugs because of a version mismatch between the
   UI and the underlying server code.
   3. I'd like to think a long term approach would be for the core Solr
   build to ship a JAR (or any other assembly) to Maven Central like
   'solr-dashboard'... but I'm not sure how that aligns with the move away
   from Solr being considered a webapp. It seems a shame that all of the Java
   code ends up in Maven central, but the web layer dead-ends in the ant build.

I might be missing something really obvious and there is already a way to
do this. Is there some other distribution of the dashboard statics? Other
than the downloadable zip that is.

Ta,
Greg