[kudu-CR] thirdparty: reorganize tree

[Adar Dembo (Code Review)]

Adar Dembo has submitted this change and it was merged.

Change subject: thirdparty: reorganize tree
..


thirdparty: reorganize tree

This patch changes the organization of the thirdparty tree. The new layout
looks like this:
- installed: _all_ installed dependencies, with 'common',
 'uninstrumented', and 'tsan' subdirectories.
- build: build directories for all dependencies.
- src: source directories for all dependencies.

Additionally, the patch changes the build logic for each dependency so that
its build output is fully isolated from its source directory, and from other
build output (if the dependency is built multiple times).

Why do this?
1. Build isolation simplifies building dependencies multiple times (i.e. for
   different sanitizers) and makes it much safer.
2. It also means cleaning up build output doesn't mean redownloading all of
   the sources (i.e. no need to 'git clean -xdf thirdparty').
3. The grouping of all installed locations under the shared 'installed'
   subdirectory makes it a tad easier to blow it all away.
4. It also eases the transition for the remaining LLVM and thirdparty
   patches, as the conflicting build output will be copied to a different
   set of directories.
5. It slims down thirdparty/.gitignore; adding a new dependency no longer
   requires updating .gitignore.

Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Reviewed-on: http://gerrit.cloudera.org:8080/4550
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert 
---
M .ycm_extra_conf.py
M CMakeLists.txt
M README.adoc
M build-support/dist_test.py
M build-support/jenkins/build-and-test.sh
M build-support/lint.sh
M build-support/run_dist_test.py
M docs/installation.adoc
M docs/support/scripts/make_site.sh
M java/kudu-client/pom.xml
M src/kudu/client/client_samples-test.sh
M src/kudu/scripts/benchmarks.sh
M src/kudu/scripts/tpch.sh
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
18 files changed, 334 insertions(+), 217 deletions(-)

Approvals:
  Dan Burkert: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] thirdparty: reorganize tree

[Dan Burkert (Code Review)]

Dan Burkert has posted comments on this change.

Change subject: thirdparty: reorganize tree
..


Patch Set 2: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] thirdparty: reorganize tree

[Adar Dembo (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/4550

to look at the new patch set (#2).

Change subject: thirdparty: reorganize tree
..

thirdparty: reorganize tree

This patch changes the organization of the thirdparty tree. The new layout
looks like this:
- installed: _all_ installed dependencies, with 'common',
 'uninstrumented', and 'tsan' subdirectories.
- build: build directories for all dependencies.
- src: source directories for all dependencies.

Additionally, the patch changes the build logic for each dependency so that
its build output is fully isolated from its source directory, and from other
build output (if the dependency is built multiple times).

Why do this?
1. Build isolation simplifies building dependencies multiple times (i.e. for
   different sanitizers) and makes it much safer.
2. It also means cleaning up build output doesn't mean redownloading all of
   the sources (i.e. no need to 'git clean -xdf thirdparty').
3. The grouping of all installed locations under the shared 'installed'
   subdirectory makes it a tad easier to blow it all away.
4. It also eases the transition for the remaining LLVM and thirdparty
   patches, as the conflicting build output will be copied to a different
   set of directories.
5. It slims down thirdparty/.gitignore; adding a new dependency no longer
   requires updating .gitignore.

Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
---
M .ycm_extra_conf.py
M CMakeLists.txt
M README.adoc
M build-support/dist_test.py
M build-support/jenkins/build-and-test.sh
M build-support/lint.sh
M build-support/run_dist_test.py
M docs/installation.adoc
M docs/support/scripts/make_site.sh
M java/kudu-client/pom.xml
M src/kudu/client/client_samples-test.sh
M src/kudu/scripts/benchmarks.sh
M src/kudu/scripts/tpch.sh
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
18 files changed, 334 insertions(+), 217 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/50/4550/2
-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] thirdparty: reorganize tree

[Adar Dembo (Code Review)]

Adar Dembo has posted comments on this change.

Change subject: thirdparty: reorganize tree
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/4550/1/thirdparty/build-definitions.sh
File thirdparty/build-definitions.sh:

Line 1: #!/bin/bash
> Is this necessary?
Every other shell script in the Kudu repo uses it, so I figure it's best to be 
consistent. Plus, I might be introducing bash-isms with these changes, so 
better safe than sorry.


Line 43: restore_env() {
> Should this now include MODE_SUFFIX?  I think that might be good so that we
Ahh, didn't realize PREFIX was also saved/restored. Yeah, I'll add MODE_SUFFIX 
too.


Line 439:   rsync -a $BOOST_SOURCE/boost $PREFIX/include
> Not your change, but this rsync incantation is using different flags than t
You referring to the lack of -v, or --delete?

The former is because boost tree is stupendously huge, so logging every file 
(-v) is kind of a drag.

But --delete should probably go in there, otherwise we may not notice header 
removals during upgrades. I'll add it.


-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] thirdparty: reorganize tree

[Dan Burkert (Code Review)]

Dan Burkert has posted comments on this change.

Change subject: thirdparty: reorganize tree
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/4550/1/thirdparty/build-definitions.sh
File thirdparty/build-definitions.sh:

Line 1: #!/bin/bash
Is this necessary?


Line 43: restore_env() {
Should this now include MODE_SUFFIX?  I think that might be good so that we 
will fail fast if we try to use it without setting it.

Although now that I'm looking, I don't think we set -u, so it may not make a 
difference.


Line 439:   rsync -a $BOOST_SOURCE/boost $PREFIX/include
Not your change, but this rsync incantation is using different flags than the 
others, is that intentional?


-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] thirdparty: reorganize tree

[Adar Dembo (Code Review)]

Hello Dan Burkert, Todd Lipcon,

I'd like you to do a code review.  Please visit

http://gerrit.cloudera.org:8080/4550

to review the following change.

Change subject: thirdparty: reorganize tree
..

thirdparty: reorganize tree

This patch changes the organization of the thirdparty tree. The new layout
looks like this:
- installed: _all_ installed dependencies, with 'common',
 'uninstrumented', and 'tsan' subdirectories.
- build: build directories for all dependencies.
- src: source directories for all dependencies.

Additionally, the patch changes the build logic for each dependency so that
its build output is fully isolated from its source directory, and from other
build output (if the dependency is built multiple times).

Why do this?
1. Build isolation simplifies building dependencies multiple times (i.e. for
   different sanitizers) and makes it much safer.
2. It also means cleaning up build output doesn't mean redownloading all of
   the sources (i.e. no need to 'git clean -xdf thirdparty').
3. The grouping of all installed locations under the shared 'installed'
   subdirectory makes it a tad easier to blow it all away.
4. It also eases the transition for the remaining LLVM and thirdparty
   patches, as the conflicting build output will be copied to a different
   set of directories.
5. It slims down thirdparty/.gitignore; adding a new dependency no longer
   requires updating .gitignore.

Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
---
M .ycm_extra_conf.py
M CMakeLists.txt
M README.adoc
M build-support/dist_test.py
M build-support/jenkins/build-and-test.sh
M build-support/lint.sh
M build-support/run_dist_test.py
M docs/installation.adoc
M docs/support/scripts/make_site.sh
M java/kudu-client/pom.xml
M src/kudu/client/client_samples-test.sh
M src/kudu/scripts/benchmarks.sh
M src/kudu/scripts/tpch.sh
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
18 files changed, 331 insertions(+), 217 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/50/4550/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I06aa75ab5e81f2563986244950e9dcda06c2d383
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: Todd Lipcon