[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-18 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/4720/2/bin/impala-config.sh
File bin/impala-config.sh:

Line 145: export CDH_MAJOR_VERSION=5
> Bump?
There was a separate cleanup effort to remove CDH references from Apache Impala 
so I think this will be going away soon.


Line 146: export HADOOP_LZO="${HADOOP_LZO-$IMPALA_HOME/../hadoop-lzo}"
> How is this assigned?
This either comes from the environment (e.g. build script or .bashrc), 
otherwise it assumes that the Impala/hadoop-lzo/Impala-lzo repos are all under 
the same directory.


Line 289: export KUDU_JAVA_VERSION=1.0.0-SNAPSHOT
> This shouldn't be configurable too?
I'm not quite sure what the right thing to do here is since the Kudu 
integration is in flux and we're getting Kudu build artifacts in a different 
way than other components, so I wanted to punt for now until that's stabilised.


Line 325: export 
HADOOP_HOME="$CDH_COMPONENTS_HOME/hadoop-${IMPALA_HADOOP_VERSION}/"
> It's a bit confusing with 2 concurrent and related reviews. But it seems to
There's not much point making this more configurable right now since we assume 
these CDH component tarballs are constructed in a special way our test cluster 
expects. I'd rather keep it non-configurable until we've validated that we can 
drop in different build artifacts just to avoid confusion.

I validated that we can build Impala (but not run tests) with these pointing to 
nothing-in-particular.


Line 345: export 
SENTRY_HOME="$CDH_COMPONENTS_HOME/sentry-${IMPALA_SENTRY_VERSION}"
> same as 325
See above.


Line 348: export HIVE_HOME="$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}/"
> same as 325
See above.


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-18 Thread Charlie Helin (Code Review) 
Charlie Helin has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/4720/2/bin/impala-config.sh
File bin/impala-config.sh:

Line 145: export CDH_MAJOR_VERSION=5
Bump?


Line 146: export HADOOP_LZO="${HADOOP_LZO-$IMPALA_HOME/../hadoop-lzo}"
How is this assigned?


Line 289: export KUDU_JAVA_VERSION=1.0.0-SNAPSHOT
This shouldn't be configurable too?


Line 325: export 
HADOOP_HOME="$CDH_COMPONENTS_HOME/hadoop-${IMPALA_HADOOP_VERSION}/"
It's a bit confusing with 2 concurrent and related reviews. But it seems to me 
that HADOOP_HOME also should be configurable, otherwise we would assume that 
the components are laid out in particular fashion.


Line 345: export 
SENTRY_HOME="$CDH_COMPONENTS_HOME/sentry-${IMPALA_SENTRY_VERSION}"
same as 325


Line 348: export HIVE_HOME="$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}/"
same as 325


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Internal Jenkins (Code Review) 
Internal Jenkins has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 3: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Tim Armstrong (Code Review) 
Tim Armstrong has uploaded a new patch set (#3).

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..

IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

This is to help with IMPALA-4277 to make it easier to build against
Hadoop/Hive distributions where the directory layout doesn't exactly
match our current CDH dependencies, or where we may want to
temporarily override a version without making a source change.

Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
---
M bin/impala-config.sh
M buildall.sh
M cmake_modules/FindHDFS.cmake
M common/thrift/CMakeLists.txt
4 files changed, 27 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/4720/3
-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/4720/2/bin/impala-config.sh
File bin/impala-config.sh:

PS2, Line 333: exanded
"expanded"


PS2, Line 335: ${HADOOP_HOME}/share/hadoop/tools/lib/*
Just checking, but does this need updating too?


PS2, Line 404: ${HADOOP_HOME}/lib/native/
There are more here.


PS2, Line 424: ${HADOOP_HOME}/lib/native
And here.


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Charlie Helin (Code Review) 
Charlie Helin has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1: -Code-Review

> Did a grep for HADOOP_HOME, and this might need to be changed too:
 > https://github.com/apache/incubator-impala/blob/75a857c0ceb6a58691ff45f0df8d88586edc8acd/buildall.sh#L357

Yes here it seems like ${HADOOP_HOME}\lib should categorically be substituted 
by ${HADOOP_LIB_DIR}. But regarding this line it also looks like we need to 
parameterize HADOOP LZO?

-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

Did a grep for HADOOP_HOME, and this might need to be changed too:
https://github.com/apache/incubator-impala/blob/75a857c0ceb6a58691ff45f0df8d88586edc8acd/buildall.sh#L357

-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/4720/1/bin/impala-config.sh
File bin/impala-config.sh:

Line 305: export 
IMPALA_LLAMA_MINIKDC_VERSION=${IMPALA_LLAMA_MINIKDC_VERSION:-1.0.0}
> This is actually separate from Llama - a test utility we stole from an old 
I'm also going to post a separate patch to remove Llama (needed to do some 
testing there to make sure it didn't break anything).


PS1, Line 341: $HADOOP_HOME/bin
> If HADOOP_INCLUDE_DIR and HADOOP_LIB_DIR point to a different path than HAD
I think if they were set to incompatible versions it might cause a problem 
since we'd be building against the wrong version of Hadoop (e.g Hadoop 2 API 
versus Hadoop 3 API). I think it's reasonable to assume that it won't be set up 
to point to incompatible versions.


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4720/1/bin/impala-config.sh
File bin/impala-config.sh:

PS1, Line 341: $HADOOP_HOME/bin
If HADOOP_INCLUDE_DIR and HADOOP_LIB_DIR point to a different path than 
HADOOP_HOME/(include)|(lib)/ , wouldn't this cause an issue?


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4720/1/bin/impala-config.sh
File bin/impala-config.sh:

Line 305: export 
IMPALA_LLAMA_MINIKDC_VERSION=${IMPALA_LLAMA_MINIKDC_VERSION:-1.0.0}
> Should this not be removed, in C6 we won't have llama?
This is actually separate from Llama - a test utility we stole from an old 
version of the Llama codebase and have kept around. Not quite sure what we're 
going to do about it but there's a JIRA IMPALA-4292.

It also doesn't affect the final build bits.


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Charlie Helin (Code Review) 
Charlie Helin has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4720/1/bin/impala-config.sh
File bin/impala-config.sh:

Line 305: export 
IMPALA_LLAMA_MINIKDC_VERSION=${IMPALA_LLAMA_MINIKDC_VERSION:-1.0.0}
Should this not be removed, in C6 we won't have llama?


-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Charlie Helin 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-17 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..


Patch Set 1:

Is this necessary for the master branch? Once we move permanently to C6, we 
wouldn't need these optional settings, am I right?

-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

2016-10-13 Thread Tim Armstrong (Code Review) 
Tim Armstrong has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4720

Change subject: IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
..

IMPALA-4277: allow overriding of Hive/Hadoop versions/locations

This is to help with IMPALA-4277 to make it easier to build against
Hadoop/Hive distributions where the directory layout doesn't exactly
match our current CDH dependencies, or where we may want to
temporarily override a version without making a source change.

Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
---
M bin/impala-config.sh
M cmake_modules/FindHDFS.cmake
M common/thrift/CMakeLists.txt
3 files changed, 23 insertions(+), 17 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/4720/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong