[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392751#comment-16392751
]
Steve Loughran commented on HIVE-18861:
---
thx for your help nurturing this in.
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch,
> HIVE-18861.patch, HIVE-18861.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391555#comment-16391555
]
Steve Loughran commented on HIVE-18861:
---
I don't see these tests being related' there's nothing druid or aws related
directly in the stacks. e.g. TestSequenceFileHCatStorer.testWriteChar
{code}
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 6017:
JobID: job_local1096640948_0048 Reason: ExitCodeException exitCode=1: chmod:
cannot access
‘/tmp/hadoop/mapred/staging/hiveptest1096640948/.staging/job_local1096640948_0048/job.split’:
No such file or directory
{code}
However, without experience running the Hive tests, I could be utterly wrong.
What to do now?
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch,
> HIVE-18861.patch, HIVE-18861.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390604#comment-16390604
]
Hive QA commented on HIVE-18861:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913465/HIVE-18861.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12952 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390552#comment-16390552
]
Hive QA commented on HIVE-18861:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9542/dev-support/hive-personality.sh
|
| git revision | master / 073dc88 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9542/yetus/patch-asflicense-problems.txt
|
| modules | C: druid-handler U: druid-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9542/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch,
> HIVE-18861.patch, HIVE-18861.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390366#comment-16390366
]
Steve Loughran commented on HIVE-18861:
---
Not seeing any updates after 9h. Cancelling and reattaching
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch,
> HIVE-18861.patch, HIVE-18861.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388972#comment-16388972
]
Ashutosh Chauhan commented on HIVE-18861:
-
Patch needs to be named according to:
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388104#comment-16388104
]
Ashutosh Chauhan commented on HIVE-18861:
-
Our infra has issues. Usually reuploading a patch triggers a QA run. Can you
please try that?
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387697#comment-16387697
]
Steve Loughran commented on HIVE-18861:
---
[~ashutoshc]: I dont see jira running tests here...is there something I need to
do?
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386345#comment-16386345
]
Steve Loughran commented on HIVE-18861:
---
thanks! If this goes it in, it will be first contribution to hive...removing my
own code from the classpath :)
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386281#comment-16386281
]
Ashutosh Chauhan commented on HIVE-18861:
-
+1 pending tests
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386228#comment-16386228
]
Steve Loughran commented on HIVE-18861:
---
Dependencies before the patch when built against hadoop branch-3.1
{code}
[INFO] | +- io.druid.extensions:druid-hdfs-storage:jar:0.11.0:compile
[INFO] | | +- (org.apache.hadoop:hadoop-client:jar:3.1.0-SNAPSHOT:compile -
version managed from 2.7.3; omitted for duplicate)
[INFO] | | +- org.apache.hadoop:hadoop-aws:jar:2.7.3:compile
[INFO] | | | \- (org.apache.hadoop:hadoop-common:jar:3.1.0-SNAPSHOT:compile
- version managed from 2.7.3; omitted for duplicate)
[INFO] | | \- com.amazonaws:aws-java-sdk-s3:jar:1.10.77:compile
[INFO] | | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.77:compile
[INFO] | | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile -
omitted for duplicate)
[INFO] | | \- com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile
[INFO] | |+- (commons-logging:commons-logging:jar:1.1.3:compile -
omitted for conflict with 1.0.4)
[INFO] | |+- (org.apache.httpcomponents:httpclient:jar:4.5.2:compile -
version managed from 4.3.6; omitted for duplicate)
[INFO] | |+-
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
{code}
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386231#comment-16386231
]
Steve Loughran commented on HIVE-18861:
---
And after
{code}
[INFO] | +- io.druid.extensions:druid-hdfs-storage:jar:0.11.0:compile
[INFO] | | \- (org.apache.hadoop:hadoop-client:jar:3.1.0-SNAPSHOT:compile -
version managed from 2.7.3; omitted for duplicate)
{code}
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386225#comment-16386225
]
Steve Loughran commented on HIVE-18861:
---
Patch 001: pulls the hadoop JAR and the aws sdk version, which is imported in
parallel.
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
