[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978721#comment-15978721
]
Daryn Sharp commented on HADOOP-14333:
--
Quiet bothered by hive using private hdfs apis with the added touch of
reflection. It's one thing to hack your own project because when you break it
you fix it, but completely different to hack another project... There should
be a followup jira to remove the method to ensure hive makes the change.
+1 After adding @Deprecated to the method and moving this jira to the hdfs
project.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch,
> HADOOP-14333.003.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978642#comment-15978642
]
Steve Loughran commented on HADOOP-14333:
-
Filesystems are a special case, not just because HDFS adds stuff, but because
there are fundamental differences between different filesystems (case
sensitivity, full posix seek+write, atomic dir rename, o(1) File rename,
consistent world view). You can't declare that something supports this just
through an interface, as (a) it varies at runtime and (b)
{{FSDataOutputStream}} shows how base classes declare functionality which
subclasses end up rejecting by dynamically throwing exceptions.
without getting into the versioning row, note HADOOP-9565 has narrowed down to
some method on FileSystem to probe for features, something like
{code}
boolean hasFeature(Path, String)
{code}
Implementations can switch on the feature string, return true iff the feature
is present and enabled. There's been discussion of a similar problem related to
output stream features, we could do some similar interface here.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch,
> HADOOP-14333.003.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977997#comment-15977997
]
Hadoop QA commented on HADOOP-14333:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HADOOP-14333 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864415/HADOOP-14333.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b90b8e2835ac 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 667966c |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12142/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12142/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12142/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
>
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977968#comment-15977968
]
Yongjun Zhang commented on HADOOP-14333:
Thanks much [~andrew.wang] and [~shahrs87].
Uploaded v3 to drop the unused import.
Hi Rushabh, really appreciate that you will review first thing tomorrow
morning, thanks a lot!
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch,
> HADOOP-14333.003.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977880#comment-15977880
]
Rushabh S Shah commented on HADOOP-14333:
-
Can we please hold off committing patch v2 until tomorrow ?
I work in CST and don't have enough time to review.
Will post review comments first thing tomorrow morning.
Thanks !
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977832#comment-15977832
]
Andrew Wang commented on HADOOP-14333:
--
+1 LGTM, can remove the unused SocketTimeoutException on commit. I'll file Hive
and Hadoop JIRAs for tracking adding and using new public APIs.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977809#comment-15977809
]
Hadoop QA commented on HADOOP-14333:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-client: The
patch generated 1 new + 68 unchanged - 0 fixed = 69 total (was 68) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HADOOP-14333 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864385/HADOOP-14333.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f99e5f0f4ff2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f356f0f |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12139/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12139/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12139/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12139/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977792#comment-15977792
]
Hadoop QA commented on HADOOP-14333:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-client: The
patch generated 1 new + 69 unchanged - 0 fixed = 70 total (was 69) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HADOOP-14333 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864379/HADOOP-14333.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 27e0b2c1993d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f356f0f |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12138/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12138/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12138/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12138/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977761#comment-15977761
]
Yongjun Zhang commented on HADOOP-14333:
Thanks [~andrew.wang].
Discussed with Andrew and he suggested not to do retries since RPC layer is
already doing that. Cleaned up the code and uploaded rev002.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch, HADOOP-14333.002.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977751#comment-15977751
]
Andrew Wang commented on HADOOP-14333:
--
I took a look at Hive's usage of DFSClient. They're digging into our private
bits to get access to the KeyProvider, which they use to get key metadata and
also to create keys, delete keys, etc.
This seems pretty easy to handle in a supportable way by adding a new HdfsAdmin
API for {{getKeyProvider}}. The issue is that the Hive team never told us they
needed access to a KP, so we were unaware of this need.
We're already swallowing this exception in e.g. {{getTrashRoot}} for
compatibility reasons, so the question is really whether we swallow it in
DFSClient vs. in DFS. This doesn't seem worth making a stink over, when the
impact is that existing versions of Hive won't work with 2.8.1 or 3.0.0-alpha3
HDFS clients.
My proposal:
* We move swallowing the exception back to DFSClient for now
* Add whatever APIs Hive requires to HdfsAdmin, get them to use these new APIs
* Clean things up in HDFS as we want
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977749#comment-15977749
]
Yongjun Zhang commented on HADOOP-14333:
Thanks for the discussion here guys.
I just posted a patch for reference. I included some retries for socket time
out. Wonder if we can consider this as a temporary solution.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14333.001.patch
>
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977691#comment-15977691
]
Allen Wittenauer commented on HADOOP-14333:
---
bq. But I think Transparent encryption is only supported by HDFS Filesystem
(DistributedFileSystem). Thats why it's not a part of FileSystem api.
This argument never ever works out in our favor. Just because Hadoop code
doesn't implement feature X for file system Y, doesn't mean file system Y
doesn't actually have feature X. Case in point: quotas. The only public APIs
are in DistributedFileSystem because people (wrongly) thought that
directory-based quotas are unique to Hadoop. As a result, we now have to
publish DFS JavaDocs publicly and it still leaves the gap for other file
systems that also implement this feature.
This really needs to get fixed before it's too late.
bq. The hive code that is broken is listed in the description.
That doesn't answer my question. This exception was added in unreleased
versions of Hadoop. So I'm not sure how this blocks Hive given they can push
out a new release before or immediately after these versions of Hadoop get out.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977642#comment-15977642
]
Yongjun Zhang commented on HADOOP-14333:
Thanks [~aw] and [~shahrs87].
Agree that Hive project should have raised the issue before using it.
{quote}
unblock from what, exactly?
{quote}
The hive code that is broken is listed in the description. I was trying to
propose that, as a temporary solution, we can let isHDFSEncryptionEnabled not
to throw. Then we can coordinate a change between hadoop and hive to fix
together.
Thanks Rushabh for the sample code. I think in the case of SocketTimeOut, we
should do some retry. If it's other exception, we return false. Let me post a
patch to see if it makes sense.
Thanks,
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977638#comment-15977638
]
Rushabh S Shah commented on HADOOP-14333:
-
bq. why is this not floated up to FileSystem and FileContext?
I was not a part of api design when it was implemented.
But I think Transparent encryption is only supported by HDFS Filesystem
(DistributedFileSystem). Thats why it's not a part of FilsSystem api.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977635#comment-15977635
]
Allen Wittenauer commented on HADOOP-14333:
---
bq. The right way to ask whether the path is encrypted or not is to call
DistributedFileSystem#getEZForPath(final Path path)
(hopefully) quick question: why is this not floated up to FileSystem and
FileContext? It seems something that should be part of the standard file
system API rather than (the useless) LimitedPrivate.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977598#comment-15977598
]
Rushabh S Shah commented on HADOOP-14333:
-
I didn't read the update from [~aw] before my last update.
bq. Yes. Break the rules, live with the consequences. The whole point of having
interface stability connotations is to prevent stuff like this. If this was a
problem, the Hive project should have raised the issue before using it.
I agree with him.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
[
https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977595#comment-15977595
]
Rushabh S Shah commented on HADOOP-14333:
-
If you read back [~daryn]'s
[comment|https://issues.apache.org/jira/browse/HADOOP-14104?focusedCommentId=15924596=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15924596]
in HADOOP-14104, we discussed the exact same thing and came to decision that
since DFSClient is a private interface, we are allowed to change the signature
of isHDFSEncryptionEnabled and are allowed to throw IOException.
Since getTrashRoot was doing the wrong thing before HADOOP-14104 (Refer to
https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2498),
we decided to do the same thing. At least that is limited to shell commands
not jobs.
bq. Maybe we can temporarily change the behavior here to issue a warning
message and return false when exception is throw.
On this, I still don't think to return false in case of any exception.
If we want to temporarily fix hive use case, we can call
{{DFSClient#getKeyProviderUri}} from {{DistributedFileSystem#getTrashRoot}}.
This means {{DFSClient#isHDFSEncryptionEnabled}} is just called from hive code.
Do something like this in {{DFSClient#isHDFSEncryptionEnabled}}
{code:title=DFSClient.java|borderStyle=solid}
/**
* Probe for encryption enabled on this filesystem.
* @return true if encryption is enabled
*/
public boolean isHDFSEncryptionEnabled() {
try {
URI keyProviderUri = getKeyProviderUri();
} catch(IOException ioe) {
return false;
}
return keyProviderUri !=null
{code}
Probably add a comment on isHDFSEncryptionEnabled saying that this is here only
to unblock hive and no-one should use this.
> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke
> hacky hive code
> --
>
> Key: HADOOP-14333
> URL: https://issues.apache.org/jira/browse/HADOOP-14333
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Though Hive should be fixed not to access DFSClient which is private to
> HADOOP, removing the throws added by HADOOP-14104 is a quicker solution to
> unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
> try {
> DFSClient.class.getMethod("isHDFSEncryptionEnabled");
> } catch (NoSuchMethodException e) {
> // the method is available since Hadoop-2.7.1
> // if we run with an older Hadoop, check this ourselves
> return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI,
> "").isEmpty();
> }
> return client.isHDFSEncryptionEnabled();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
