[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=491773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491773 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 28/Sep/20 00:50 Start Date: 28/Sep/20 00:50 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1279: URL: https://github.com/apache/hive/pull/1279 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491773) Time Spent: 1h 50m (was: 1h 40m) > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=487725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487725 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 22/Sep/20 03:30 Start Date: 22/Sep/20 03:30 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1279: URL: https://github.com/apache/hive/pull/1279#issuecomment-695860130 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487725) Time Spent: 1h 40m (was: 1.5h) > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=486729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486729 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 21/Sep/20 00:49 Start Date: 21/Sep/20 00:49 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1279: URL: https://github.com/apache/hive/pull/1279#issuecomment-695860130 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 486729) Time Spent: 1.5h (was: 1h 20m) > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461812 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 22/Jul/20 00:50 Start Date: 22/Jul/20 00:50 Worklog Time Spent: 10m Work Description: umamaheswararao commented on pull request #1279: URL: https://github.com/apache/hive/pull/1279#issuecomment-662179244 My latest commit has only removal of a comment: https://github.com/apache/hive/pull/1279/commits/2db5d300789bbc8325cae33eeb47840130f525d6 The patch already has green tests in previous runs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 461812) Time Spent: 1h 20m (was: 1h 10m) > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[
https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461683
]
ASF GitHub Bot logged work on HIVE-16490:
-
Author: ASF GitHub Bot
Created on: 21/Jul/20 17:37
Start Date: 21/Jul/20 17:37
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#1279:
URL: https://github.com/apache/hive/pull/1279#discussion_r458272410
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -1223,11 +1223,10 @@ public static boolean isHdfsEncryptionSupported() {
private final Configuration conf;
public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
- DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri,
conf);
-
this.conf = conf;
- this.keyProvider = dfs.getClient().getKeyProvider();
this.hdfsAdmin = new HdfsAdmin(uri, conf);
+ // HDFS-11687 added getKeyProvider method in hdfsAdmin from Hadoop-2.9.0
Review comment:
@belugabehr , Thank you for review! Done. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 461683)
Time Spent: 1h 10m (was: 1h)
> Hive should not use private HDFS APIs for encryption
>
>
> Key: HIVE-16490
> URL: https://issues.apache.org/jira/browse/HIVE-16490
> Project: Hive
> Issue Type: Improvement
> Components: Encryption
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Naveen Gangam
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When compiling against bleeding edge versions of Hive and Hadoop, we
> discovered that HIVE-16047 references a private HDFS API, DFSClient, to get
> at various encryption related information. The private API was recently
> changed by HADOOP-14104, which broke Hive compilation.
> It'd be better to instead use publicly supported APIs. HDFS-11687 has been
> filed to add whatever encryption APIs are needed by Hive. This JIRA is to
> move Hive over to these new APIs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[
https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461681
]
ASF GitHub Bot logged work on HIVE-16490:
-
Author: ASF GitHub Bot
Created on: 21/Jul/20 17:36
Start Date: 21/Jul/20 17:36
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#1279:
URL: https://github.com/apache/hive/pull/1279#discussion_r458272410
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -1223,11 +1223,10 @@ public static boolean isHdfsEncryptionSupported() {
private final Configuration conf;
public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
- DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri,
conf);
-
this.conf = conf;
- this.keyProvider = dfs.getClient().getKeyProvider();
this.hdfsAdmin = new HdfsAdmin(uri, conf);
+ // HDFS-11687 added getKeyProvider method in hdfsAdmin from Hadoop-2.9.0
Review comment:
Done. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 461681)
Time Spent: 1h (was: 50m)
> Hive should not use private HDFS APIs for encryption
>
>
> Key: HIVE-16490
> URL: https://issues.apache.org/jira/browse/HIVE-16490
> Project: Hive
> Issue Type: Improvement
> Components: Encryption
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Naveen Gangam
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When compiling against bleeding edge versions of Hive and Hadoop, we
> discovered that HIVE-16047 references a private HDFS API, DFSClient, to get
> at various encryption related information. The private API was recently
> changed by HADOOP-14104, which broke Hive compilation.
> It'd be better to instead use publicly supported APIs. HDFS-11687 has been
> filed to add whatever encryption APIs are needed by Hive. This JIRA is to
> move Hive over to these new APIs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[
https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461602
]
ASF GitHub Bot logged work on HIVE-16490:
-
Author: ASF GitHub Bot
Created on: 21/Jul/20 14:16
Start Date: 21/Jul/20 14:16
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1279:
URL: https://github.com/apache/hive/pull/1279#discussion_r458131677
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -1223,11 +1223,10 @@ public static boolean isHdfsEncryptionSupported() {
private final Configuration conf;
public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
- DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri,
conf);
-
this.conf = conf;
- this.keyProvider = dfs.getClient().getKeyProvider();
this.hdfsAdmin = new HdfsAdmin(uri, conf);
+ // HDFS-11687 added getKeyProvider method in hdfsAdmin from Hadoop-2.9.0
Review comment:
Please remove this comment. Keep this information in JIRA. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 461602)
Time Spent: 50m (was: 40m)
> Hive should not use private HDFS APIs for encryption
>
>
> Key: HIVE-16490
> URL: https://issues.apache.org/jira/browse/HIVE-16490
> Project: Hive
> Issue Type: Improvement
> Components: Encryption
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Naveen Gangam
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When compiling against bleeding edge versions of Hive and Hadoop, we
> discovered that HIVE-16047 references a private HDFS API, DFSClient, to get
> at various encryption related information. The private API was recently
> changed by HADOOP-14104, which broke Hive compilation.
> It'd be better to instead use publicly supported APIs. HDFS-11687 has been
> filed to add whatever encryption APIs are needed by Hive. This JIRA is to
> move Hive over to these new APIs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461282 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 20/Jul/20 21:21 Start Date: 20/Jul/20 21:21 Worklog Time Spent: 10m Work Description: umamaheswararao commented on pull request #1279: URL: https://github.com/apache/hive/pull/1279#issuecomment-661339873 Thanks @ashutoshc for the review!. I have updated it. Please take a look. Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 461282) Time Spent: 40m (was: 0.5h) > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[
https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461246
]
ASF GitHub Bot logged work on HIVE-16490:
-
Author: ASF GitHub Bot
Created on: 20/Jul/20 20:06
Start Date: 20/Jul/20 20:06
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#1279:
URL: https://github.com/apache/hive/pull/1279#discussion_r457661391
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -1223,11 +1223,31 @@ public static boolean isHdfsEncryptionSupported() {
private final Configuration conf;
public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
- DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri,
conf);
-
this.conf = conf;
- this.keyProvider = dfs.getClient().getKeyProvider();
this.hdfsAdmin = new HdfsAdmin(uri, conf);
+ this.keyProvider = getKeyProvider();
+}
+
+private KeyProvider getKeyProvider() throws IOException {
+ if (isMethodExist(HdfsAdmin.class, "getKeyProvider")) {
Review comment:
That is great. I will remove that then. Thanks for pointing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 461246)
Time Spent: 0.5h (was: 20m)
> Hive should not use private HDFS APIs for encryption
>
>
> Key: HIVE-16490
> URL: https://issues.apache.org/jira/browse/HIVE-16490
> Project: Hive
> Issue Type: Improvement
> Components: Encryption
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Naveen Gangam
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When compiling against bleeding edge versions of Hive and Hadoop, we
> discovered that HIVE-16047 references a private HDFS API, DFSClient, to get
> at various encryption related information. The private API was recently
> changed by HADOOP-14104, which broke Hive compilation.
> It'd be better to instead use publicly supported APIs. HDFS-11687 has been
> filed to add whatever encryption APIs are needed by Hive. This JIRA is to
> move Hive over to these new APIs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[
https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=461243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-461243
]
ASF GitHub Bot logged work on HIVE-16490:
-
Author: ASF GitHub Bot
Created on: 20/Jul/20 20:03
Start Date: 20/Jul/20 20:03
Worklog Time Spent: 10m
Work Description: ashutoshc commented on a change in pull request #1279:
URL: https://github.com/apache/hive/pull/1279#discussion_r457659517
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -1223,11 +1223,31 @@ public static boolean isHdfsEncryptionSupported() {
private final Configuration conf;
public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
- DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri,
conf);
-
this.conf = conf;
- this.keyProvider = dfs.getClient().getKeyProvider();
this.hdfsAdmin = new HdfsAdmin(uri, conf);
+ this.keyProvider = getKeyProvider();
+}
+
+private KeyProvider getKeyProvider() throws IOException {
+ if (isMethodExist(HdfsAdmin.class, "getKeyProvider")) {
Review comment:
this is not needed. Hive's minimum version required for Hadoop is 3.0
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 461243)
Time Spent: 20m (was: 10m)
> Hive should not use private HDFS APIs for encryption
>
>
> Key: HIVE-16490
> URL: https://issues.apache.org/jira/browse/HIVE-16490
> Project: Hive
> Issue Type: Improvement
> Components: Encryption
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Naveen Gangam
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When compiling against bleeding edge versions of Hive and Hadoop, we
> discovered that HIVE-16047 references a private HDFS API, DFSClient, to get
> at various encryption related information. The private API was recently
> changed by HADOOP-14104, which broke Hive compilation.
> It'd be better to instead use publicly supported APIs. HDFS-11687 has been
> filed to add whatever encryption APIs are needed by Hive. This JIRA is to
> move Hive over to these new APIs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-16490) Hive should not use private HDFS APIs for encryption
[ https://issues.apache.org/jira/browse/HIVE-16490?focusedWorklogId=460791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460791 ] ASF GitHub Bot logged work on HIVE-16490: - Author: ASF GitHub Bot Created on: 19/Jul/20 07:16 Start Date: 19/Jul/20 07:16 Worklog Time Spent: 10m Work Description: umamaheswararao opened a new pull request #1279: URL: https://github.com/apache/hive/pull/1279 https://issues.apache.org/jira/browse/HIVE-16490 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 460791) Remaining Estimate: 0h Time Spent: 10m > Hive should not use private HDFS APIs for encryption > > > Key: HIVE-16490 > URL: https://issues.apache.org/jira/browse/HIVE-16490 > Project: Hive > Issue Type: Improvement > Components: Encryption >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Naveen Gangam >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > > When compiling against bleeding edge versions of Hive and Hadoop, we > discovered that HIVE-16047 references a private HDFS API, DFSClient, to get > at various encryption related information. The private API was recently > changed by HADOOP-14104, which broke Hive compilation. > It'd be better to instead use publicly supported APIs. HDFS-11687 has been > filed to add whatever encryption APIs are needed by Hive. This JIRA is to > move Hive over to these new APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
