[jira] [Work logged] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?focusedWorklogId=487458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487458
]
ASF GitHub Bot logged work on HIVE-24159:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 03:03
Start Date: 22/Sep/20 03:03
Worklog Time Spent: 10m
Work Description: abstractdog merged pull request #1495:
URL: https://github.com/apache/hive/pull/1495
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487458)
Time Spent: 50m (was: 40m)
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main]
> tez.TezTask: Failed to execute tez graph.
> org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
>
[jira] [Work logged] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?focusedWorklogId=486885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486885
]
ASF GitHub Bot logged work on HIVE-24159:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 12:16
Start Date: 21/Sep/20 12:16
Worklog Time Spent: 10m
Work Description: abstractdog merged pull request #1495:
URL: https://github.com/apache/hive/pull/1495
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486885)
Time Spent: 40m (was: 0.5h)
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main]
> tez.TezTask: Failed to execute tez graph.
> org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
>
[jira] [Work logged] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?focusedWorklogId=484293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484293
]
ASF GitHub Bot logged work on HIVE-24159:
-
Author: ASF GitHub Bot
Created on: 15/Sep/20 04:52
Start Date: 15/Sep/20 04:52
Worklog Time Spent: 10m
Work Description: ashutoshc commented on pull request #1495:
URL: https://github.com/apache/hive/pull/1495#issuecomment-692462659
+1 LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484293)
Time Spent: 0.5h (was: 20m)
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main]
> tez.TezTask: Failed to execute tez graph.
> org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
>
[jira] [Work logged] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?focusedWorklogId=483913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483913
]
ASF GitHub Bot logged work on HIVE-24159:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 10:31
Start Date: 14/Sep/20 10:31
Worklog Time Spent: 10m
Work Description: abstractdog commented on pull request #1495:
URL: https://github.com/apache/hive/pull/1495#issuecomment-691967429
@ashutoshc : could you please take a quick look? thanks...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483913)
Time Spent: 20m (was: 10m)
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?focusedWorklogId=483912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483912
]
ASF GitHub Bot logged work on HIVE-24159:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 10:30
Start Date: 14/Sep/20 10:30
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #1495:
URL: https://github.com/apache/hive/pull/1495
### What changes were proposed in this pull request?
Check secure env before taking care of delegation tokens.
### Why are the changes needed?
Broken kafka_storage_handler.q test after HIVE-23408
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested with temporarily enabled qtest.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483912)
Remaining Estimate: 0h
Time Spent: 10m
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
