[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-24159:
Fix Version/s: 4.0.0
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main]
> tez.TezTask: Failed to execute tez graph.
> org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
> ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516)
> ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232)
> [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
> [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343)
> [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1465)
> [classes/:?]
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1438)
> [classes/:?]
> at
>
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-24159:
Description:
As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
upstream that the kafka qtest fails. Instead of setting up a kerberized
environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
seen hive.server2.authentication.kerberos.principal used in *.q files) I
managed to make the test with a simple UserGroupInformation.isSecurityEnabled()
check, which can be also useful for every non-secure environment.
For reference, the exception was:
{code}
2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main]
tez.TezTask: Failed to execute tez graph.
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at
org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
at
org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333)
~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301)
~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282)
~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516)
~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232)
[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
[hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
[hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
[hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343)
[hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1465)
[classes/:?]
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1438)
[classes/:?]
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:194)
[classes/:?]
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
[classes/:?]
at
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver(TestMiniHiveKafkaCliDriver.java:60)
[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_151]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24159:
--
Labels: pull-request-available (was: )
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[
https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-24159:
Description:
As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
upstream that the kafka qtest fails. Instead of setting up a kerberized
environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
seen hive.server2.authentication.kerberos.principal used in *.q files) I
managed to make the test with a simple UserGroupInformation.isSecurityEnabled()
check, which can be also useful for every non-secure environment.
For reference, the exception was:
{code}
{code}
was:As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
upstream that the kafka qtest fails. Instead of setting up a kerberized
environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
seen hive.server2.authentication.kerberos.principal used in *.q files) I
managed to make the test with a simple UserGroupInformation.isSecurityEnabled()
check, which can be also useful for every non-secure environment.
> Kafka storage handler broken in secure environment pt2: short-circuit on
> non-secure environment
> ---
>
> Key: HIVE-24159
> URL: https://issues.apache.org/jira/browse/HIVE-24159
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized
> upstream that the kafka qtest fails. Instead of setting up a kerberized
> environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't
> seen hive.server2.authentication.kerberos.principal used in *.q files) I
> managed to make the test with a simple
> UserGroupInformation.isSecurityEnabled() check, which can be also useful for
> every non-secure environment.
> For reference, the exception was:
> {code}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24159: Description: As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized upstream that the kafka qtest fails. Instead of setting up a kerberized environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't seen hive.server2.authentication.kerberos.principal used in *.q files) I managed to make the test with a simple UserGroupInformation.isSecurityEnabled() check, which can be also useful for every non-secure environment. > Kafka storage handler broken in secure environment pt2: short-circuit on > non-secure environment > --- > > Key: HIVE-24159 > URL: https://issues.apache.org/jira/browse/HIVE-24159 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized > upstream that the kafka qtest fails. Instead of setting up a kerberized > environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't > seen hive.server2.authentication.kerberos.principal used in *.q files) I > managed to make the test with a simple > UserGroupInformation.isSecurityEnabled() check, which can be also useful for > every non-secure environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
