[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24159: Fix Version/s: 4.0.0 > Kafka storage handler broken in secure environment pt2: short-circuit on > non-secure environment > --- > > Key: HIVE-24159 > URL: https://issues.apache.org/jira/browse/HIVE-24159 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized > upstream that the kafka qtest fails. Instead of setting up a kerberized > environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't > seen hive.server2.authentication.kerberos.principal used in *.q files) I > managed to make the test with a simple > UserGroupInformation.isSecurityEnabled() check, which can be also useful for > every non-secure environment. > For reference, the exception was: > {code} > 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main] > tez.TezTask: Failed to execute tez graph. > org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient > at > org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451) > ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.kafka.clients.admin.Admin.create(Admin.java:59) > ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] > at > org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) > ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333) > ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301) > ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282) > ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516) > ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232) > [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247) > [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193) > [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) > [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343) > [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1465) > [classes/:?] > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1438) > [classes/:?] > at >
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24159: Description: As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized upstream that the kafka qtest fails. Instead of setting up a kerberized environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't seen hive.server2.authentication.kerberos.principal used in *.q files) I managed to make the test with a simple UserGroupInformation.isSecurityEnabled() check, which can be also useful for every non-secure environment. For reference, the exception was: {code} 2020-09-14T03:30:01,217 ERROR [a42ef4c6-190c-47a6-86ad-8bf13b8a2dc1 main] tez.TezTask: Failed to execute tez graph. org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451) ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.admin.Admin.create(Admin.java:59) ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[kafka-clients-2.4.1.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaDelegationTokenForBrokers(DagUtils.java:333) ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getKafkaCredentials(DagUtils.java:301) ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addCredentials(DagUtils.java:282) ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:516) ~[hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:223) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:232) [hive-exec-3.1.3000.7.1.4.0-SNAPSHOT.jar:3.1.3000.7.1.4.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247) [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193) [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343) [hive-cli-3.1.3000.7.1.4.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1465) [classes/:?] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1438) [classes/:?] at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:194) [classes/:?] at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104) [classes/:?] at org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver(TestMiniHiveKafkaCliDriver.java:60) [test-classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151] at
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24159: -- Labels: pull-request-available (was: ) > Kafka storage handler broken in secure environment pt2: short-circuit on > non-secure environment > --- > > Key: HIVE-24159 > URL: https://issues.apache.org/jira/browse/HIVE-24159 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized > upstream that the kafka qtest fails. Instead of setting up a kerberized > environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't > seen hive.server2.authentication.kerberos.principal used in *.q files) I > managed to make the test with a simple > UserGroupInformation.isSecurityEnabled() check, which can be also useful for > every non-secure environment. > For reference, the exception was: > {code} > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24159: Description: As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized upstream that the kafka qtest fails. Instead of setting up a kerberized environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't seen hive.server2.authentication.kerberos.principal used in *.q files) I managed to make the test with a simple UserGroupInformation.isSecurityEnabled() check, which can be also useful for every non-secure environment. For reference, the exception was: {code} {code} was:As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized upstream that the kafka qtest fails. Instead of setting up a kerberized environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't seen hive.server2.authentication.kerberos.principal used in *.q files) I managed to make the test with a simple UserGroupInformation.isSecurityEnabled() check, which can be also useful for every non-secure environment. > Kafka storage handler broken in secure environment pt2: short-circuit on > non-secure environment > --- > > Key: HIVE-24159 > URL: https://issues.apache.org/jira/browse/HIVE-24159 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized > upstream that the kafka qtest fails. Instead of setting up a kerberized > environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't > seen hive.server2.authentication.kerberos.principal used in *.q files) I > managed to make the test with a simple > UserGroupInformation.isSecurityEnabled() check, which can be also useful for > every non-secure environment. > For reference, the exception was: > {code} > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24159) Kafka storage handler broken in secure environment pt2: short-circuit on non-secure environment
[ https://issues.apache.org/jira/browse/HIVE-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24159: Description: As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized upstream that the kafka qtest fails. Instead of setting up a kerberized environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't seen hive.server2.authentication.kerberos.principal used in *.q files) I managed to make the test with a simple UserGroupInformation.isSecurityEnabled() check, which can be also useful for every non-secure environment. > Kafka storage handler broken in secure environment pt2: short-circuit on > non-secure environment > --- > > Key: HIVE-24159 > URL: https://issues.apache.org/jira/browse/HIVE-24159 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > As kafka_storage_handler.q was disabled by HIVE-23985, I haven't realized > upstream that the kafka qtest fails. Instead of setting up a kerberized > environment in qtest (which doesn't seem to be a usual usecase, e.g. haven't > seen hive.server2.authentication.kerberos.principal used in *.q files) I > managed to make the test with a simple > UserGroupInformation.isSecurityEnabled() check, which can be also useful for > every non-secure environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)