[jira] [Updated] (HIVE-10021) Alter index rebuild statements submitted through HiveServer2 fail when Sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10021: Attachment: HIVE-10021.2.patch Rather than passing the userName around, use the one saved in the SessionState. Alter index rebuild statements submitted through HiveServer2 fail when Sentry is enabled -- Key: HIVE-10021 URL: https://issues.apache.org/jira/browse/HIVE-10021 Project: Hive Issue Type: Bug Components: HiveServer2, Indexing Affects Versions: 0.13.1, 2.0.0 Environment: CDH 5.3.2 Reporter: Richard Williams Assignee: Aihua Xu Attachments: HIVE-10021.2.patch, HIVE-10021.patch When HiveServer2 is configured to authorize submitted queries and statements through Sentry, any attempt to issue an alter index rebuild statement fails with a SemanticException caused by a NullPointerException. This occurs regardless of whether the index is a compact or bitmap index. The root cause of the problem appears to be the fact that the static createRootTask function in org.apache.hadoop.hive.ql.optimizer.IndexUtils creates a new org.apache.hadoop.hive.ql.Driver object to compile the index builder query, and this new Driver object, unlike the one used by HiveServer2 to compile the submitted statement, is used without having its userName field initialized with the submitting user's username. Adding null checks to the Sentry code is insufficient to solve this problem, because Sentry needs the userName to determine whether or not the submitting user should be able to execute the index rebuild statement. Example stack trace from the HiveServer2 logs: {noformat} FAILED: NullPointerException null java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333) at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988) at org.apache.hadoop.security.Groups.getGroups(Groups.java:161) at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:46) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getGroups(HiveAuthzBinding.java:370) at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:314) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440) at org.apache.hadoop.hive.ql.optimizer.IndexUtils.createRootTask(IndexUtils.java:258) at org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler.getIndexBuilderMapRedTask(CompactIndexHandler.java:149) at org.apache.hadoop.hive.ql.index.TableBasedIndexHandler.generateIndexBuildTaskList(TableBasedIndexHandler.java:67) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.getIndexBuilderMapRed(DDLSemanticAnalyzer.java:1171) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterIndexRebuild(DDLSemanticAnalyzer.java:1117) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:410) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:204) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1019) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100) at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:173) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:370) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:357) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:238) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:393) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:99) at
[jira] [Updated] (HIVE-10021) Alter index rebuild statements submitted through HiveServer2 fail when Sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10021: Attachment: HIVE-10021.patch Alter index rebuild statements submitted through HiveServer2 fail when Sentry is enabled -- Key: HIVE-10021 URL: https://issues.apache.org/jira/browse/HIVE-10021 Project: Hive Issue Type: Bug Components: HiveServer2, Indexing Affects Versions: 0.13.1 Environment: CDH 5.3.2 Reporter: Richard Williams Attachments: HIVE-10021.patch When HiveServer2 is configured to authorize submitted queries and statements through Sentry, any attempt to issue an alter index rebuild statement fails with a SemanticException caused by a NullPointerException. This occurs regardless of whether the index is a compact or bitmap index. The root cause of the problem appears to be the fact that the static createRootTask function in org.apache.hadoop.hive.ql.optimizer.IndexUtils creates a new org.apache.hadoop.hive.ql.Driver object to compile the index builder query, and this new Driver object, unlike the one used by HiveServer2 to compile the submitted statement, is used without having its userName field initialized with the submitting user's username. Adding null checks to the Sentry code is insufficient to solve this problem, because Sentry needs the userName to determine whether or not the submitting user should be able to execute the index rebuild statement. Example stack trace from the HiveServer2 logs: {noformat} FAILED: NullPointerException null java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333) at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988) at org.apache.hadoop.security.Groups.getGroups(Groups.java:161) at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:46) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getGroups(HiveAuthzBinding.java:370) at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:314) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440) at org.apache.hadoop.hive.ql.optimizer.IndexUtils.createRootTask(IndexUtils.java:258) at org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler.getIndexBuilderMapRedTask(CompactIndexHandler.java:149) at org.apache.hadoop.hive.ql.index.TableBasedIndexHandler.generateIndexBuildTaskList(TableBasedIndexHandler.java:67) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.getIndexBuilderMapRed(DDLSemanticAnalyzer.java:1171) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterIndexRebuild(DDLSemanticAnalyzer.java:1117) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:410) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:204) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1019) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100) at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:173) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:370) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:357) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:238) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:393) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at
[jira] [Updated] (HIVE-10021) Alter index rebuild statements submitted through HiveServer2 fail when Sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao updated HIVE-10021: Description: When HiveServer2 is configured to authorize submitted queries and statements through Sentry, any attempt to issue an alter index rebuild statement fails with a SemanticException caused by a NullPointerException. This occurs regardless of whether the index is a compact or bitmap index. The root cause of the problem appears to be the fact that the static createRootTask function in org.apache.hadoop.hive.ql.optimizer.IndexUtils creates a new org.apache.hadoop.hive.ql.Driver object to compile the index builder query, and this new Driver object, unlike the one used by HiveServer2 to compile the submitted statement, is used without having its userName field initialized with the submitting user's username. Adding null checks to the Sentry code is insufficient to solve this problem, because Sentry needs the userName to determine whether or not the submitting user should be able to execute the index rebuild statement. Example stack trace from the HiveServer2 logs: {noformat} FAILED: NullPointerException null java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333) at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988) at org.apache.hadoop.security.Groups.getGroups(Groups.java:161) at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:46) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getGroups(HiveAuthzBinding.java:370) at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:314) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440) at org.apache.hadoop.hive.ql.optimizer.IndexUtils.createRootTask(IndexUtils.java:258) at org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler.getIndexBuilderMapRedTask(CompactIndexHandler.java:149) at org.apache.hadoop.hive.ql.index.TableBasedIndexHandler.generateIndexBuildTaskList(TableBasedIndexHandler.java:67) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.getIndexBuilderMapRed(DDLSemanticAnalyzer.java:1171) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterIndexRebuild(DDLSemanticAnalyzer.java:1117) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:410) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:204) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1019) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100) at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:173) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:715) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:370) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:357) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:238) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:393) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at