[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536425#comment-15536425 ] Naveen Gangam commented on HIVE-14784: -- Thanks for the review and the commit [~ychena] > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0 > > Attachments: HIVE-14784.1.patch, HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533942#comment-15533942 ] Yongzhi Chen commented on HIVE-14784: - +1 > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.1.patch, HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533540#comment-15533540 ] Naveen Gangam commented on HIVE-14784: -- forgot to comment on this. They do not appear to be related. Other prior builds also have the same failures. So +1 for me. > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.1.patch, HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at >
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533435#comment-15533435 ] Yongzhi Chen commented on HIVE-14784: - Are the failures related to the change? > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.1.patch, HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531287#comment-15531287 ] Hive QA commented on HIVE-14784: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12830753/HIVE-14784.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9882 tests executed *Failed tests:* {noformat} TestNegativeCliDriver-nopart_insert.q-input41.q-having1.q-and-765-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1334/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1334/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1334/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12830753 - PreCommit-HIVE-Build > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.1.patch, HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} >
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503888#comment-15503888 ] Yongzhi Chen commented on HIVE-14784: - Had a discussion with Naveen, He will add warning related to old log files no long exist. > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745)
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498831#comment-15498831 ] Hive QA commented on HIVE-14784: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12828958/HIVE-14784.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10555 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1224/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1224/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1224/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12828958 - PreCommit-HIVE-MASTER-Build > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[ https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498013#comment-15498013 ] Naveen Gangam commented on HIVE-14784: -- patch posted on review board at https://reviews.apache.org/r/51982/ [~aihuaxu] [~ctang.ma] [~ychena] can one or all of you please review this? Thanks > Operation logs are disabled automatically if the parent directory does not > exist. > - > > Key: HIVE-14784 > URL: https://issues.apache.org/jira/browse/HIVE-14784 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14784.patch > > > Operation logging is disabled automatically for the query if for some reason > the parent directory (named after the hive session id) that gets created when > the session is established gets deleted (for any reason). For ex: if the > operation logdir is /tmp which automatically can get purged at a configured > interval by the OS. > Running a query from that session leads to > {code} > 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: > Unable to create operation log file: > /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599 > java.io.IOException: No such file or directory > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:1012) > at > org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) > at > org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > This later leads to errors like (more prominent when using HUE as HUE does > not close hive sessions and attempts to retrieve the operations logs days > after they were created). > {code} > WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching > results: > org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated > with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, > getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at >