[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181673#comment-16181673 ] Prasanth Jayachandran commented on HIVE-16426: -- [~ychena] Thanks for the clarification! > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 3.0.0 > > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > 3. Add checkpoints to related file operations to improve response time for > query cancelling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180123#comment-16180123 ] Yongzhi Chen commented on HIVE-16426: - When Cancel happens, releaseDriverContext() will be called. It calls driverCxt.shutdown(); This methods will shutdown every related running task by calling task's shutdown method. How to implement it depend on each task, for example for MapReduce task its shutdown majorly kill the job. > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 3.0.0 > > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > 3. Add checkpoints to related file operations to improve response time for > query cancelling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180002#comment-16180002 ] Prasanth Jayachandran commented on HIVE-16426: -- I don't understand how this patch handles the already running background task? When a query timeout is set, timeout monitor will set the operation state to TIMEOUT. With this patch, only the client is provided with SQLTimeoutException but the task that is actually executing on the cluster is not interrupted/cleaned up. Same is the case when user Cancels the query by ctrl + c. Isn't it? > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 3.0.0 > > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > 3. Add checkpoints to related file operations to improve response time for > query cancelling. -- This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972854#comment-15972854 ] Aihua Xu commented on HIVE-16426: - The change looks good to me. +1 > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > 3. Add checkpoints to related file operations to improve response time for > query cancelling. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968949#comment-15968949 ] Yongzhi Chen commented on HIVE-16426: - The failure is not related. > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > 3. Add checkpoints to related file operations to improve response time for > query cancelling. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968632#comment-15968632 ] Hive QA commented on HIVE-16426: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12863412/HIVE-16426.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10574 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4687/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4687/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4687/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12863412 - PreCommit-HIVE-Build > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at >
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968483#comment-15968483 ] Hive QA commented on HIVE-16426: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12863358/HIVE-16426.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10568 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver (batchId=84) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4683/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4683/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4683/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12863358 - PreCommit-HIVE-Build > Query cancel: improve the way to handle files > - > > Key: HIVE-16426 > URL: https://issues.apache.org/jira/browse/HIVE-16426 > Project: Hive > Issue Type: Improvement >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16426.1.patch > > > 1. Add data structure support to make it is easy to check query cancel status. > 2. Handle query cancel more gracefully. Remove possible file leaks caused by > query cancel as shown in following stack: > {noformat} > 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities: > [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories. > java.io.InterruptedIOException: Call interrupted > at org.apache.hadoop.ipc.Client.call(Client.java:1496) > at org.apache.hadoop.ipc.Client.call(Client.java:1439) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy20.delete(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) > at com.sun.proxy.$Proxy21.delete(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675) > at > org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671) > at > org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) > at > org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at >