Re: Computation timeout
You're right about the typos, but both parameters have defaults of 0 ms: - hive.server2.session.check.interval https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval - hive.server2.idle.operation.timeout https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout -- Lefty On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Rats, I think I just figured it out. #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. As for #1, I think it is the default value, so I am not sure I have to set it. Can you confirm that there is a typo on the name of your properties (missing last letter) and that is not the actual name of the properties ? I'll try again and keep you informed Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 20:15 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net : No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
Indeed, I was checking this out on the exact same page, but I'm almost convinced that I saw on a documentation that the default value was 3000 for the check.interval. As I can't find it again, let's say I was tired and my eyes betrayed me. Thanks a lot, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-30 9:46 GMT+02:00 Lefty Leverenz leftylever...@gmail.com: You're right about the typos, but both parameters have defaults of 0 ms: - hive.server2.session.check.interval https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval - hive.server2.idle.operation.timeout https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout -- Lefty On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Rats, I think I just figured it out. #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. As for #1, I think it is the default value, so I am not sure I have to set it. Can you confirm that there is a typo on the name of your properties (missing last letter) and that is not the actual name of the properties ? I'll try again and keep you informed Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 20:15 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net : No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
My bad, I think I just mixed up the properties. At the end of the day, everything seems to work as you described. Thanks a lot ! Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-30 9:31 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Rats, I think I just figured it out. #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. As for #1, I think it is the default value, so I am not sure I have to set it. Can you confirm that there is a typo on the name of your properties (missing last letter) and that is not the actual name of the properties ? I'll try again and keep you informed Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 20:15 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net : No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
Hmm, if you did see it somewhere please let us know. I verified the defaults in copies of HiveConf.java for releases up to 1.1.0: HiveConf-branches grep 'hive.server2.session.check.interval' * branch14-HiveConf.java: HIVE_SERVER2_SESSION_CHECK_INTERVAL(hive.server2.session.check.interval, 0ms, brnch1.0-HiveConf.java: HIVE_SERVER2_SESSION_CHECK_INTERVAL(hive.server2.session.check.interval, 0ms, brnch1.1-HiveConf.java: HIVE_SERVER2_SESSION_CHECK_INTERVAL(hive.server2.session.check.interval, 0ms, HiveConf-branches grep 'hive.server2.idle.operation.timeout' * branch14-HiveConf.java: HIVE_SERVER2_IDLE_OPERATION_TIMEOUT(hive.server2.idle.operation.timeout, 0ms, brnch1.0-HiveConf.java: HIVE_SERVER2_IDLE_OPERATION_TIMEOUT(hive.server2.idle.operation.timeout, 0ms, brnch1.1-HiveConf.java: HIVE_SERVER2_IDLE_OPERATION_TIMEOUT(hive.server2.idle.operation.timeout, 0ms, -- Lefty On Thu, Jul 30, 2015 at 3:53 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Indeed, I was checking this out on the exact same page, but I'm almost convinced that I saw on a documentation that the default value was 3000 for the check.interval. As I can't find it again, let's say I was tired and my eyes betrayed me. Thanks a lot, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-30 9:46 GMT+02:00 Lefty Leverenz leftylever...@gmail.com: You're right about the typos, but both parameters have defaults of 0 ms: - hive.server2.session.check.interval https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval - hive.server2.idle.operation.timeout https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout -- Lefty On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Rats, I think I just figured it out. #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. As for #1, I think it is the default value, so I am not sure I have to set it. Can you confirm that there is a typo on the name of your properties (missing last letter) and that is not the actual name of the properties ? I'll try again and keep you informed Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 20:15 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu
Re: Computation timeout
Rats, I think I just figured it out. #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. As for #1, I think it is the default value, so I am not sure I have to set it. Can you confirm that there is a typo on the name of your properties (missing last letter) and that is not the actual name of the properties ? I'll try again and keep you informed Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 20:15 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
this works for me: In hive-site.xml: 1. hive.server2.session.check.interva=3000; 2. hive.server2.idle.operation.timeou=-3; restart HiveServer2. at beeline, I do analyze table X compute statistics for columns, which takes longer than 30s. it was aborted by HS2 because of above settings. I guess it didn't work for you because you didn't have #1. --Xuefu On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
Yes, I set it to negative 60. It's not a problem if the session is killed. That's actually what I try to do, because I can't allow to a user to try to end an infinite request. Therefore I'll try your solution :) Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:14 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Okay. To confirm, you set it to negative 60s? The next thing you can try is to set hive.server2.idle.session.timeou=6 (60sec) and hive.server2.idle.session.check.operation=false. I'm pretty sure this works, but the user's session will be killed though. --Xuefu On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
Re: Computation timeout
I confirm : I just tried hive.server2.idle.operation.timeout setting it to -60 (seconds), but my veery slow job have not been killed. The issue here is what if another user come and try to submit a MapReduce job but the cluster is stuck in an infinite loop ?. Do you or anyone else have another idea ? Thanks, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:34 GMT+02:00 Loïc Chanel loic.cha...@telecomnancy.net: No, because I thought the idea of infinite operation was not very compatible with the idle word (as the operation will not stop running), but I'll try :-) Thanks for the idea, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 15:27 GMT+02:00 Xuefu Zhang xzh...@cloudera.com: Have you tried hive.server2.idle.operation.timeout? --Xuefu On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel loic.cha...@telecomnancy.net wrote: Hi all, As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive, I am desperately trying to set a timeout to Hive requests. My idea is that some users can make mistakes such as a join with wrong keys, and therefore start an infinite loop believing that they are just launching a very heavy job. Therefore, I'd like to set a limit to the time a request should take, in order to kill the job automatically if it exceeds it. As such a notion cannot be set directly in YARN, I saw that MapReduce2 provides with its own native timeout property, and I would like to know if Hive provides with the same property someway. Did anyone heard about such a thing ? Thanks in advance for your help, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne
