Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-17 18:56:20.310048) Review request for hive. Changes --- updating diff against the latest trunk Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs (updated) - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1104456 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1104456 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1104456 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review671 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/626/#comment1339 The loop wrongly assumes that calling Thread.currentThread().interrupt() will throw ThreadInterrupted while it only sets interrupted flag on the thread. The result is infinite loop with executionDone never set to true. The loop is actually not needed as executor.shutdown() does the same thing - waits for all queued jobs to complete. - georgedj On 2011-05-10 21:55:56, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-10 21:55:56) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
On 2011-05-14 07:39:31, Ning Zhang wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java, line 1194 https://reviews.apache.org/r/626/diff/3/?file=18468#file18468line1194 We shouldn't check interrupt here since ObjectStore should be run at the metastore server side (if Hive is running non-local mode for metastore). So ^C on CLI won't be detected here. Instead we should check it in HiveMetaStoreClient.java. Since interrupting JDO is quite complicated, I suggest removing the interrupt handling for JDO completely in this JIRA and file a separate JIRA for that. ok, reverting ObjectStore changes. On 2011-05-14 07:39:31, Ning Zhang wrote: trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java, line 326 https://reviews.apache.org/r/626/diff/2-3/?file=17625#file17625line326 Is this comment correct? Nope, comment fixed. - georgedj --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review670 --- On 2011-05-10 21:55:56, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-10 21:55:56) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-16 17:25:41.603093) Review request for hive. Changes --- - Fixed function name typo. - Updated inaccurate comment. - Reverted all changes in ObjectStore. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs (updated) - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review673 --- Ship it! Looks good to me. George, can you rebase to the latest trunk and upload it to the JIRA? I'll start testing. - Ning On 2011-05-16 17:25:41, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-16 17:25:41) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review670 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/626/#comment1337 Is this comment correct? trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java https://reviews.apache.org/r/626/#comment1336 typo: interrup - interrupt. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/626/#comment1335 We shouldn't check interrupt here since ObjectStore should be run at the metastore server side (if Hive is running non-local mode for metastore). So ^C on CLI won't be detected here. Instead we should check it in HiveMetaStoreClient.java. Since interrupting JDO is quite complicated, I suggest removing the interrupt handling for JDO completely in this JIRA and file a separate JIRA for that. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/626/#comment1334 typo: interrup - interrupt trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/626/#comment1338 why these are removed? - Ning On 2011-05-10 21:55:56, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-10 21:55:56) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-05-10 21:55:56.707055) Review request for hive. Changes --- Fixed some white space formatting Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs (updated) - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1100312 trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptCallback.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/common/HiveInterruptUtils.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1100312 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1100312 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Tested interrupting the command on large data set in different stages of the processing (JDO, MapReduce, building the input summary). example queries: explain select count(*) from xyz where ds = '2011-02-01'; select count(*) from xyz where ds = '2011-02-01'; Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-04-28 20:54:48.288055) Review request for hive. Changes --- Added code to kill all running jobs before interrupting the current CLI thread. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs (updated) - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1097569 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1097569 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review609 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/626/#comment1252 This change kills the launched MR job. however the following interrupt() only called to the current thread. This does not really behaves what it is expected to be when there are multiple threads running. For example when we are getting partitions from metastore (JDO may use its thread pool) and when getting HDFS summaries (we are also using thread pool). For these situations we probably need to interrupt all child threads. - Ning On 2011-04-28 20:54:48, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-04-28 20:54:48) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1097569 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1097569 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/#review505 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/626/#comment1030 Does this handle the case when a MapReduce job is launched and CLI got a CTL+C? Currently we have a JVM shutdownHook defined in HadoopJobExecHelper.java to kill the MR job before the JVM shut down (the old behavior when hitting a CTL+C). Here if we don't shut down the JVM, you should do the same thing to kill the running MR jobs. - Ning On 2011-04-20 16:49:05, georgedj wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-04-20 16:49:05) Review request for hive. Summary --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095439 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Thanks, georgedj
Re: Review Request: HIVE-243 - ^C breaks out of running query, but not whole CLI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/626/ --- (Updated 2011-04-20 16:49:05.429702) Review request for hive. Summary (updated) --- Fixed by adding INT signal hander that interrupts the CLI thread. The CLI thread gets InterruptedException and it stops the current command. This addresses bug HIVE-243. https://issues.apache.org/jira/browse/HIVE-243 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095439 Diff: https://reviews.apache.org/r/626/diff Testing --- Manual test Ctrl+C on bunch of command (show tables, select, etc) Thanks, georgedj
