[jira] [Commented] (HIVE-15662) check startTime in SparkTask to make sure startTime is not less than submitTime
[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840026#comment-15840026
]
zhihai xu commented on HIVE-15662:
--
Thanks [~csun] for reviewing and committing the patch!
> check startTime in SparkTask to make sure startTime is not less than
> submitTime
> ---
>
> Key: HIVE-15662
> URL: https://issues.apache.org/jira/browse/HIVE-15662
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15662.000.patch, HIVE-15662.001.patch
>
>
> Check startTime in SparkTask to make sure startTime is not less than
> submitTime. We saw a corner case when the sparkTask is finished in less than
> 1 second, the startTime may not be set because RemoteSparkJobMonitor will
> sleep for 1 second then check the state, in this case, right after sleep for
> one second, the spark job is already completed. one example query with 3
> spark tasks, the second one finished quickly for around 1 second:
> {code}
> SparkTask1:
> "finishTime":1484638391978
> "submitTime":1484638385933
> "startTime": 1484638386973
> SparkTask2:
> "finishTime":1484638393019
> "submitTime":1484638391979
> "startTime": 1484638386973
> SparkTask3:
> "finishTime":1484638432123
> "submitTime":1484638393020
> "startTime": 1484638394057
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15662) check startTime in SparkTask to make sure startTime is not less than submitTime
[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833035#comment-15833035
]
Hive QA commented on HIVE-15662:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12848661/HIVE-15662.001.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10974 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[cascade_dbdrop]
(batchId=226)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
(batchId=226)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3105/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3105/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3105/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12848661 - PreCommit-HIVE-Build
> check startTime in SparkTask to make sure startTime is not less than
> submitTime
> ---
>
> Key: HIVE-15662
> URL: https://issues.apache.org/jira/browse/HIVE-15662
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15662.000.patch, HIVE-15662.001.patch
>
>
> Check startTime in SparkTask to make sure startTime is not less than
> submitTime. We saw a corner case when the sparkTask is finished in less than
> 1 second, the startTime may not be set because RemoteSparkJobMonitor will
> sleep for 1 second then check the state, in this case, right after sleep for
> one second, the spark job is already completed. one example query with 3
> spark tasks, the second one finished quickly for around 1 second:
> {code}
> SparkTask1:
> "finishTime":1484638391978
> "submitTime":1484638385933
> "startTime": 1484638386973
> SparkTask2:
> "finishTime":1484638393019
> "submitTime":1484638391979
> "startTime": 1484638386973
> SparkTask3:
> "finishTime":1484638432123
> "submitTime":1484638393020
> "startTime": 1484638394057
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15662) check startTime in SparkTask to make sure startTime is not less than submitTime
[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832657#comment-15832657
]
zhihai xu commented on HIVE-15662:
--
Thanks for the review [~csun]! I attached a new patch HIVE-15662.001.patch
which add the comment for this race condition. Please review thanks
> check startTime in SparkTask to make sure startTime is not less than
> submitTime
> ---
>
> Key: HIVE-15662
> URL: https://issues.apache.org/jira/browse/HIVE-15662
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15662.000.patch, HIVE-15662.001.patch
>
>
> Check startTime in SparkTask to make sure startTime is not less than
> submitTime. We saw a corner case when the sparkTask is finished in less than
> 1 second, the startTime may not be set because RemoteSparkJobMonitor will
> sleep for 1 second then check the state, in this case, right after sleep for
> one second, the spark job is already completed. one example query with 3
> spark tasks, the second one finished quickly for around 1 second:
> {code}
> SparkTask1:
> "finishTime":1484638391978
> "submitTime":1484638385933
> "startTime": 1484638386973
> SparkTask2:
> "finishTime":1484638393019
> "submitTime":1484638391979
> "startTime": 1484638386973
> SparkTask3:
> "finishTime":1484638432123
> "submitTime":1484638393020
> "startTime": 1484638394057
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15662) check startTime in SparkTask to make sure startTime is not less than submitTime
[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832429#comment-15832429
]
Chao Sun commented on HIVE-15662:
-
+1. Maybe better to add a comment above the code to explain when that will
happen. Thanks.
> check startTime in SparkTask to make sure startTime is not less than
> submitTime
> ---
>
> Key: HIVE-15662
> URL: https://issues.apache.org/jira/browse/HIVE-15662
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15662.000.patch
>
>
> Check startTime in SparkTask to make sure startTime is not less than
> submitTime. We saw a corner case when the sparkTask is finished in less than
> 1 second, the startTime may not be set because RemoteSparkJobMonitor will
> sleep for 1 second then check the state, in this case, right after sleep for
> one second, the spark job is already completed. one example query with 3
> spark tasks, the second one finished quickly for around 1 second:
> {code}
> SparkTask1:
> "finishTime":1484638391978
> "submitTime":1484638385933
> "startTime": 1484638386973
> SparkTask2:
> "finishTime":1484638393019
> "submitTime":1484638391979
> "startTime": 1484638386973
> SparkTask3:
> "finishTime":1484638432123
> "submitTime":1484638393020
> "startTime": 1484638394057
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15662) check startTime in SparkTask to make sure startTime is not less than submitTime
[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830836#comment-15830836
]
Hive QA commented on HIVE-15662:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12848175/HIVE-15662.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 78 failed/errored test(s), 10964 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins]
(batchId=219)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=219)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=219)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_part]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input19] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample5] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[serde_opencsv]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[specialChar] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=19)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[dboutput]
(batchId=222)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[fileformat_base64]
(batchId=222)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[udf_row_sequence]
(batchId=222)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook]
(batchId=222)
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testCliDriver[case_with_row_sequence]
(batchId=225)
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testCliDriver[invalid_row_sequence]
(batchId=225)
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testCliDriver[serde_regex]
(batchId=225)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=158)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
(batchId=156)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_values]
(batchId=157)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_unencrypted_tbl]
(batchId=159)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_load_data_to_encrypted_tables]
(batchId=157)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=157)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_encrypted_tbl]
(batchId=159)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_unencrypted_tbl]
(batchId=159)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_unencrypted_nonhdfs_external_tables]
(batchId=157)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[cascade_dbdrop]
(batchId=226)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
(batchId=226)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=136)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape1]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape2]
(batchId=154)
