[jira] [Commented] (IMPALA-8926) TestResultSpooling::_test_full_queue is flaky
[ https://issues.apache.org/jira/browse/IMPALA-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944582#comment-16944582 ] ASF subversion and git services commented on IMPALA-8926: - Commit c67e0868e032d0aec95018612b121c18aed3b914 in impala's branch refs/heads/master from Sahil Takiar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=c67e086 ] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests Disable the tests TestResultSpooling::test_full_queue(_large_fetch) until we figure out why they are flaky. Replace the sleep in TestAdmissionController::test_release_backend with assert_eventually to reduce flakiness. Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Reviewed-on: http://gerrit.cloudera.org:8080/14337 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > TestResultSpooling::_test_full_queue is flaky > - > > Key: IMPALA-8926 > URL: https://issues.apache.org/jira/browse/IMPALA-8926 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Critical > > Has happened a few times, error message is: > {code:java} > query_test/test_result_spooling.py:116: in test_full_queue_large_fetch > self._test_full_queue(vector, query, fetch_size=num_rows) > query_test/test_result_spooling.py:148: in _test_full_queue assert > re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) \ E > assert None is not None E+ where None = 0x7f35f0aee320>('RowBatchSendWaitTime: [1-9]', 'Query > (id=e948cdd2bbde9430:082830be):\n DEBUG MODE WARNING: Query profile > created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: > 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: > 0.000ns\n') E+where = re.search E >+and 'Query (id=e948cdd2bbde9430:082830be):\n DEBUG MODE > WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n > - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - > WriteIoWaitTime: 0.000ns\n' = BeeswaxConnection.get_runtime_profile of > 0xcde2310>>( 0xcdf0810>) E+ where BeeswaxConnection.get_runtime_profile of > > = > 0xcde2310>.get_runtime_profile E+where > = > .client {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8926) TestResultSpooling::_test_full_queue is flaky
[ https://issues.apache.org/jira/browse/IMPALA-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942283#comment-16942283 ] Michael Ho commented on IMPALA-8926: Bumping priority as a number of regular regression builds are affected. May be worth considering disabling the test to unbreak the build for now. > TestResultSpooling::_test_full_queue is flaky > - > > Key: IMPALA-8926 > URL: https://issues.apache.org/jira/browse/IMPALA-8926 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Critical > > Has happened a few times, error message is: > {code:java} > query_test/test_result_spooling.py:116: in test_full_queue_large_fetch > self._test_full_queue(vector, query, fetch_size=num_rows) > query_test/test_result_spooling.py:148: in _test_full_queue assert > re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) \ E > assert None is not None E+ where None = 0x7f35f0aee320>('RowBatchSendWaitTime: [1-9]', 'Query > (id=e948cdd2bbde9430:082830be):\n DEBUG MODE WARNING: Query profile > created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: > 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: > 0.000ns\n') E+where = re.search E >+and 'Query (id=e948cdd2bbde9430:082830be):\n DEBUG MODE > WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n > - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - > WriteIoWaitTime: 0.000ns\n' = BeeswaxConnection.get_runtime_profile of > 0xcde2310>>( 0xcdf0810>) E+ where BeeswaxConnection.get_runtime_profile of > > = > 0xcde2310>.get_runtime_profile E+where > = > .client {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8926) TestResultSpooling::_test_full_queue is flaky
[ https://issues.apache.org/jira/browse/IMPALA-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942043#comment-16942043 ] Sahil Takiar commented on IMPALA-8926: -- Yeah, both the {{test_full_queue}} tests are still pretty flaky :(. Still trying to understand why. > TestResultSpooling::_test_full_queue is flaky > - > > Key: IMPALA-8926 > URL: https://issues.apache.org/jira/browse/IMPALA-8926 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > Has happened a few times, error message is: > {code:java} > query_test/test_result_spooling.py:116: in test_full_queue_large_fetch > self._test_full_queue(vector, query, fetch_size=num_rows) > query_test/test_result_spooling.py:148: in _test_full_queue assert > re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) \ E > assert None is not None E+ where None = 0x7f35f0aee320>('RowBatchSendWaitTime: [1-9]', 'Query > (id=e948cdd2bbde9430:082830be):\n DEBUG MODE WARNING: Query profile > created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: > 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: > 0.000ns\n') E+where = re.search E >+and 'Query (id=e948cdd2bbde9430:082830be):\n DEBUG MODE > WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n > - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - > WriteIoWaitTime: 0.000ns\n' = BeeswaxConnection.get_runtime_profile of > 0xcde2310>>( 0xcdf0810>) E+ where BeeswaxConnection.get_runtime_profile of > > = > 0xcde2310>.get_runtime_profile E+where > = > .client {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8926) TestResultSpooling::_test_full_queue is flaky
[ https://issues.apache.org/jira/browse/IMPALA-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941247#comment-16941247 ] Michael Ho commented on IMPALA-8926: Hi [~stakiar], just hit another failure in a build with the latest commit for this JIRA. Can you please take a look ? {noformat} Error Message query_test/test_result_spooling.py:117: in test_full_queue_large_fetch self._test_full_queue(vector, query, fetch_size=num_rows) query_test/test_result_spooling.py:154: in _test_full_queue assert re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) E assert None E+ where None = ('RowBatchSendWaitTime: [1-9]', 'Query (id=7845739f7afbf276:0ca527e1):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: 0.000ns\n') E+where = re.search E +and 'Query (id=7845739f7afbf276:0ca527e1):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: 0.000ns\n' = >() E+ where > = .get_runtime_profile E+where = .client Stacktrace query_test/test_result_spooling.py:117: in test_full_queue_large_fetch self._test_full_queue(vector, query, fetch_size=num_rows) query_test/test_result_spooling.py:154: in _test_full_queue assert re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) E assert None E+ where None = ('RowBatchSendWaitTime: [1-9]', 'Query (id=7845739f7afbf276:0ca527e1):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: 0.000ns\n') E+where = re.search E+and 'Query (id=7845739f7afbf276:0ca527e1):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: 0.000ns\n' = >() E+ where > = .get_runtime_profile E+where = .client {noformat} > TestResultSpooling::_test_full_queue is flaky > - > > Key: IMPALA-8926 > URL: https://issues.apache.org/jira/browse/IMPALA-8926 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > Has happened a few times, error message is: > {code:java} > query_test/test_result_spooling.py:116: in test_full_queue_large_fetch > self._test_full_queue(vector, query, fetch_size=num_rows) > query_test/test_result_spooling.py:148: in _test_full_queue assert > re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) \ E > assert None is not None E+ where None = 0x7f35f0aee320>('RowBatchSendWaitTime: [1-9]', 'Query > (id=e948cdd2bbde9430:082830be):\n DEBUG MODE WARNING: Query profile > created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: > 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: > 0.000ns\n') E+where = re.search E >+and 'Query (id=e948cdd2bbde9430:082830be):\n DEBUG MODE > WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n > - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - > WriteIoWaitTime: 0.000ns\n' = BeeswaxConnection.get_runtime_profile of > 0xcde2310>>( 0xcdf0810>) E+ where BeeswaxConnection.get_runtime_profile of > > = > 0xcde2310>.get_runtime_profile E+where > = > .client {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8926) TestResultSpooling::_test_full_queue is flaky
[ https://issues.apache.org/jira/browse/IMPALA-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937179#comment-16937179 ] ASF subversion and git services commented on IMPALA-8926: - Commit 750f65912046d39a82adbc4907ea8bc673c18964 in impala's branch refs/heads/master from Sahil Takiar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=750f659 ] IMPALA-8926, IMPALA-8957: Fix result spooling flaky tests TestResultSpooling::_test_full_queue was flaky because there was a race condition in the test where the result spooling queue would not fill up quickly enough. The original way around this was to sleep for a fixed amount of time in hope that the queue would fill up by the time the thread woke up. The new approach periodically searches the runtime profile for specific patterns that indicate the queue is full. TestFetchAndSpooling.test_rows_sent_counters was flaky because the RowsSentRate can be 0 if the results are spooled fast enough (because the time spent spooling results is 0). The fix is to use the DEBUG_ACTION BPRS_BEFORE_ADD_BATCH to introduce a delay when spooling results, so that the RowsSentRate is guaranteed to be non-zero. TestFetch.test_rows_sent_counters was flaky because ClientFetchWaitTimer can be 0 if the Coordinator does not end up waiting any time for results to be fetched. The fix is to wait until the query has 'FINISHED' (results are available to fetch) and then sleep so that the ClientFetchWaitTimer is a non-zero value. Cleaned up a few other tests as well. Testing: * Looped both tests for a few hours without failure Change-Id: I3042f592bc79785e43ebc7b09ac1270eae8ed66f Reviewed-on: http://gerrit.cloudera.org:8080/14275 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > TestResultSpooling::_test_full_queue is flaky > - > > Key: IMPALA-8926 > URL: https://issues.apache.org/jira/browse/IMPALA-8926 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > Has happened a few times, error message is: > {code:java} > query_test/test_result_spooling.py:116: in test_full_queue_large_fetch > self._test_full_queue(vector, query, fetch_size=num_rows) > query_test/test_result_spooling.py:148: in _test_full_queue assert > re.search(send_wait_time_regex, self.client.get_runtime_profile(handle)) \ E > assert None is not None E+ where None = 0x7f35f0aee320>('RowBatchSendWaitTime: [1-9]', 'Query > (id=e948cdd2bbde9430:082830be):\n DEBUG MODE WARNING: Query profile > created while running a DEBUG buil...: 0.000ns\n - WriteIoBytes: > 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime: > 0.000ns\n') E+where = re.search E >+and 'Query (id=e948cdd2bbde9430:082830be):\n DEBUG MODE > WARNING: Query profile created while running a DEBUG buil...: 0.000ns\n > - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n - > WriteIoWaitTime: 0.000ns\n' = BeeswaxConnection.get_runtime_profile of > 0xcde2310>>( 0xcdf0810>) E+ where BeeswaxConnection.get_runtime_profile of > > = > 0xcde2310>.get_runtime_profile E+where > = > .client {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org